Averting a data center legal crisis

01/14/2014


Design, build, and repair costs

Colin Wicker is a senior attorney in Dorsey & Whitney's construction and design practice group. Courtesy: Dorsey & WhitneyObviously, the engineers who design data centers and the contractors who build and maintain them are aware of the critical importance of continued data center operation. They strive to design and build data centers that physically protect the equipment while reliably and continuously supplying electricity and cooling. They aim to provide robust systems. Depending on the client’s budgets and needs, a data center will also have varying levels of redundancy. Redundant systems and components allow owners to conduct maintenance without having to shut down a data center, and can also keep a data center operating despite a failure in a particular piece of equipment or system. The robustness, reliability, and redundancy of a data center are often discussed in terms of the four-tier system created by the Uptime Institute, an industry consulting firm. A Tier 1 data center is just a server room with nonredundant systems, while a Tier 4 data center has fully redundant subsystems so that it is fault-tolerant and concurrently maintainable.

Unlike in most other commercial projects, the electrical and cooling systems are usually the key cost drivers for a data center project. Taken together, the electrical and HVAC systems can be as much as 75% of the total cost of building a data center. These costs are driven by both the amount of electrical load and cooling required to keep the computers operational and at the correct temperatures, and the redundancies designed and built into those systems to avoid unplanned outages and allow for system maintenance that does not require a shutdown of the IT equipment.

Top tier data centers are expensive. Obviously, the cost of any particular data center will depend on a number of factors including size, locale, efficiency of design and construction, and the level of redundancy in the key building systems. However, industry sources and reports in the media provide some rough idea of the magnitude of the costs incurred by those building data centers. In a 2013 report on a survey of 1,000 data center facilities operators, the Uptime Institute stated that it believed typical data center construction costs are approximately $10 million per MW. In 2011, James Hamilton, a vice president and distinguished engineer at Amazon Web Services, similarly estimated at an Amazon Technology Open House that an 8 MW data center would cost $88 million, which does not differ too much from the Uptime Institute’s rule of thumb. Recent news reports show that the very largest data centers can cost in the hundreds of millions of dollars. Facebook, for instance, is spending $300 million to build the first phase of a three-phase data center in Altoona, Iowa, outside of Des Moines. In that same metro area, Microsoft is spending $700 million to expand one of its data centers. The National Security Agency’s troubled new data center in Bluffdale, Utah, which may be the largest data center in the world, is said to have cost approximately $1.5 billion.

Data centers are not just expensive to design and build. They can also be expensive and difficult to repair. If a data center is hosting critical corporate applications, the corporate owner will want a thorough and careful repair to avoid the costs and disruptions of an additional outage. Sears says in its lawsuit that its Troy, Mich., data center failed after four UPS modules went offline. Sears claims to have been billed more than $2.2 million for subsequent repairs. In addition to those repair costs, Sears also spent more than $500,000 on generator fuel and rental equipment used to keep the data center operational while portions of the electrical system were repaired. The new NSA data center in Utah, which is not fully operational, has already experienced serious problems and is in need of repair. It has experienced 10 arc flash incidents, each of which caused up to $100,000 in equipment damage. Over 50,000 man hours have been spent trying to determine the reason for the repeated and serious problems with the electrical system. The costs to remedy the problems are not yet known as the government and its contractors are still determining the source of the problems and how they can best be fixed, but the costs likely will be significant.

If a data center must be kept operational while it is repaired, that can add significantly to the costs. The safety precautions, planning, and methodical approach necessary to safely carry out repair work while continuing to provide power and cooling services to the computer equipment hosted in the same facility slows down the repair process and adds to the cost. Depending on the nature of the failure, the data center owner may also need to rent temporary equipment or purchase fuel to keep generators running, as Sears did after the failures at its data center in Michigan.

If a data center outage is the result of some party’s negligence or breach of its contractual duties, the potential liabilities can be significant. A party responsible for an outage could face claims for lost profits, repair costs, costs spent putting in place temporary solutions while repairs are conducted, and other miscellaneous costs like fuel and the costs of engineers and IT industry consultants involved in investigating a failure. The potential liability from a data center outage can easily rise into the millions. Sears, for example, is claiming that it has at least $4.9 million in damages as the result of outages at its data center in Michigan. And in 2011, Northrup Grumman spent $4.7 million to settle a lawsuit arising from failures at the State of Virginia’s data center.



No comments