Averting a data center legal crisis

Engineers involved in data center design can limit their potential exposure by taking a few simple steps.

By Jocelyn L. Knoll, JD, and Colin Wicker, JD, Dorsey & Whitney LLP, Minneapolis January 14, 2014

When a data center suffers an unexpected outage, the losses to its owners or users can be substantial. So can the potential liability for those involved in designing, constructing, and maintaining the data center, including the consulting engineer. Engineers involved with the design, expansion, or modification of data centers, however, can limit their potential exposure by taking a few simple steps.

Data centers are crucial to the operations of today’s corporations. The most prominent data center owners are Internet-focused corporations like Google and Microsoft, and some public attention is paid to those companies’ data centers. For example, newspapers across the country recently reported on rumors (which subsequently turned out to be false) that Google was building seaborne data centers on barges moored in San Francisco and Portland, Maine. Less attention is paid by the press and public to other owners, renters, and users of data centers, but the data centers used by the corporations are themselves critically important. The public may not appreciate them, but everyday life in the U.S. depends on the continued reliable operation of data centers. Necessary corporate applications like e-mail, payroll, finance, credit card processing, inventory tracking, and e-commerce operations run on the hardware contained in data centers.

The data centers used for a particular application may be corporate-owned, owned by a co-location provider, or owned by a provider of cloud services. Any given corporation may use a mix of strategies in its corporate IT infrastructure. Regardless, firms count on the continued reliable operation of the data centers hosting their applications whether those data centers are owned by them or someone else. Simply put, if the computers at the data center shut down unexpectedly, the e-commerce website, corporate e-mail, and other necessary applications may stop operation.

Consequences of data center failure

Not surprisingly given the criticality of the applications they support, when data centers do fail, the financial consequences can be serious. A 2011 study by the Ponemon Institute of 41 data center outages found an average outage cost of $5,617 a minute, which works out to $337,020 an hour. The Aberdeen Group has a lower, though still significant, assessment of the cost of data center outages. It calculated in February 2012 that an hour of downtime cost an average of $161,000. Of course, particular outage costs vary widely depending on the nature and size of the organization, the timing of the outage, and the nature of the affected applications. For example, a large retailer could lose far more than $5,617 a minute if an outage took down its website during December. A 2013 survey of data center operators conducted by the Ponemon Institute gives some idea of how costly data center outages can be at the high end. Seventeen percent of respondents estimated that a one-hour outage would cost their organization more than $500,000, and 6% said a one-hour outage would cost more than $1 million.

A lawsuit filed in Illinois in June 2013 provides some insight into the costs of a specific outage. Sears sued some of the vendors that provided it with data center-related maintenance and monitoring services. The lawsuit arose out of two outages at Sears’ data center in Troy, Mich. Sears estimated that it lost $1,580,000 in profits as a result of a five-hour data center failure it suffered on January 5, 2013, and another $630,000 as a result of a subsequent failure on January 24. Those lost profits are in addition to the costs of repairing the data center and running generators for weeks while repairs were carried out.

Even if the critical building systems in a data center are quickly returned to normal operation following a failure, it can take much longer to bring a company’s IT applications fully back on line. When a co-location data center owned by Equinix lost power for a single minute in the middle of the night on July 10, 2012, Salesforce.com, which leased space at the data center, was not able to fully bring its service back on line for at least nine hours. Even if an application suffers only a brief outage, the losses (and potential liability for those who may responsible) can be significant if the application itself is sufficiently important. Google may have lost more than $500,000 as a result of a five-minute outage in August 2013.

Design, build, and repair costs

Obviously, the engineers who design data centers and the contractors who build and maintain them are aware of the critical importance of continued data center operation. They strive to design and build data centers that physically protect the equipment while reliably and continuously supplying electricity and cooling. They aim to provide robust systems. Depending on the client’s budgets and needs, a data center will also have varying levels of redundancy. Redundant systems and components allow owners to conduct maintenance without having to shut down a data center, and can also keep a data center operating despite a failure in a particular piece of equipment or system. The robustness, reliability, and redundancy of a data center are often discussed in terms of the four-tier system created by the Uptime Institute, an industry consulting firm. A Tier 1 data center is just a server room with nonredundant systems, while a Tier 4 data center has fully redundant subsystems so that it is fault-tolerant and concurrently maintainable.

Unlike in most other commercial projects, the electrical and cooling systems are usually the key cost drivers for a data center project. Taken together, the electrical and HVAC systems can be as much as 75% of the total cost of building a data center. These costs are driven by both the amount of electrical load and cooling required to keep the computers operational and at the correct temperatures, and the redundancies designed and built into those systems to avoid unplanned outages and allow for system maintenance that does not require a shutdown of the IT equipment.

Top tier data centers are expensive. Obviously, the cost of any particular data center will depend on a number of factors including size, locale, efficiency of design and construction, and the level of redundancy in the key building systems. However, industry sources and reports in the media provide some rough idea of the magnitude of the costs incurred by those building data centers. In a 2013 report on a survey of 1,000 data center facilities operators, the Uptime Institute stated that it believed typical data center construction costs are approximately $10 million per MW. In 2011, James Hamilton, a vice president and distinguished engineer at Amazon Web Services, similarly estimated at an Amazon Technology Open House that an 8 MW data center would cost $88 million, which does not differ too much from the Uptime Institute’s rule of thumb. Recent news reports show that the very largest data centers can cost in the hundreds of millions of dollars. Facebook, for instance, is spending $300 million to build the first phase of a three-phase data center in Altoona, Iowa, outside of Des Moines. In that same metro area, Microsoft is spending $700 million to expand one of its data centers. The National Security Agency’s troubled new data center in Bluffdale, Utah, which may be the largest data center in the world, is said to have cost approximately $1.5 billion.

Data centers are not just expensive to design and build. They can also be expensive and difficult to repair. If a data center is hosting critical corporate applications, the corporate owner will want a thorough and careful repair to avoid the costs and disruptions of an additional outage. Sears says in its lawsuit that its Troy, Mich., data center failed after four UPS modules went offline. Sears claims to have been billed more than $2.2 million for subsequent repairs. In addition to those repair costs, Sears also spent more than $500,000 on generator fuel and rental equipment used to keep the data center operational while portions of the electrical system were repaired. The new NSA data center in Utah, which is not fully operational, has already experienced serious problems and is in need of repair. It has experienced 10 arc flash incidents, each of which caused up to $100,000 in equipment damage. Over 50,000 man hours have been spent trying to determine the reason for the repeated and serious problems with the electrical system. The costs to remedy the problems are not yet known as the government and its contractors are still determining the source of the problems and how they can best be fixed, but the costs likely will be significant.

If a data center must be kept operational while it is repaired, that can add significantly to the costs. The safety precautions, planning, and methodical approach necessary to safely carry out repair work while continuing to provide power and cooling services to the computer equipment hosted in the same facility slows down the repair process and adds to the cost. Depending on the nature of the failure, the data center owner may also need to rent temporary equipment or purchase fuel to keep generators running, as Sears did after the failures at its data center in Michigan.

If a data center outage is the result of some party’s negligence or breach of its contractual duties, the potential liabilities can be significant. A party responsible for an outage could face claims for lost profits, repair costs, costs spent putting in place temporary solutions while repairs are conducted, and other miscellaneous costs like fuel and the costs of engineers and IT industry consultants involved in investigating a failure. The potential liability from a data center outage can easily rise into the millions. Sears, for example, is claiming that it has at least $4.9 million in damages as the result of outages at its data center in Michigan. And in 2011, Northrup Grumman spent $4.7 million to settle a lawsuit arising from failures at the State of Virginia’s data center.

Responding to a data center crisis

Given the tremendous potential legal liability resulting from an outage, how should engineers, owners, and contractors respond to an outage or potential outage? As an initial matter, parties should take reasonable and business-like decisions to prevent or reduce their damages. An owner should attempt to mitigate its damages by taking steps to avoid or shorten an outage. If a court or arbitrator later determines that a data center owner or operator increased its damages by failing to act reasonably in response to a crisis or potential crisis, it may reduce the amount of damages the owner can recover.

And what about engineers, IT vendors, and contractors who become aware of an outage, either as a result of contact from the owner or through some other means? Even if these parties believe they may be later blamed for the incident, they should still consider offering assistance to the owner if they believe they can help reduce the duration and cost of an outage. Doing so could reduce all parties’ potential losses and may also preserve client relationships and one’s professional reputation.

Engineers, however, also need to protect themselves and their businesses. If a data center for which you provided services experiences a serious outage or other failure, you should promptly consult with a lawyer with experience in engineering malpractice matters and notify your malpractice insurance carrier. The failure to provide prompt notice could, depending on the terms of the policy and laws of a particular state, result in a loss of coverage. While it may be in your interest to assist an owner in solving a problem, you should avoid prematurely admitting fault or agreeing to take responsibility for the outage. Doing so could subject you to liability for an incident that may turn out not to be your fault and could also result in a loss of insurance coverage for the incident. Care must also be taken to preserve all relevant documents and data. An experienced lawyer can provide critical advice with regard to your relationship with the owner and insurers so that you preserve your rights while also, to the extent possible and appropriate, providing information and assistance. However, providing assistance and information could be difficult if, as sometime happens, the problems with a data center are discovered years after construction is substantially complete.

Latent defects

As part of their IT planning process, corporations typically expect to increase their use of a data center over time. That is, even if the data center is designed to provide a certain amount of power and cooling for the computers on the raised floor, the data center owner typically does not plan to use all that data center capacity initially. Instead, an owner will forecast its future needs, which can be difficult as technology and use keep changing, and procure data centers to satisfy those needs into the future. As a result, the capabilities of a data center’s infrastructure may not be fully utilized until years after the substantial completion of construction.

Also, data centers are sometimes built in phases with planned expansion. In such cases, some of the infrastructure necessary for later phases, such as underground electrical conduits, may be designed and constructed during the initial phase to avoid later disruptions and inefficiencies. As with the planned increase in the use of a data center, this situation can result in portions of a system that are not used or not fully used for a number of years. In either of these scenarios, systems may be designed and built to a designated capacity but may not experience use at or close to that capacity for years. As a result, failures of design and/or construction could remain undetected for some time. For example, if portions of the electrical system do not have the ampacities required by the owner or specified in the contract documents, the defect may not be found until electrical loads increase. We are aware of at least three instances in recent years in which defects were discovered years after a data center was constructed.

When prompt legal action is necessary

When a defect of construction or design is discovered years after construction, it can cause particular legal issues. Most U.S. jurisdictions have statutes of repose, which are laws that can protect designers and contractors from liability for incidents that occur years after construction is substantially complete. The repose periods differ state by state, but 10 years is a common period. So, an owner who discovers a problem with the design or construction of its data center years after construction is finished will not be able to successfully sue the engineer or contractor if the repose period has run. However, while statutes of repose generally protect engineers, they can have a downside. An engineer who is sued by someone, like an owner, may have little or no time to in turn bring its own claims against other parties that may be responsible, such as a subconsultant, contractor, or subcontractor.

If a problem with a data center is discovered years after construction, perhaps as a result of an outage, it is particularly important to promptly engage legal counsel. Owners may have very little time to file suit, if that’s appropriate, and engineers and contractors may need to be ready to quickly bring their own claims to preserve their rights. We always recommend promptly consulting with an experienced lawyer in these situations, but the situation is particularly urgent when it involves a data center that has been operational for years.

Jocelyn L. Knoll is chair of the construction and design practice group, a member of the energy and insurance practice groups, and a member of Dorsey & Whitney’s policy committee. She has extensive experience in construction and energy law, representing a broad range of clients across the construction, energy, mission critical, and manufacturing industries. Colin Wicker is a senior attorney in the firm’s construction and design practice group.