Meeting electrical infrastructure demands in data centers

A data center’s electrical system requires a robust, reliable infrastructure that far surpasses that of its commercial and industrial facility peers.


Learning objectives

  • Understand today’s data center demands and how to meet them.
  • Know the requirements for key equipment and its installation.
  • Understand how to properly specify wiring for various voltages.

To describe a data center using an analogy, a data center is a womb with no view—for computers. Designed to make complex equipment comfortable, the data center requires a robust and highly reliable electrical infrastructure that far surpasses that of its commercial and industrial facility peers.

These high-reliability infrastructure differences are achieved through meeting unique operational efficiencies, properly selecting and installing electrical equipment, and specifying proper wiring and design methods with appropriate voltages, all while meeting maintainability requirements for planned maintenance.

The first step in this process is to define key electrical system base requirements/data center goals. These are typical for a high-reliability installation:

Bus ducts in data centers have many joints, typically found every 10 ft in straight runs, resulting in equipment reliability and concurrent maintainability issues. Therefore, precise design is critical. Courtesy: Universal Electric Corp.1. Redundant components and systems are equivalent to a person leaving the house in the morning with pants that are too big, so he grabs a belt and a pair of suspenders. If the belt breaks, the suspenders will keep the pants up and vice versa. Either way, he’s covered.

2. Concurrent maintainability means assuring that every component and system (both power and cooling) that supplies the computers can be taken out of service for replacement, repair, or maintenance without shutting the computers down.

3. Fault tolerance, different from concurrent maintainability, means that when any component or system breaks or fails, the systems automatically reconfigure so that the computers don’t shut down. Fault tolerance is an automatic process; concurrent maintainability is a manual process. Part of fault tolerance is compartmentation so that a fire or explosion in one area does not result in total loss of power, cooling, or both to the computers.

4. Complete standby power backup is achieved with a generator plant that is set to provide power when the utility company is not available.

5. Selective overcurrent coordination of circuit breakers and/or fuses is achieved so that during a fault, only the minimum amount of the system is shut down. Ideally, the system will open only the circuit breakers that supply the single piece of failed equipment and nothing else upstream.

6. Modular, scalable construction allows a data center to expand down the road without overbuilding capacity on day one. This is crucial for two reasons: First, everyone is watching their pocketbooks, so if 10 MW of computers will eventually be needed but only 5 MW is needed on day one, the total cost of ownership (TCO) can be minimized by building a modular, scalable shell for 10 MW, but only 5 MW of interior infrastructure for day one. Second, the modular, scalable data center is easier to maintain. Data centers with an excess of unused capacity are a maintenance headache. Careful consideration of the ultimate facility configuration and expansion phases is necessary to minimize risk and eliminate the need for computer equipment shutdown during expansion.

7. Underground circuits are employed in data centers for two reasons: Contractors think they are less expensive to install, and they provide physical security and compartmentation for the data center’s wiring system. However, it is important to note that they require special calculations during the design phase. The Neher-McGrath calculations, found in the National Electrical Code (NEC) 310.15.C and ANNEX B, must be used to design all underground circuits. These calculations often result in the number and size of conductors run underground being substantially larger than would be required overhead. Thus, the anticipated economy compared to overhead circuits is often a false hope.

8. Emphasis on operating efficiency (reduced operating expense, or OPEX) and minimizing TCO can be met by reducing power usage effectiveness (PUE).

Each one of these requirements/goals is crucial because, unlike the typical commercial or industrial facility, the data center load is continuous, with increased ambient temperatures in many areas. For example, the back sections of data cabinets can be 104 to 113 F where branch circuit wiring is installed, while hot aisles can reach the same 104 to 113 F where branch circuit wiring is run upstream from the cabinets. These elevated temperatures result from higher supply air temperature to the computer equipment as a strategy to reduce PUE. Electrical equipment rooms (except those containing storage batteries) can operate at up to 104 F to reduce PUE. The extreme temperatures of a data center make its design for high operating temperatures in addition to code requirements that much more critical than a typical commercial or industrial facility design.

Operations and maintenance

An alternative to bus duct, the cable is assembled like a cable tray with large single-conductor power cables run within it and can be easily modified in the field to fit field conditions. With only two terminations (one at each end, with solid cable in bBeyond unique base design requirements, skilled data center designers also must consider equipment maintenance during design, as ease of maintenance will be crucial to meeting continuous and reliable data center operations. Because a lot of maintenance is required to sustain the critical environment, concurrent maintainability, arc flash labeling, and mean time to repair (MTTR) reduction all play a role in maintaining a data center’s electrical operations.

Designing the data center’s electrical systems to achieve concurrent maintainability means creating an arrangement where any piece of equipment or system that supplies the computers can be taken offline for maintenance purposes while the load continues to operate.

On occasion, maintenance is performed on a piece of equipment while it’s energized (hot work). While a high-reliability data center is designed for concurrent maintainability, some operators choose hot work to reduce maintenance time. While there are a lot of safety procedures in place for this type of maintenance, the best way to understand the risks associated with each piece of data center equipment is to understand its arc flash labeling. This label reflects the arc flash hazard calculated for each piece of equipment and identifies the level of personal protective equipment (PPE) and distances required for safe maintenance. It’s important to understand that some maintenance on small parts in restricted places cannot be performed with NFPA 70E levels 3 and 4 PPE.

Minimizing the time it takes to repair a piece of crucial data center electrical equipment and get it back in service to meet the load’s needs (MTTR) also is important in calculating data center maintenance in advance and when specifying equipment. Proper specification can reduce MTTR. For example, a 4000 A, low voltage, drawout-mounted circuit breaker can be withdrawn and replaced from shelf stock in 15 minutes, while a similar stationary-mounted circuit breaker might take an hour or more to replace.

<< First < Previous 1 2 3 Next > Last >>

HEINZ , CA, United States, 06/06/13 06:35 PM:

In the example on P40:
For a continuous load of 400A, the NEC requires a feeder that can carry 1.25 the continuous 400A = 500A.
So how does a 100% rated CB save on cabling costs.
Also, for a standard 80% rated CB, the terminal temp rating up to 100A is 60*, and 75* for over 100A. What is the temp rating for the terminals of a 100% rated CB?
Consulting-Specifying Engineer's Product of the Year (POY) contest is the premier award for new products in the HVAC, fire, electrical, and...
Consulting-Specifying Engineer magazine is dedicated to encouraging and recognizing the most talented young individuals...
The MEP Giants program lists the top mechanical, electrical, plumbing, and fire protection engineering firms in the United States.
Economics of HVAC systems; NFPA 110-2016; Designing and choosing modular data centers
Combined heat and power; Assessing replacement of electrical systems; Energy codes and lighting; Salary Survey; Fan efficiency
Commissioning lighting control systems; 2016 Commissioning Giants; Design high-efficiency hot water systems for hospitals; Evaluating condensation and condensate
Tying a microgrid to the smart grid; Paralleling generator systems; Previewing NEC 2017 changes
Driving motor efficiency; Preventing Arc Flash in mission critical facilities; Integrating alternative power and existing electrical systems
Putting COPS into context; Designing medium-voltage electrical systems; Planning and designing resilient, efficient data centers; The nine steps of designing generator fuel systems
As brand protection manager for Eaton’s Electrical Sector, Tom Grace oversees counterfeit awareness...
Amara Rozgus is chief editor and content manager of Consulting-Specifier Engineer magazine.
IEEE power industry experts bring their combined experience in the electrical power industry...
Michael Heinsdorf, P.E., LEED AP, CDT is an Engineering Specification Writer at ARCOM MasterSpec.
click me