Meeting electrical infrastructure demands in data centers
A data center’s electrical system requires a robust, reliable infrastructure that far surpasses that of its commercial and industrial facility peers.
- Understand today’s data center demands and how to meet them.
- Know the requirements for key equipment and its installation.
- Understand how to properly specify wiring for various voltages.
To describe a data center using an analogy, a data center is a womb with no view—for computers. Designed to make complex equipment comfortable, the data center requires a robust and highly reliable electrical infrastructure that far surpasses that of its commercial and industrial facility peers.
These high-reliability infrastructure differences are achieved through meeting unique operational efficiencies, properly selecting and installing electrical equipment, and specifying proper wiring and design methods with appropriate voltages, all while meeting maintainability requirements for planned maintenance.
The first step in this process is to define key electrical system base requirements/data center goals. These are typical for a high-reliability installation:
1. Redundant components and systems are equivalent to a person leaving the house in the morning with pants that are too big, so he grabs a belt and a pair of suspenders. If the belt breaks, the suspenders will keep the pants up and vice versa. Either way, he’s covered.
2. Concurrent maintainability means assuring that every component and system (both power and cooling) that supplies the computers can be taken out of service for replacement, repair, or maintenance without shutting the computers down.
3. Fault tolerance, different from concurrent maintainability, means that when any component or system breaks or fails, the systems automatically reconfigure so that the computers don’t shut down. Fault tolerance is an automatic process; concurrent maintainability is a manual process. Part of fault tolerance is compartmentation so that a fire or explosion in one area does not result in total loss of power, cooling, or both to the computers.
4. Complete standby power backup is achieved with a generator plant that is set to provide power when the utility company is not available.
5. Selective overcurrent coordination of circuit breakers and/or fuses is achieved so that during a fault, only the minimum amount of the system is shut down. Ideally, the system will open only the circuit breakers that supply the single piece of failed equipment and nothing else upstream.
6. Modular, scalable construction allows a data center to expand down the road without overbuilding capacity on day one. This is crucial for two reasons: First, everyone is watching their pocketbooks, so if 10 MW of computers will eventually be needed but only 5 MW is needed on day one, the total cost of ownership (TCO) can be minimized by building a modular, scalable shell for 10 MW, but only 5 MW of interior infrastructure for day one. Second, the modular, scalable data center is easier to maintain. Data centers with an excess of unused capacity are a maintenance headache. Careful consideration of the ultimate facility configuration and expansion phases is necessary to minimize risk and eliminate the need for computer equipment shutdown during expansion.
7. Underground circuits are employed in data centers for two reasons: Contractors think they are less expensive to install, and they provide physical security and compartmentation for the data center’s wiring system. However, it is important to note that they require special calculations during the design phase. The Neher-McGrath calculations, found in the National Electrical Code (NEC) 310.15.C and ANNEX B, must be used to design all underground circuits. These calculations often result in the number and size of conductors run underground being substantially larger than would be required overhead. Thus, the anticipated economy compared to overhead circuits is often a false hope.
8. Emphasis on operating efficiency (reduced operating expense, or OPEX) and minimizing TCO can be met by reducing power usage effectiveness (PUE).
Each one of these requirements/goals is crucial because, unlike the typical commercial or industrial facility, the data center load is continuous, with increased ambient temperatures in many areas. For example, the back sections of data cabinets can be 104 to 113 F where branch circuit wiring is installed, while hot aisles can reach the same 104 to 113 F where branch circuit wiring is run upstream from the cabinets. These elevated temperatures result from higher supply air temperature to the computer equipment as a strategy to reduce PUE. Electrical equipment rooms (except those containing storage batteries) can operate at up to 104 F to reduce PUE. The extreme temperatures of a data center make its design for high operating temperatures in addition to code requirements that much more critical than a typical commercial or industrial facility design.
Operations and maintenance
Beyond unique base design requirements, skilled data center designers also must consider equipment maintenance during design, as ease of maintenance will be crucial to meeting continuous and reliable data center operations. Because a lot of maintenance is required to sustain the critical environment, concurrent maintainability, arc flash labeling, and mean time to repair (MTTR) reduction all play a role in maintaining a data center’s electrical operations.
Designing the data center’s electrical systems to achieve concurrent maintainability means creating an arrangement where any piece of equipment or system that supplies the computers can be taken offline for maintenance purposes while the load continues to operate.
On occasion, maintenance is performed on a piece of equipment while it’s energized (hot work). While a high-reliability data center is designed for concurrent maintainability, some operators choose hot work to reduce maintenance time. While there are a lot of safety procedures in place for this type of maintenance, the best way to understand the risks associated with each piece of data center equipment is to understand its arc flash labeling. This label reflects the arc flash hazard calculated for each piece of equipment and identifies the level of personal protective equipment (PPE) and distances required for safe maintenance. It’s important to understand that some maintenance on small parts in restricted places cannot be performed with NFPA 70E levels 3 and 4 PPE.
Minimizing the time it takes to repair a piece of crucial data center electrical equipment and get it back in service to meet the load’s needs (MTTR) also is important in calculating data center maintenance in advance and when specifying equipment. Proper specification can reduce MTTR. For example, a 4000 A, low voltage, drawout-mounted circuit breaker can be withdrawn and replaced from shelf stock in 15 minutes, while a similar stationary-mounted circuit breaker might take an hour or more to replace.