Data centers’ intricate design: Electrical and power systems
Data centers are important structures that hold vital information for businesses, schools, public agencies, and private individuals. Electrical, power, and lighting systems play a key role in the design of these data centers.
Tim Chadwick, PE, LEED AP, President, AlfaTech Consulting Engineers, San Jose, Calif.
Robert C. Eichelman, PE, LEED AP, ATD, DCEP, Technical Director, EYP Architecture & Engineering, Albany, N.Y.
Barton Hogge, PE, ATD, LEED AP, Principal, Affiliated Engineers Inc., Chapel Hill, N.C.
Bill Kosik, PE, CEM, LEED AP, BEMP, Building Energy Technologist, Chicago
Keith Lane, PE, RCDD, NTS, RTPM, LC, LEED AP BD+C, President/Chief Engineer, Lane Coburn & Associates LLC, Seattle
Robert Sty, PE, SCPM, LEDC AP, Principal, Technologies Studio Leader, SmithGroupJJR, Phoenix
Debra Vieira, PE, ATD, LEEP AP, Senior Electrical Engineer, CH2M, Portland, Ore.
CSE: What PUE goals have clients asked you to achieve in difficult situations? Describe the project.
Chadwick: Our typical data centers are being designed at a peak PUE of 1.3 (to size infrastructure). However, we are seeing annualized performances of PUE in the ranges of 1.07 to 1.20. These low PUEs have required unique design solutions that have relied on new technologies or cooling and power strategies not applied on the scale of the proposed projects. We have been successful at achieving documented, operating PUEs of as low as 1.06 to 1.09 with various Facebook data center sites.
Hogge: In our experience, the PUE metric itself isn't the challenge as much as providing education to understand the difference between a meaningful PUE result and a short-term view. We've been asked for low-maximum snapshot partial PUE as opposed to an annualized number. We've been challenged to deliver a low PUE with a very low IT load on Day 1, although equipment has been deployed for final build-out. Topologies that right-size and plan for growth in a modular fashion help this challenge, but honest conversation about the "why" and the "how" for setting up and analyzing PUE metrics usually leads to an informed client who embraces the best parts of this initiative.
Lane: It is important to clarify the difference between maximum and average PUE. The maximum PUE will be the worst-case design day and failure mode—and will include UPS battery recharge that will dictate the size of the electrical equipment to ensure that at no time the electrical system will be overloaded. The average PUE is a better indicator of the running efficiency and running costs of a data center, but both maximum and average PUE must be considered during the design and implementation of a data center. We have been involved in data centers that have incorporated many electrical and mechanical efficiencies (efficient UPS and transformer, 230 V server distribution, hot-aisle containments, warmer cold aisles, increased delta T, and outside air economizers) that are seeing peak PUEs less than 1.3 and average PUEs of less than 1.2. Be wary of false PUE claims that don't take all losses into consideration. Also, be sure to understand both maximum and average PUE, as they are both critical to understanding the efficiency and the requirements of the electrical distribution system.
Eichelman: As energy efficiency requirements continue to become more stringent, maximum PUE values are being reduced. Federal government facilities are now required to have PUEs of no more than 1.5 for existing facilities and between 1.2 and 1.4 for new facilities, which is fairly consistent with the requirements for most data center owners. To achieve these requirements, very efficient electrical and HVAC equipment and distribution are required. Typical approaches include use of high-efficiency UPS systems and transformers, water- or air-side economizers, high-temperature chilled-water systems, rack- or aisle-level containment systems, direct water-cooled IT equipment, and containerized data center solutions.
One of the main challenges in meeting an aggressive PUE requirement relates to the ramp-up of the IT deployment. As a data center module comes online, the PUE may be very high, as there is often little equipment initially installed on the floor while the corresponding infrastructure is fully built-out. As equipment is added, the PUE continues to fall with the lowest PUE being achieved at full build-out. To reduce these inefficiencies, a modular infrastructure is needed. This infrastructure should be rapidly deployable and planned to match the next IT installation as much as possible.
Sty: The National Renewable Energy Laboratory (NREL) Energy Systems Integration Facility (ESIF) established a PUE goal of 1.06 to all of the design-build teams pursuing the project. In addition, NREL had established requirements to use the waste heat from the high-performance computing data center to provide heating in other areas of the building. The PUE goal was achieved, in part, by the use of direct water-cooled IT cabinets that accept entering water temperatures of 75°F, and leaving water temperatures close to 100°F. This allowed the use of direct evaporative water-side economizer cooling and eliminated the need for mechanical refrigeration. The return water is used for preheating laboratory make-up air heating coils and radiant heating in the offices, which even further reduced the need for the cooling towers. The NREL ESIF facility achieved LEED Platinum using these strategies.
CSE: Describe a recent electrical/power system challenge you encountered when working on a data center project.
Vieira: A high-density data center, located in the Pacific Northwest, with an incredibly small footprint was scalable with multiple electrical services. The distribution substations along with the generators were located outdoors. There were multiple 1600-amp concrete-encased duct banks, which were increased in size due to the client's request to use aluminum conductors. These duct banks were further increased in size due to thermal heating associated with the burial depths and proximity to other duct banks. The design team struggled to physically fit all of the duct banks within the data center footprint. Extensive coordination with the electrical contractor, engineer, and equipment vendors was required.
Hogge: We recently designed a data center where the electric utility was not experienced with this sort of customer. The design team worked closely with the utility to alter policy to allow the service switchgears and backup power distribution system to operate in a flexible and seamless transfer. This highlighted the importance of involving the utility company during the concept design stage and clearly articulating how you intend to operate the power distribution system, sometimes detailing the design downstream of a service transformer.