Focus on data centers: Electrical and power

Designing efficient and effective data centers and mission critical facilities is a top priority for consulting engineers. Six engineers share their knowledge of electrical and power systems in data centers.

By Consulting-Specifying Engineer January 21, 2013

Participants

  • Cyrus Gerami, PE, LEED, CxA, Associate, Senior Project Engineer/Manager, exp Global Inc., Maitland, Fla.
  • Kerr Jonstone, IEng, MIET, Senior Electrical Engineer, CH2M Hill, Glasgow, Scotland 
  • Keith Lane, PE, RCDD/NTS, LC, LEED AP, President, Lane Coburn & Assocs., Bothell, Wash. 
  • James McEnteggart, PE, Vice President, Primary Integration Solutions Inc., Charlotte, N.C.
  • Robert M. Menuet, PE, Senior Principal, GHT Ltd., Arlington, Va. 
  • Brian Rener, PE LEED AP, Electrical Platform Leader and Quality Assurance Manager, M+W Group, Chicago, IL.

Electrical/Power

CSE: Describe some recent electrical/power system challenges you encountered when designing a new building or retro-commissioning an existing building.

Rener: We recently faced a challenge in coordinating the ride-through time of rotary UPSs and the standby generators. The generators require a certain amount of time to ramp up and come to full load while the rotary UPSs have limited ride-through time.

McEnteggart: On a recent project, our client was expanding the UPS system to support new white space. The challenge was testing the new equipment without putting the ongoing data center operations at risk. Because we were involved during design, we were able to ensure that the system was configured to allow independent operation of the new equipment while allowing the existing UPS system to continue to support the critical space.

Johnstone: On a Terremark project, a considerable electrical systems challenge we encountered was where we converted an existing catering facility into a new data center. One of the main electrical systems challenges was achieving the adequate space required for the considerable amount of electrical equipment, which is essential in ensuring a reliable and high availability electrical system. We overcame these challenges by working extremely closely as a project and client team to develop an electrical system efficiently space planned ensuring all plant was compartmentalized but located in extremely close proximity to related plant, which increased efficiency while reducing installation cost.

CSE: How do you balance the need for reliable power with the desire for efficiency and sustainability?

Lane: Good engineers can provide a design that provides both reliable power and energy efficiency. Right-sizing equipment, understanding the client’s reliability needs (they are not all the same), selecting the most efficient equipment, using higher voltages, and working closely with the mechanical engineer can ensure success.

Gerami: Reliability must be the primary factor for mission critical “essential facilities.”

McEnteggart: The catcher system is a good example of a design concept that attempts to balance reliability with efficiency. It allows high utilization of the installed infrastructure; however, it introduces the risk of fault propagation from the primary system to the redundant system. This can allow a fault in one part of the system to cascade through the redundant system and take down multiple parts of the facility. On paper, it is as reliable as some of the more conservative designs, but the owner has to decide if the potential efficiency of the catcher system is worth the risk to the system’s reliability.

Rener: When using backup power systems such as UPSs, there has been a great focus on energy efficiency conversion technologies. There is a wider variety of topologies from UPS vendors to provide highly energy efficient designs. Coincidentally, higher power densities had resulted in the use of higher voltages such as 400 Vac and even 380 Vdc, which also results in less energy loss.

Johnstone: On a Terremark project, we implemented a static flywheel UPS system, which provided reliable backup power until the facility’s standby diesel generators started and were available to take the facility’s full load. The flywheel’s autonomy was 15 seconds at full IT design load. While this appears insufficient time in terms of a UPS autonomy, it was proved during factory tests—and more importantly on-site during commissioning of the electrical systems—that we were able to have the generators online and supporting the full load of the facility in less than 10 seconds. This meant we didn’t have a large quantity of batteries that are typically found in traditional data center designs; by removing these batteries we became considerably more efficient in terms of not just the space and the requirement to condition the rooms to tight environmental tolerances, but also the sustainable aspect of disposal/replacement of a large quantity of batteries over the facility lifespan was an added benefit.

CSE: Describe a recent project in which you specified standby or emergency power in a mission critical facility. What challenges did you face, and how did you overcome them?

Rener: A common issue with many projects we have, due to the size and number of diesel generators, is environment concerns. Exhaust emissions have become concerns due to more stringent air quality permitting requirements, and limitation on run times.

McEnteggart: We are the commissioning agents on the design team for phase one of a series of planned expansions of a data center in South America. The owner has limited space for the required number of generators to provide emergency power for phase two, and wisely wants to avoid painting the data center into a corner. So the team is assessing the design and operational impacts of a number of alternative solutions, including building a superstructure to vertically stack the generators. In our experience, this solution complicates fuel distribution, but it has proven to be a viable solution under severe space constraints.

Johnstone: On the Terremark project, we implemented diesel standby generators configured in a distributed redundant configuration. This created the challenge of ensuring all generator sets were synchronized after a mains failure. We had to work closely with the generator vendor to develop a control and monitoring system in order to facilitate this design requirement effectively.

Lane: As the load factors in data centers are rising, it is important to specify the correct standby generator for your facility. If your data center is anticipated to have a load factor in the 80 to 90 range, you should not specify a standard standby rated generator rated at a 70% load factor. We specified and installed mission critical rated generators at the Sabey Quincy Data Center. These generators are rated for a load factor of 85%.

CSE: What PUE goals are clients requesting, and what tactics are you using to accomplish this?

McEnteggart: Typical PUE requests range from 1.3 to 1.5. To achieve this, the design must incorporate into the building’s mechanical and electrical infrastructure techniques such as air-side economization and evaporative cooling, water cooling, new UPS systems with “energy saver” mode, and high-voltage power supply where these are appropriate.

Rener: We are seeing requests for PUE less than 1.3 and sometimes much less. The key has been the ability to use “free cooling” depending on site location, using hot and cold aisle containment, and pushing the thermal limits for operating inlet temperatures of data equipment. The use of 400 Vac distribution is also allowing us to avoid additional transformers and the losses associated with stepping down to 120/208 V.

Lane: We see a lot of claims of extremely low PUE numbers in the industry. We must be careful to dig deeper and understand what these numbers represent: for example, best case scenario, calculations, average or peak real PUE numbers. The electrical infrastructure must be sized based on the peak worst-case PUE, but the cost of running the facility will be based on average PUE.

I have summarized below a few of the methods to reduce PUE:

Sabey Data Centers use the most efficient, state-of-the-art, full online double-conversion UPS systems on the market, with efficiencies as high as 97% over a broad range of loading percentages. Additionally, mechanical equipment is served by two sources of power. The roof-mounted units come integral with an ATS to allow a transfer at the roof unit level to the secondary source if the preferred source goes out. Two control circuits are provided and UPS backed. IT PODS (racks) are fully contained to eliminate hot/cold air exchange and to significantly increase energy efficiency. Roof-mounted air handling units (AHUs) provide cold air outside the racks. The rack enclosures are considered the “hot aisle.”

Johnstone: Clients are targeting very low PUEs—industry advertising indicates PUEs in the 1.1 to 1.2 levels. However, these are not achievable for every solution. In some aspects there is still an element of informing the client about the achievable solutions based upon their design criteria, location, and tier aspirations, and not all clients are educated in the implications of achieving very low PUEs. Adopting good design principles will typically achieve below 1.5 PUE on all but the most demanding projects.

CSE: What is your primary choice for standby, backup, and emergency power?

Rener: The old standard of a diesel generator is still commonplace. In many cases due to high power demands, the standby equipment is now specified at medium voltage (15 kV). However, we are getting requests for more “battery-less” continuous power systems to marry up to the generators. This commonly points to rotary UPS or in some cases generator rotary combination systems. There are interesting new developments such as compressed air storage, fuel cells, and other new technologies.

McEnteggart: From an operational perspective, diesel generators are still my primary choice for emergency power. Some owners are interested in microturbines, gas turbines, or cogeneration. While these may be economically attractive alternatives for data centers, an owner cannot rely on these systems to be there when they are needed—especially if they are burning natural gas. As we learned on the East Coast during Hurricane Sandy, natural gas utilities may shut down the flow for safety reasons. As far as I know, this did not impact any large commercial data centers, but in my mind, the benefit of natural-gas-fired emergency power isn’t worth the risk.

Johnstone: We do not believe that there is a single primary choice on this subject as every project has its own challenges that can and do very often result in very different solution in terms of standby, backup, and emergency power. These challenges are influenced by clients’ preference, geographical location, product support, space availability, and cost of ownership.