Data center power strategies

Engineers should take a closer look at the different power strategies being used to distribute power, and how they impact the data center.

By Kenneth Kutsmeda, PE, LEED AP, Jacobs, Philadelphia November 18, 2013

Learning objectives

  1. Understand the different strategies used to distribute power in a data center.
  2. Learn how to measure power efficiency in data centers.
  3. Know which distribution variation is most appropriate for the application.

Nikola Tesla’s alternate current (ac) versus Thomas Edison’s direct current (dc) is a battle that has been going on for more than a century and continues today in the data center industry. Although ac power is the standard, based on its potential for eliminating conversion losses and improving efficiency, many believe that dc power is the future of data center distribution. Still others believe that the same level of efficiency can be achieved with ac by using more efficient equipment with higher voltage distribution such as 415/230 V and 480/277 V.

So how do you know what power strategy is best for your data center application? What are the advantages and challenges of each type of power distribution technique? These are important questions that need to be evaluated when planning a data center. The goal of this article is to take a closer look at the different power strategies being used to distribute power and how they impact the data center.

Electrical efficiency

One of the most common metrics for measuring efficiency in data centers is power usage effectiveness (PUE) created by The Green Grid. It compares the total data center facility power to the power used to run the IT equipment. The optimum data center would have a PUE value of 1.0, where all the power going into the data center is being directly used to power the IT equipment. Any value above 1.0 means that a portion of the total facility power is being diverted to data center support systems such as cooling, lighting, and the power system. The higher the PUE number, the larger portion of power is consumed by the support systems relative to the IT equipment itself, resulting in a less efficient data center.

In the recent past, the primary focus with lowering the PUE and increasing efficiency has been on the mechanical systems and the ability to use free cooling. As data center owners strive to further reduce cost, the focus has shifted toward electrical systems. Electrical systems waste energy in the form of losses due to inefficiencies in the electrical equipment and distribution system.  On average, the electrical distribution system losses account for 12% of the total energy consumed by the data center. For a data center with 2000 kW of IT load (2700 kW total load), that equates to an annual cost of $280,000 (see Figure 2).

Power system design tips

Review these six key items when planning a data center power distribution system:

  • Install or replace existing power and IT equipment with energy-efficient equipment
  • Review the proposed IT equipment to determine if the systems can operate on 240 Vac or 380 Vdc
  • Review all the advantages and challenges of the different power systems
  • Determine how much of the existing infrastructure would need to be replaced to change power systems
  • Design flexibility into the power system that will allow the data center to adapt in the future
  • Design a power system that is modular and scalable to eliminate partial loading 

Similar to the mechanical systems, modifications can be made to the electrical system to make it more efficient and save energy. The key to a good mission critical facility design is not to degrade the reliability of the facility in the process.

Typical electrical distribution systems

The typical legacy data center electrical distribution system is made up of five major components. Power is supplied to the data center at medium voltage from a utility/generator power source. The power is stepped down from medium voltage to distribution voltage (480 V) by a substation transformer. The power then goes through an uninterruptible power supply (UPS) system that conditions the power and provides ride-through capability during an outage until the generator starts. The power is then stepped down to substation voltage (208/120 V) by a power distribution unit (PDU). The PDU supplies power to the IT power supply where it is rectified and stepped down to 12 Vdc, which is the internal operating voltage of the IT equipment (see Figure 3).

The four components in the legacy electrical distribution system with the highest losses are:

  • Substation transformer: Transformer no-load and core losses
  • UPS: Rectifier and inverter losses
  • PDU transformer: Transformer no-load and core losses
  • IT power supply: Rectifier and transformer losses.

One method for increasing efficiency is to replace those pieces of equipment with more efficient equipment. Prior to 2005, when the NEMA TP1 Guide for Determining Energy Efficiency for Distribution Transformers was adopted, transformer efficiencies were around 97%. Today with ultra-high-efficient transformers that efficiency is above 99.5%. Conventional double conversion UPS systems range from 84% efficient at 25% load to 94% at 100% load. Using flywheel or passive standby UPS topology can increase that range to 94% efficient at 25% load and 99% at 100% load.

Another method for increasing efficiency is to eliminate partial loading of the data center. Eliminating partial loading reduces losses by allowing the equipment to operate at its peak operating efficiency. This can be performed by designing a power system that is modular and scalable, one that grows with the load, or by designing a power system that uses flexible tiers, and matches the reliability and redundancy to the different programs within the data center.

A third method is to eliminate the inefficient electrical equipment altogether. Increasing efficiency by eliminating the equipment that has the most losses is the reason why different power strategies are being investigated for data center distribution.

415/240 Vac distribution

A power distribution strategy that is becoming more widely used in the data center is 415/240 Vac. This strategy eliminates the PDU and distributes power at the higher voltage form the UPS straight to the server cabinet. The primary goal is to gain efficiency by eliminating the transformer losses associated with the PDU and by allowing the IT loads to operate more efficiently at a higher voltage (see Figure 4).

In North America, the standard power distribution system is set up in a “wye” configuration with a phase-to-phase voltage of 208 V and a phase-to-neutral voltage of 120 V. In Europe the standard power distribution system is set up in the same “wye” configuration but with a higher voltage distribution. The phase-to-phase voltage is 415 V and the phase-to-neutral voltage is 240 V.

In an effort to standardize between North America and Europe, IT power supplies were developed to accommodate a range of voltages of 100 to 240 V. The concept behind this power strategy is to push the IT power supply to the high side of its voltage range (240 V) and use an established European voltage.

Advantages:

  • Energy efficiency (5% to 7% reduction in losses)
  • Reduced load on the cooling systems
  • Increased reliability
  • Smaller feeder and branch circuit conductor sizes to deliver the same amount of power
  • Gain white space in the data center (two cabinets per PDU eliminated)
  • Reduced maintenance costs (PDU and mechanical systems)
  • Power distribution equipment is readily available.

Challenges:

  • Higher levels of available fault current
  • Potential for arc flash requires higher levels of personal protective equipment (PPE) to work on equipment
  • Full neutral conductor required throughout the system
  • Harmonic influences on the rest of the system.

The main challenge with a 415/240 Vac distribution system is the high levels of available fault current. Removing the PDU from the system also removes the transformer impedance that limits the available fault current downstream in the data center.

Therefore, it is recommended that a short circuit analysis be performed early in the design to determine the available interrupting current (AIC) rating of all electrical equipment and to ensure the equipment is capable of withstanding the higher interrupting current. One option to consider when designing a 415/240 Vac system is breaking up the distribution system into smaller, more modular pieces. By using smaller high-impedance substation transformers, the engineer can reduce the overall fault current on the entire system. Another option to consider is the use of current limiting devices. Since current limiting devices tend to have quick reaction time, it is also recommended that a coordination study be performed to verify that reliability of the system has not been affected.

480/277 Vac distribution

The 480/277 Vac power distribution strategy is similar to the 415/240 Vac in that it eliminates the PDU and distributes power at a higher voltage straight to the server cabinet. The primary goal, advantages, and challenges of the 480/277 Vac power distribution strategy are exactly the same as the 415/230 Vac power distribution strategy (see Figure 5).

A major disadvantage of the 480/277 Vac power distribution strategy is that 277 V exceeds the 240 V rating of most IT equipment power supplies. Implementation of this strategy requires the purchase of custom-made servers with power supplies designed to operate at 277 V. For this reason, the 480/277 Vac power distribution strategy is not as prevalent as the 415/240 Vac power distribution strategy. Currently it is only used in very large facilities where the energy savings outweigh the cost of custom servers due to the high volume of servers that are purchased.

600 Vac distribution

The 600 Vac power distribution strategy is based on using the standard Canadian voltage of 575/347 Vac. Power is stepped down to 600 Vac at the substation transformer and distributed to the UPS system. Power then is distributed from the UPS system at 600 Vac to a PDU located near the data center. At the PDU the voltage is stepped down to either 415/240 V or 208/120 V and distributed to the IT equipment (see Figure 4).

Advantages:

  • Reduction in copper cost (smaller equipment buses and smaller feeders to deliver the same amount of power)
  • Use the full rating of 600 V electrical equipment
  • Lower available fault current (PDU transformer impedance).

Challenges:

  • No gain in efficiency (PDU transformer losses)
  • No gain of white space in data center
  • No reduction in maintenance costs.

Although the 600 Vac distribution strategy does not eliminate PDU transformer losses or reduce maintenance costs, it can lower initial capital expenditure costs. A 600 Vac system takes advantage of the reduced current at higher voltages resulting in smaller or less conductors. Using smaller or fewer conductors will decrease that amount of copper and reduces cost. Higher voltage also allows for larger substations. Depending on the size of the data center, using larger substations may result in a reduction in the total number of substations required.  

380 Vdc power

Contrary to common belief, dc power is very common in the world today. The telecom and transportation industries have been using dc power for years. Alternative and renewable energy generation sources such as solar power, wind power, and fuel cells are dc-based power sources. Most electronic devices in residential homes and in offices internally operate on dc power. And, most importantly, energy storage devices such as batteries and UPS systems operate on dc power.

When you look at a typical traditional data center distribution system, the power gets rectified from ac to dc, inverted from dc to ac, transformed from 480 Vac to 208 Vac, rectified again from ac to dc, and then transformed down to 12 Vdc before powering the IT equipment. Every time the power is converted, losses occur in the form of heat resulting in a decrease in energy efficiency.

The 380 Vdc power distribution strategy distributes dc power from the UPS (dc rectifier) straight to the IT power supply. The primary goal is to gain efficiency by eliminating the inverter losses in the UPS, the rectifier losses in the IT power supply, and the transformer losses associated with the PDU (see Figure 4).

Advantages:

  • Energy efficiency (8% to 10% reduction in losses)
  • Reduced load on the cooling systems
  • Increased reliability
  • Smaller physical footprint
  • Integrates with alternate energy sources
  • Reduced maintenance costs.

Challenges:

  • Limited knowledge and difficult to find electricians with experience on dc systems
  • dc current does not have a zero crossing, difficult to extinguish the arc
  • Have to account for voltage drop on the positive and negative feeders
  • dc arc flash hazards (NFPA 70E provides guidelines for dc arc flash protection).

In addition to the limited number of electricians with dc power experience, the main challenge with dc power in the past has been the lack of standards. This, however, is starting to change. Both the European Telecommunications Standards Institute (ETSI) and the EMerge Alliance have standardized on 380 Vdc and produced guidelines for dc power distribution.  

Unless the data center is completely powered by an alternate source of power, such as fuel cells, it is most likely being provided ac power from the utility. In a dc power system the UPS is used to rectify the power from ac to dc. Because the distribution to the data center is dc, any bypass of the UPS system will also need a rectifier. Consequently, dc systems are more cost effective in a fully redundant (Tier IV) system where a second UPS (dc rectifier) is used as the bypass. Additional things to be aware of when designing a dc power distribution system include using proper protection devices rated for use in dc systems and following the specific requirements for a dc grounding system (refer to IEEE Standard 1100-2005 – IEEE Recommended Practice for Powering and Grounding Electronic Equipment).

In an effort to increase efficiency and reduce cost, different power strategies for distributing power to the data center are starting to be used. Whether you are planning to update an existing data center, expand an existing data center, or build a new data center, designing the power distribution system is a critical part of the plan and one that must be evaluated to determine which system is the correct system for the application.

Theoretical case study

The two power strategies for distributing power to the data center that seem to be gaining the most popularity include the 415/240 V higher ac architecture and the 380 Vdc architecture. A theoretical case study was performed by Jacobs-KlingStubbins to compare the capital expenditure (CAPEX) and operating expenditure (OPEX) of these two power distribution strategies against the typical 208/120 V data center. The case study was based on a theoretical simplified data center with 2 MW IT load, 2 N redundancy (Tier IV), six 750 kVA UPS modules, and 30 5 kW cabinets per row.

The 415/240 Vac system had a 12% CAPEX savings and a 20% OPEX savings when compared to the legacy 208/120 V data center. The 380 Vdc system had a 14% CAPEX savings and a 28% OPEX savings when compared to the legacy 208/120 V data center. It should be noted that unlike the legacy and the 415 Vac systems, the 380 Vdc used the redundant UPS (dc rectifier) as the bypass and did not include a separate bypass on each of the UPS (dc rectifier) systems.


Kenneth Kutsmeda is an engineering design principal at Jacobs (KlingStubbins) in Philadelphia. For more than 18 years, he has been responsible for engineering, designing, and commissioning power distribution systems for mission critical facilities. His project experience includes data centers, specialized research and development buildings, and large-scale technology facilities containing medium-voltage distribution.


Author Bio: Kenneth Kutsmeda is an engineering design principal at Jacobs. For 20 years, he has been responsible for engineering, designing, and commissioning power distribution systems for mission critical facilities. He is a member of the Consulting-Specifying Engineer editorial advisory board and a 2010 40 Under 40 winner.