Data center efficiency, cooling considerations
Engineers should know how inefficiencies in electrical systems affect data center cooling load.
When developing data center energy-use estimations, energy engineers must account for all sources of energy use in the facility. Most energy consumers are obvious: computers, cooling plant and related equipment, lighting, and other miscellaneous electrical loads (see Figure 1). These items are easily understood because the power goes in and some type of useful work comes out. However, when the conversation turns toward the electrical distribution system and major components such as transformers and UPSs, it is unclear how (and how much) energy is consumed in these systems.
So one may ask, “Can an electrical distribution system really consume energy?” Energy is “lost” due to inefficiencies in the equipment and distribution systems. Calculating this loss is essential because the energy is dissipated as heat and requires even more energy from the cooling system to ensure the proper internal environmental conditions are met. Although the phrase “energy loss” is used throughout this article, in an isolated system, energy can change location within the system, it can change form, but it can never be created nor destroyed.
Electrical system energy consumption includes all the power losses starting from the utility through the building transformers, switchgear, UPS, power distribution units (PDUs), and remote power panels, ultimately ending at the IT equipment. Each of these components has some level of inefficiency resulting in a transformation of power into heat.
Some of these components have a linear response to the percent of total load they are designed to handle. Others exhibit a very nonlinear behavior, which is important to understand when estimating overall energy consumption in a data center with varying IT loads. Having multiple concurrently energized power distribution paths can increase the availability (reliability) of IT operations. However, running multiple electrical systems at partial load can also decrease the overall system efficiency.
In order to clearly define a process for explaining energy loss in an electrical system, it is important to understand the primary drivers that have the biggest influence on system efficiency:
- UPS module efficiency
- Part-load efficiencies
- System modularity
- System topology (reliability)
- Estimating cooling load based on electrical system losses.
While it is probable that in most data centers reliability and maintainability are the predominant design requirements, it is possible to achieve the uptime goals and optimize energy efficiency at the same time. Electrical and mechanical engineers need to work collaboratively to make it happen. The focus of this article is to illustrate how inefficiencies in electrical systems impact the cooling load and to define the main points that mechanical engineers must understand when designing data center cooling systems. It is not the intention of this article to suggest concepts for electrical distribution systems or UPS module configurations.
UPS module efficiency
There are many different types of UPS technologies. Some are better suited for smaller loads, where others are used almost exclusively for very large IT loads. The final UPS technology selection really depends on the specific case. With this in mind, it is important to know that different UPS sizes and circuit types have varying efficiency curves—it is not a one-size-fits-all proposition.
Looking at the efficiency curves of legacy UPS equipment installed in government facilities from the U.S. Dept. of Energy self-reported UPS efficiencies, it is clear that the passive standby topology has the highest efficiencies—94% is the lowest measured value, 99% is the highest (see Figure 2). In contrast, the lowest measured double-conversion UPS topology efficiency is 58.7%; the highest is 93.7%.
While modern UPSs are typically more efficient, it is still important that mechanical engineers understand UPS type, size, and efficiencies at different loading points in order to properly size cooling equipment and estimate energy consumption. Even in the early stages of concept design, the mechanical engineer should be working closely with the electrical engineer to document the initial assumptions on UPS type and efficiency, so these can be used in early energy simulation models.
Each UPS type has a different part-load performance. This is important because the data center will most likely operate at loads much lower than the ultimate capacity. The U.S. Environmental Protection Agency is in the process of finalizing a uniform testing and reporting procedure for UPS equipment. Among the requirements are procedures for standardized UPS equipment testing at 100%, 75%, 50%, 25%, and 0% loading for ac output UPS equipment.
This is the same approach used by the European Commission’s code of conduct for UPS equipment. It is prudent to conduct analysis of energy consumption and power usage effectiveness (PUE) at these same points to determine how the UPS and electrical distribution system as a whole react to IT loads that are operating at less than the ultimate capacity.
Figure 3 shows the total facility PUE for two different reliability configurations at different part-load conditions. The charts indicate (and what would be typical of this type of comparison in general) that the N+1 configuration has lower efficiencies and higher PUE values at low loads. As the loads approach 100% of the ultimate load, the efficiencies of the systems begin to converge. At lower part-load values, the higher reliability systems generally have higher overall electrical system losses than a lower reliability system has. As the load percentage approaches unity, the gap narrows between the two systems. The absolute losses of the N+1 system will be 50% greater at 25% load than the N system, but this margin drops to 23% greater than the N system at 100% load.
When calculating annual data center energy consumption, it is advisable to include a schedule on the IT load that more closely resembles the actual operational schedule of the IT equipment, thus providing a more accurate estimate of energy consumption. This schedule could contain the predicted weekly or daily operation of the computers (based on historic workload data) but, more importantly, the long-term ramp-up of the power requirements for the computers. With this type of information, the overall annual energy consumption planning and analysis will be more accurate.
Like the UPS equipment efficiency, the modularity of the electrical system has a large impact on overall system efficiency. UPS modules are typically designed as systems that consist of multiple modules. Within the system, there could be redundant UPS modules or redundancy in the systems themselves. This depends entirely on how the electrical engineer intends to meet the owner’s reliability, expandability, and cost requirements. The greater the number of UPS modules, the smaller the portion of the overall load each module will be required to handle.
The effects of this become pronounced in high-reliability systems at low loads where it is possible to have a single UPS module working at 25% (or lower) of its rated capacity. Figure 4 shows an example of how the loading levels vary for different electrical system topologies. In this specific example, the curves demonstrate the different loading levels for the overall UPS system based on project-specific design parameters. Note that based on the curve for the 2N system, the percentage of load will never exceed 50% of the total IT load. Referencing DOE data, a UPS loaded to 50% of capacity will have an approximate average efficiency of 90%. The other 10% is given up as heat and must be conditioned by the cooling system.
System topology (reliability)
When all of the UPS modules, systems, and electrical equipment are pieced together to create a cohesive electrical distribution system—which should be designed to ideally meet certain reliability and availability requirements—efficiency values at the various loading percentages are developed for the entire system. The entire system now includes all the power distribution elements upstream and downstream of the UPS equipment.
In addition to the loss incurred by the UPS equipment, the losses from transformers, switchgear, PDUs (with and without static transfer switches), and distribution wiring must be accounted for. Depending on the specific installation, these items will add 3% to 5% to the UPS losses. When all of these components are analyzed in different system topologies, loss curves can be generated so the efficiency levels can be compared to the system reliability, assisting in the decision-making process. For example, Figure 5 shows nine different electrical system topologies and the fit-curves that depict the efficiencies across a range of different loading levels. There is a negative correlation between the anticipated electrical system reliability and its efficiency. Generally, the higher the reliability is, the lower the efficiency.
How UPS efficiency affects PUE
Ultimately, it is the overall energy consumption and PUE of the data center facility that matters. While we’ve looked at the UPS system concepts in isolation, this is not meant to create a silo around the electrical system—integration is important and must be considered.
Early in the design process, a timeline of the anticipated IT load growth must be developed in order to properly design the power and cooling systems from a modular growth standpoint. If modeled properly, the part-load efficiencies for the electrical system—and the cooling system—will drive the algorithms that determine the amount of energy used, as well as the amount dissipated as heat.
It is remarkable that the calculated PUE values are extremely high at extremely low loads (less than 10% of total load). This is an essential concept to understand. It would be quite disappointing to calculate the PUE after the first year of operation and find that it is unacceptably high. Knowing that if the IT loads were at very low percentages during this time period will bring some comfort. It is possible that PUE values at low loads can exceed 10.0 (see Figure 6). Even at 10% load, the PUE is twice as much as the PUE at full build-out PUE. Keep in mind that the UPS is just a part of the equation that is driving the PUE values up; the PUE is burdened with overhead items required to operate the data center, such as lighting, administrative space, and back-of-the-house power. Determining the maximum annual PUE is important to sizing the maximum capacity of the electrical system as well.
Estimating cooling load based on electrical system losses
Because losses from the electrical systems result in additional heat gain that must be conditioned (except for equipment located outdoors or in nonconditioned spaces), the mechanical engineer should consider these data when sizing the cooling equipment and evaluating annual energy consumption. The cooling equipment efficiency will determine the amount of cooling energy required to offset the heat from the electrical losses.
It is essential to include cooling system energy usage from electrical losses in lifecycle studies for UPSs and other electrical system components. It is possible that lower cost, lower efficiency UPS equipment will have a higher lifecycle cost due to the cooling energy required—even if the capital cost is significantly less than a high-efficiency system. For example, two different data center electrical systems were analyzed to determine annual energy use attributed to electrical system losses. The analysis included the cooling energy for the facility and the electrical losses. The first scenario analyzed was a representative legacy data center electrical system with total system efficiency of 80%. The second scenario was a highly-efficient UPS and electrical distribution with total system efficiency of 90%. When the output from the energy model was studied, it became quite evident that the efficiency of the UPS and electrical distribution system largely affected the cooling energy and overall facility PUE (see Figure 7). In addition to the energy that is lost, the cooling load resulting from the loss had a negative impact on the annual energy use and PUE for the facility. Consequently, the electrical system inefficiencies had a two-fold effect on energy consumption.
For a recent data center project located in Prague, the client requested a detailed analysis of the power and cooling system performance, comparing several options at four loading points: 25%, 50%, 75%, and 100%. An important element of the analysis was having a granular understanding of how energy use changes as the IT load increases over the life of the project.
In order to analyze the electrical systems in these scenarios, we first determined the number of UPS modules, the number of systems, and the different topologies under consideration. The four levels of loading were identified as levels 1 through 4. Within each level, we analyzed how many UPS modules would be needed to handle the IT load. This was done in a way to minimize over-buying UPS equipment and to match the load as close as practical with the modules.
Two primary electrical topologies were analyzed: N+1 and 2N. For each topology, we determined energy use at each of the four levels to calculate the overall facility PUE. The energy efficiency increased as the percentage of IT load increased (see Figure 8). Additionally, operating the data center within any of the four loading point ranges yielded unique PUE values for each of the scenarios.
Data center reliability and availability are fundamentally important to the center’s operation. Fortunately, the industry has recently responded well with myriad new products and services to help increase energy efficiency, reduce costs, and improve reliability. When planning a new data center or considering a retrofit to an existing one, the combined effect of the different disciplines collaborating in the overall planning and strategy for power, cooling, and IT systems can produce strategies and tactics that yield high levels of efficiency and reliability. Using the right kind of tools and analysis techniques is an essential part of accomplishing this goal.
Kosik is principal data center energy technologist with HP Technology Services. Kosik is one of the main technical contributors shaping HP Technologies Services’ energy and sustainability expertise and consults on client assignments worldwide. A member of the Consulting-Specifying Engineer editorial advisory board, he has written more than 20 articles and spoken at more than 35 industry conferences.