Data center design considerations

This article provides guidelines on distribution systems’ levels of redundancy, the correct generator rating to use, and whether solar power can be used in a data center.

12/27/2017


Learning objectives:

  • Know the difference between 2N, 3M2, and N+1 system topologies.
  • Understand the characteristics of the system topologies.
  • Learn the criteria for rating generators.
  • Understand the different transformer types.

Over the past several years, mission critical clients seem to be asking the same series of questions regarding data center designs. These questions relate to the best distribution system and best level of redundancy, the correct generator rating to use, whether solar power can be used in a data center, and more. The answer to these questions is “It depends,” which really doesn’t help address the root of their questions. For every one of these topics, an entire white paper can be written to highlight the attributes and deficiencies, and in many cases, white papers are currently available. However, sometimes a simple and concise overview is what is required rather than an in-depth analysis. The following are the most common questions that this CH2M office has received along with a concise overview.

Figure 1: Conceptual one-line configurations of electrical topologies highlighting redundancy. Courtesy: CH2MWhat is the best system topology?

There isn’t a single “best” system topology. There is only the best topology for an individual data center end user. The electrical distribution system for a data center can be configured in multiple topologies. While the options and suboptions can be myriad, the following topologies are commonly deployed (see Figure 1).

  • 2N: Simply designing twice as much equipment as needed for the base (i.e., N) load and using static transfer switches (STS), automatic transfer switches (ATS), and the information technology (IT) and HVAC equipment’s dual cording to transfer the load between systems. The systems are aligned in an “A/B” configuration and the load is divided evenly over the two systems. In the event of failure or maintenance of one system, the overall topology goes to an N level of redundancy.
  • 3M2: This topology aligns the load over more than two independent systems. The distributed redundant topology is commonly deployed in a “three-to-make-two” (3M2) configuration, which allows more of the capacity of the equipment to be used while maintaining sufficient redundancy for the load in the event of a failure (see Figure 2). The systems are aligned in an “A/B/C” configuration, where if one system fails (e.g., A), the other two (B and C) will accept and support the critical load. The load is evenly divided with each system supporting 33.4% of the load or up to 66.7% of the equipment rating. In the event of a component failure or maintenance in one system, the overall topology goes to an N level of redundancy. In theory, additional systems could be supplied, such as 4M3 or 5M4, but deployment can significantly complicate the load management and increases the probability of operator error.
  • N+1 (SR): The shared-redundant (SR) topology concept defines critical-load blocks. Each block is supported 100% by its associated electrical system. In the event of maintenance or a failure, the unsupported equipment would be transferred to a backup system that can support one or two blocks depending on the design. This backup system is shared across multiple blocks, with the number of blocks supported being left to the design team but typically in the range of 4:1 up to 6:1.
  • N+1 (CB): The common-bus (CB) redundant system is like the shared redundant system in that the IT equipment’s A and B sources are connected to an N+1 uninterruptible power supply (UPS) source, but in the event of a failure or maintenance activities, the load is transferred to a raw power source via STS. The raw power source has the capability of being backed up by generators that are required to be run during maintenance activities to maintain the critical load.

Figure 2: Mission critical electrical room showing overhead conduit routing and complexity for a 3.6-MW, three-to-make-two (3M2) distribution system. Courtesy: CH2MThe above topologies assume a low-voltage UPS installation. However, similar systems can be developed using a medium-voltage UPS. Beyond the redundancy configuration, these low-voltage UPS topologies also can be evaluated on ease-of-load management, backup power generation, their ability to deploy and commission initially and when expanding, first costs and total cost of ownership, physical footprint of the equipment comprising the topology, and time to construct the initial installation as well as expansion of the system.

A commonality between the different topologies presented is the need to transfer load between systems. No matter the system topology, the requirement to transfer load between electrical systems—either for planned maintenance activities, expansions, or failure modes—must be done. Load management refers to how the load is managed across multiple systems.

2N topology. The premise behind a 2N system is that there are two occurrences of each piece of critical electrical equipment to allow the failure or maintenance of any one piece without impacting the overall operation of the data center IT equipment. This configuration has a number of impacts:

  • Load management: Among the topologies presented here, 2N has a relatively simple load-management scheme. The system will run independently of other distribution systems and can be sized to accommodate the total demand load of the IT block and associated HVAC equipment, minimizing the failure zone. The primary consideration for load management is to ensure the total load doesn’t overload a single substation/UPS system.
  • Backup power generation: This topology uses a 2N backup generation with the simplest of schemes: having the generator paired to the distribution block. Each generator is sized for the entire block load and will carry 50% of the load under normal conditions. For large data centers, the option exists to parallel together multiple generator sets to create an “A” backup source and parallel together an equal number of generators to create a “B” backup source, distributing power via two different sets of paralleling switchgear. Typically, this is more expensive due to the addition of paralleling switchgear and controls. Selection of the voltage class usually depends on the size of load as well as physical space and cost to route cable from the generator to the switchgear. The ability to parallel generators tends to be limited by the paralleling switchgear bus ampacity ratings as well as short-circuit ratings. Beyond 6,000 amps at 480 V, consider using 15-kV-class generators.
  • Deployment: Each 2N system can be designed to accommodate a discrete IT block. This allows multiple systems to be deployed independently, facilitating procurement, construction, commissioning, and operations with no impact to existing or future systems.
  • First cost/TCO: The 2N system requires twice the quantity and capacity of electrical equipment than the load requires, causing the system to run at nominally 50% of nameplate capacity. Due to the nature of how electrical equipment operates, this tends to cause the equipment to run at a lower efficiency than can be realized in other topologies. An additional impact of the 2N system topology is that the first cost tends to be greater because of the quantity of equipment. Also, because there are additional systems in place, the ongoing operational and maintenance costs tend to be greater.
  • Spatial considerations: Because it generally has the most equipment, the 2N configuration typically has the largest physical footprint. However, this system is the simplest to construct as a facility is expanded, thereby minimizing extra work and allowing the facility to grow with the IT demands.
  • Time to market: As has been discussed, this system will have more equipment to support the topology, therefore there may be additional time to construct and commission the equipment. The systems are duplicates of each other, which allows for construction and commissioning efficiencies when multiple systems are installed, assuming the installation teams are maintained.

Figure 3: Installation of feeders and control cables for a 3.6-MW 3M2 distribution system. Courtesy: CH2MDistributed redundant (3M2) topology. The premise behind a 3M2 system is that there are three independent paths for power to flow, each path designed to run at approximately 66.7% of its rated capacity and at 100% during a failure or maintenance event. This configuration is realized by carefully assigning load such that the failover is properly distributed among the remaining systems.

This configuration has a number of impacts to the distribution:

  • Load management: The load management for the 3M2 system should be carefully considered. The load will need to be balanced between the A, B, and C systems to ensure the critical load is properly supported without overloading any single system. Load management of a system like this can be aided by a power-monitoring system.
  • Backup power generation: This topology follows the normal power flow and uses a 3M2 backup generation where the generator is paired to the distribution block. Each generator is sized for the entire block load and will carry 66.7% of their capacity under normal conditions. Parallel generator configurations are rarely used for 3M2 systems. Like 2N systems, the selection of the voltage class depends on the size of load as well as physical space and cost to route cable from the generator to switchgear (see Figure 3).
  • Deployment: Each 3M2 system can be designed to accommodate a discrete IT block. Expansion within a deployed 3M2 system is exceptionally challenging and difficult, if not impossible to commission. Deployment of multiple 3M2 systems is the best option for addressing expansion and commissioning.
  • First cost/TCO: The 3M2 system requires about 1.5 times the capacity of electrical equipment than the load requires and runs at 66.7% of its rated capacity. Because the equipment is running at a higher percentage, the 3M2 system tends to be more energy-efficient than the 2N, but less efficient than either of the shared redundant systems. An additional impact of the 3M2 system topology is that lower-capacity equipment can be used to support a similar size IT block, thereby causing the system to have a higher cost per kilowatt to install. However, if the greater capacity is realized by either sizing the IT blocks large enough to realize the benefits of this topology or by installing two IT blocks on each distribution system, then there will be a lower first cost. Essentially, the 2N system needs two substations and associated equipment for each IT block while the 3M2 system would need only three substation systems to support the IT block. First-cost savings is in addition to operational savings because there are fewer pieces of equipment to maintain. And the energy savings is because the equipment is running at a higher efficiency.
  • Spatial considerations: Similar to the first-cost discussion above, the spatial layout can either be smaller or larger than a 2N system depending on how the topology is deployed and how many IT blocks each system supports.
  • Time to market: The balance between the IT blocks supported by each system and the quantity of equipment will have an impact on the time to market, though the balance for this system is unlikely to be significant. The additional equipment should be balanced against smaller pieces of equipment, allowing faster installation time per unit. The systems are duplicates of each other, which allow for construction and commissioning efficiencies when multiple systems are installed, assuming the installation teams are maintained.

Figure 4: Rendering of a CH2M design of a data center, conference center, and office buildings for Saudi Airlines. Courtesy: CH2MN+1 shared redundant (N+1 SR). The premise behind the N+1 SR system is that each IT block is supported by one primary path. In the event of maintenance or a failure, there is a redundant but shared module that provides backup support. The shared module in this topology has the same equipment capacities and configuration as the primary power system, minimizing the types of equipment to maintain.

For example, if six IT blocks are to be installed, then seven distribution systems (substations, generators, and UPS) will need to be installed for an N+1 system. This N+1 system can easily be reconfigured to an N+2 system with minimal impact (procuring eight systems in lieu of seven). This reconfiguration would allow the system to provide full reserve capacity even while a system is being maintained.

This configuration has several impacts to consider:

  • Load management: The N+1 SR system has the simplest load management of topologies presented. As long as the local UPS and generator are not overloaded, the system will not be overloaded.
  • Backup power generation: This topology follows the normal power flow and uses an N+1 SR backup generation where the generator is paired to the distribution block. Each generator is sized for the entire block load, with the SR generator also sized to carry one block. Parallel generation can be used for block-redundant systems. However, carefully consider the need for redundancy in the paralleling switchgear. True N+1 redundancy would require redundant paralleling switchgear. However, this level of redundancy while on generator power may not be required.
  • Deployment/commissioning: The deployment of the N+1 SR system is modular because each system functions independently. However, commissioning a new system with an existing redundant system may be challenging if the redundancy needs to be always available for the critical load. In the event of a multiple-fault scenario (multiple generators failing to operate or multiple UPS failing to support the load while generators start), the faults will cascade and overload the redundant system. There are multiple ways to mitigate this risk (load-management tripping breakers or inhibiting the STS), but the concern is valid. Any of the methods implemented to prevent a cascading failure will cause some IT loads to go offline.
  • First cost/TCO: For a large-scale deployment (i.e., exceeding two modules), the N+1 SR system has the lowest installed cost per kilowatt of the systems explored here that have full UPS protection for both the normal and redundant power distribution systems, due to the lower quantity of equipment. In addition, less equipment should also result in lower ongoing operation and maintenance costs.
  • Spatial considerations: The N+1 SR layout will have the smallest spatial impact. Additional distribution is required between modules as well as a central location to house the redundant system.
  • Time to market: The balance between the IT-blocks distribution system and the quantity of equipment will have an impact on the time to market. However, due to the fact that the N+1 SR has the smallest quantity of equipment, this configuration potentially has the shortest time to market of any system explored so far. This timing is further supported due to system duplicates, which should allow for construction and commissioning efficiencies on the subsequent installations, assuming the teams are maintained.

N+1 common bus (N+1 CB). The premise behind the N+1 CB system is there is one primary path that supports each IT block. This path also has an N+1 capacity UPS to facilitate maintenance and function in the event of a UPS failure. The system is backed up by a simple transfer switch system with a backup generator.

This configuration has a number of impacts on the distribution:

  • Load management: Similar to the N+1 SR system, the load management for the N+1 CB is simple. As long as the local UPS/generator combination is not overloaded, the system will not be overloaded.
  • Backup power generation: Like the previous topology, there is a generator paired to each distribution block including the redundant block.
  • Deployment/commissioning: The deployment of the N+1 CB system is a modular deployment because each system functions independently. The only location where existing work has to be tested with the new equipment is on the common bus system.
  • First cost/TCO: The N+1 CB system potentially has the lowest installed cost per kilowatt of any of the systems. This lower cost is due to a combination of lower quantities of UPS and generators coupled with simpler distribution. Additionally, less equipment means ongoing operation and maintenance costs should be lower as well.
  • Spatial considerations: The N+1 CB layout will have a small spatial impact. Additional distribution is required between modules as well as a central location to locate the central bus system (transfer switches and generator).
  • Time to market: Similar to the N+1 SR system, the N+1 CB has significantly fewer pieces of equipment than the 2N or 3M2 systems. This equipment count should support a faster time to market. However, it is difficult to determine which of the N+1 systems would have a quicker time to market.

Figure 5: The photo shows the outdoor installation of the power electronic switch of the medium-voltage UPS at Michigan State University. Courtesy: CH2MThe above topology descriptions only highlight a few systems. There are other topologies and multiple variations on these topologies. There isn’t a ranking system for topologies; one isn’t better than another. Each topology has pros and cons that must be weighed against the performance, budget, schedule, and the ultimate function of each data center.

What generator rating should be used for a data center?

Generators need to be able to deliver backup power for an unknown number of hours when utility power is unavailable. To help select the appropriate generator, manufacturers have developed ratings for engine-generators to meet load and run time requirements under different conditions. The International Standards Organization (ISO) Standard 8528-2005, Reciprocating Internal Combustion Engine Driven Alternating Current Generating Sets, tries to provide consistency across manufacturers. However, the ISO standard only defines the minimum requirements. If the generator is capable of a higher performance, then the manufacturer can determine the listed rating. To complicate generator ratings even more, some industries have their own ratings specific to that industry and application. These various ratings can make selecting the correct generator type complicated.

There are four ratings defined by ISO-8528:

  1. Continuous power: designed for a constant load and unlimited operating hours; provides 100% of the nameplate rating for 100% of the operating hours.
  2. Prime power: designed for a variable load and unlimited running hours; provides 100% of nameplate rating for a short period but with a load factor of 70%; 10% overload is allowed for a maximum of 1 hour in 12 hours and no more than 25 hours/year.
  3. Limited running: designed for a constant load with a maximum run time of 500 hours annually; same nameplate rating as a prime-rated unit but allows for a load factor of up to 100%; there is no allowance for a 10% overload.
  4. Emergency standby power: designed for a variable load with a maximum run time of 200 hours/year; rated to run at 70% of the nameplate.

Figure 6: The photo shows the outdoor installation of the switchgear in an enclosure used in conjunction with the power electronic switch. Courtesy: CH2M

The generator industry also has two additional ratings that are not defined by ISO-8528: mission critical standby and standby. Mission critical standby allows for an 85% load factor with only 5% of the run time at the nameplate rating. A standby-rated generator can provide the nameplate rating for the duration of an outage assuming a load factor of 70% and a maximum run time of 500 hours/year.

Data center designs assume a constant load and worst-case ambient temperatures. This does not reflect real-world operation and results in overbuilt and excess equipment. Furthermore, it is unrealistic to expect 100% load for 100% of the operating hours, as the generator typically requires maintenance and oil changes after every 500 hours of run time. Realistically during a long outage, the ambient temperature will fluctuate below the maximum design temperature. Similarly, the load in a data center is not constant. Based on research performed by Caterpillar, real-world data center applications show an inherent variability in loads. This variability in both loads and ambient temperatures allows manufacturers to state that a standby-rated generator will provide nameplate power for the duration of the outage and it’s appropriate for a data center application. However, if an end user truly desires an unlimited number of run hours, then a standby-rated generator is not the appropriate choice.


<< First < Previous Page 1 Page 2 Next > Last >>

Product of the Year
Consulting-Specifying Engineer's Product of the Year (POY) contest is the premier award for new products in the HVAC, fire, electrical, and...
40 Under Forty: Get Recognized
Consulting-Specifying Engineer magazine is dedicated to encouraging and recognizing the most talented young individuals...
MEP Giants Program
The MEP Giants program lists the top mechanical, electrical, plumbing, and fire protection engineering firms in the United States.
May 2018
40 Under 40 winners, fire and life safety, performance-based designs, and more
April 2018
VRFs, lighting controls, BIM coordination, and more
March 2018
Sizing water pipes, ASHRAE 90.1, recovering waste heat, and more
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me