Designing efficient data centers: Electrical/lighting/power

Respondents

Doug Bristol, PE, Electrical Engineer, Spencer Bristol, Peachtree Corners, Ga.,
Terry Cleis, PE, LEED AP, Principal, Peter Basso Associates Inc., Troy, Mich.
Scott Gatewood, PE, Project Manager/Electrical Engineer/Senior Associate, DLR Group, Omaha, Neb.
Darren Keyser, Principal, kW Mission Critical Engineering, Troy, N.Y.
Bill Kosik, PE, CEM, LEED AP, BEMP, Senior Engineer – Mission Critical, exp, Chicago
Keith Lane, PE, RCDD, NTS, LC, LEED AP BD&C, President, Lane Coburn & Associates LLC, Seattle
John Peterson, PE, PMP, CEM, LEED AP BD+C, Program Manager, AECOM, Washington, D.C.
Brandon Sedgwick, PE, Vice President, Commissioning Engineer, Hood Patterson & Dewar Inc., Atlanta
Daniel S. Voss, Mission Critical Technical Specialist, M.A. Mortenson Co., Chicago

CSE: What are some key differences in electrical, lighting and power systems you might incorporate in a data center as compared with other projects?

Cleis: Lighting systems in data centers are often designed with three things in mind. 1) Most of the time a data center is unoccupied, therefore the lights should be off. If sensors are used, they need to provide coverage in between rows and in all the nooks and crannies that exist in a data center. The lighting and sensors design also needs to take into account tall racks, narrow aisles, and any environmental containment barriers that may be part of the space. 2) When the room is occupied, the lighting system should provide adequate light levels for small tasks taking place in racks located in narrow aisles. Shadows can play a factor on how the illumination reaches the task in these tight areas. 3) There is often a presentation component to a data center, so when the lights are all turned on, the data center should be well-lit and look like the expensive high-tech space that it is.

Bristol: The most notable difference in these systems is with lighting. Typically, all data centers use a “warehouse”-style lighting design that is often void of any fancy controls or architectural fixture types.

Lane: Typically, a data center or other type of mission critical facility will require significantly more power and reliability than other types of facilities. Ensuring the power utility can provide reliable power to the site is critical. Additionally, large standby generators, UPS systems, and electrical switchgear are required in data center facilities. The significant amount of power and the high load factor (average load/peak load) will create significantly more heat in underground duct banks than in a typical project. Therefore, Neher-Mcgrath calculations are required, and often, underground duct banks are oversized to deal with duct-bank heating issues. To ensure greater reliability, UPS systems with large battery banks are used. Battery selection, battery life and monitoring,  spill containment, and proper ventilation are a required part of the design process. Large standby generators required to back up the critical loads will dictate fuel-source design issues, and an acoustical analysis will ensure dB levels at property lines are not exceeded. The large power systems typically require fault current, selective coordination, arc flash studies, and arc flash mitigation.

Voss: Using energy-efficient lighting is mandated per codes for all types of projects, so there are not many differences in lighting. A data center’s two major requirements are electrical power and cooling (derived from electrical power). This mandates that the power systems need to be very robust, dependable, and usually have one or two levels of redundancy.

Peterson: Our design teams have been improving data center power delivery by incorporating software and batteries at the rack and row levels to provide flexible and scalable power capacity that follows power use closely, allowing for savings on capital expenditures and operating expenses of up to 50%. While the enterprise has seen a downturn in growth, these types of systems allow the enterprise to engage the end users and drive the overall benefits of reliability and efficiency. The upfront cost and implementation are often user-based, so while the data center manager may see the benefits of efficiency, being able to have everyone on board allows the organization to reach greater reliability too.

CSE: Electrical/power technology is constantly evolving, putting more demands on engineers working on data center projects. Please describe a recent electrical/power system challenge you encountered when working on a data center project.

Gatewood: Renovating existing data center stocks provides some of the more interesting opportunities and challenges. Balancing budgets against scope and phased reconstruction create dynamically more challenging opportunities. I liken the effort to transforming a Formula 1 race car—while remaining in the race! An example of transforming electrical systems is a 1990 vintage 1.2-mW Tier 2 data center that we turned into a modern 4-mW Tier 3-capable data center in a multiphased renovation. The result was creating an entirely new redundant, diversely routed power and telecommunications infrastructure without suffering an outage to the existing operations. Unique here, however, was that doing such upgrades exposed existing infrastructure to elevated fault currents and loss of selectivity. Without the budget to address all these issues, our team deployed an innovative, removable current-limiting equipped insulated-case switchgear that allowed the fragile electrical system segments to remain in place until a following capital budget permitted their correction. Here, the only sunk cost in the initial phase involved some fuses.

Sedgwick: Isolated-parallel (IP) ring bus technology remains on the cutting edge of data center electrical system design. We’ve recently helped develop and commission IP ring bus systems for several clients, which are used to parallel UPS and generator capacity on a single ring bus with reactor isolation between distributed buses. The inductive reactors limit fault current and allow the systems to equally share kilowatt and kilo volt ampere reactive load across the many parallel systems, which minimizes stranded capacity while providing adequate fault tolerance and a high degree of system redundancy.

Voss: The problem for a modular-designed distribution system was to provide maximum flexibility to distribute power to any location in the data hall, but not strand generator or UPS power. The solution was to interconnect every other UPS-distribution switchboard (via a circuit breaker in each switchboard) so one UPS could furnish power to several distribution switchboards.

CSE: How does your team work with the architects, owners, and other project team members with electrical/power systems so they are flexible and sustainable?

Bristol: We try to use simple but robust building blocks for the entire power system. In the past, data center power systems often grew into complex mazes of panels, feeding panels that, over time, make it very difficult to track power use and the facility’s efficiency. When we use simplistic building blocks, it is much easier to keep track of usage and grow efficiently without having to “trash” an older design and start over.

Voss: Our team first gains a thorough understanding of what the owner and project team are looking to achieve, and we provide fresh ideas that stimulate the entire team. The outcome is positive for all and best for the owner, as the final design provides them with the most sustainable and flexible power systems that the budget will allow.

Lane: A flexible electrical design is very important. We can talk about technology or REVIT, but the key to ensuring a flexible design for future changes and expansion is communication. It is incumbent upon the engineer to understand the owner’s requirements. The engineer must then use their experience to provide the owner with options and a cost analysis of providing the infrastructure required for flexibility. The engineer must be able to discuss the pros and cons of such infrastructure. Once the entire scope if designed and the one-line diagram is completed, as well as the electrical equipment layout, the architect, engineer, and owner must ensure that a failure-mode and maintenance-mode analysis is completed to ensure bypass modes are defined and clearances are provided.

CSE: Describe a co-location facility metering or submetering project. What did it include, and what best practices did you include?

Voss: The tenant desired to know exactly what their computing equipment’s electrical power consumption was every month. After discussing the types of metering that had worked well in the past, we proposed two solutions to the owner that we felt were nearly equal in merit and the least impactful to the existing electrical distribution systems.

CSE: Are you seeing more smart-grid or microgrid aspects for data center projects? If so, how have you served these needs? Are there any issues unique to these projects?

Peterson: Recently, a client was looking into adding future power sources including their own power plant and microgrid. The system design to incorporate this flexibility is not insignificant, especially when the client may even be deemed a power provider, if only for their own site. This seems to mimic the development of cogeneration plants at military facilities where the diesel generators and grid become the secondary and tertiary backup power sources, respectively. Development is requiring close coordination with all parties, as the desire for off-grid power has become a reality.

Sedgwick: We recently worked on a large-scale microgrid control system designed to integrate, monitor, and control a highly complex electrical power distribution system. In this case, the system comprised of multiple independent power sources feeding a high-profile 2.8 million-sq-ft campus. It was designed to optimize power use, respond to source interruptions, and actively manage the connected load to provide a high degree of uptime. The controller integrated photovoltaics, a diesel generator, a fuel cell, battery storage, and utility sources into an intelligent, redundant, optimized control system. This large system was among the most complex installations we’ve seen, combining technologies that are not typically integrated into one system. Within the systems controlled by a microgrid, each device had its own capabilities, limitations, manufacturer-specific design criteria, proprietary communication protocol, and external control-interface requirements. These disparate characteristics made integrating them into one power delivery system challenging, which is only exacerbated by harmonic distortion and the additional problems created when downstream loads are applied to the system. Environmentally focused owners often choose to deploy microgrid controllers in their facilities to maximize renewable energy usage. Unfortunately, renewable energy generation is not typically reliable enough to use in critical facilities. As the technology advances and integration standards are embraced by equipment manufacturers and controls developers, microgrid controllers will likely become more common in the data center industry.

CSE: What types of unusual standby, emergency, or backup power systems have you specified for a data center? What were the project goals?

Peterson: Lithium-ion batteries have seen growth in installations for major data center providers. Smaller footprint, longer life, better performance, and integral controls to prevent overcharging mean that we’ll likely see them soon overtake any of the traditional designs or replacements. Diesel rotary UPS should see gains as more manufacturers begin to provide products and options with better support and costs throughout the world. The reaction time of UPS systems when in full “bypass/eco” mode has improved to match most manufacturers of IT equipment, and we are following this where appropriate as long as the vendor has the guarantee that matches their confidence.

Voss: The most unusual emergency power system used 10 paralleled medium-voltage generators through two switchgears. Each generator was derated to allow it to run 24/7 in case of an extremely long utility outage. The project goals were to be capable of being fully operational without the need for utility power.

Cleis: We have designed generator systems that consist of stand-alone units that are not paralleled. This configuration of generators uses traditional transferring switches to allow for a “catcher” (extra N+1) generator to come online should any of the generators fail. These systems avoid the complexities of paralleling system equipment and also remove the paralleling equipment as a potential single point of failure in the overall system. These systems also simplify the replacement of a single generator unit because the exact size, manufacturer, and pitch of the replacement unit do not have to match those of the unit that it replaces.

CSE: Describe a data center project in which you paralleled generators to ensure uptime.

Sedgwick: We have commissioned many data centers that employ paralleled diesel generators. At one megasite in Atlanta, for example, we have commissioned six independent multi-gen parallel systems over the past decade. This facility has 47 onsite diesel generators rated at least 2 mW a piece. Each system has between eight and ten generators per bus. Critical loads in the building are distributed among the various systems with redundant devices often split between systems. To reduce upfront costs, the client installed only a portion of the generator systems during the initial build; others were added while installed systems were serving live loads. Adding generators to an existing system requires a more careful approach to start-up and commissioning than a greenfield installation does, due to the potential for it to not function properly and negatively impact the existing generators. Throughout the commissioning process, it was necessary to make sure that the site was never without a fully functional generator bus, so we developed thorough method of procedures to guide start-up and commissioning.

Voss: An enterprise data center had a total of six medium-voltage generators that were paralleled through three switchgear line-ups. The total electrical load was supported by four generators (N+2) and the entire electrical distribution system was classified as Tier 4 by Uptime Institute standards.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.