Top design trends in data centers: electrical, power, lighting
David Anderson, PE, LEED AP
Senior Mechanical Engineer, Principal
Drew Carré, PE
Senior Electrical Engineer
Kupper Engineering Inc.
Terry G. Cleis Jr., PE, LEED AP
Peter Basso Associates
Matt Koukl, DCEP-G
Principal Project Manager, Mission Critical Market Leader
Affiliated Engineers Inc.
Brian Rener, PE, LEED AP
Saahil Tumber, PE, HBDP, LEED AP
Environmental Systems Design Inc.
CSE: Are there any issues unique to designing electrical systems for these types of facilities?
Cleis: With environmental row containment or in-row cooling equipment becoming more common and as computing equipment power and cooling requirements increase at the rack level, coordinating locations for equipment, ductwork, piping and conduits is becoming an important part of planning for the overall layout of a data center. Space for all these systems needs to be planned and coordinated so that they all work together and will allow for proper maintenance and utilization of the rack spaces. This is especially critical when working in an existing building or making modifications to an existing data center.
Carré: Typically, the electrical systems for data centers are designed with redundancy and maintainability in mind. A typical office or commercial building is designed more to accommodate the load, but not necessarily to carry it under failure scenarios. Additionally, many other types of facilities will have “off hours” where equipment can be de-energized for maintenance, testing or upgrades. Data centers require that this work be done without interruption (and limited risk) to the end user or critical load. It is easy to provide two of everything you need, but also very expensive if not done correctly and without considering all of the possible failure and maintenance scenarios. It is also wise to consider how equipment will be removed later, if necessary, while maintaining the other adjacent systems.
CSE: What types of unusual standby, emergency or backup power systems have you specified for such facilities?
Rener: A key issue is to understand that most of the loads on generators for data centers are not “emergency” but optional standby. The actual code emergency loads are quite low. In some of the medium to large data centers it can be advantageous to simply provide a dedicated smaller code-compliant emergency generator system, then design your larger optional standby generator system for the rest of the data center needs. Using this approach, we have had the opportunity to design paralleled generators without concern about the “10-second rule,” but the paralleling systems available today can meet this startup time even with several large generators paralleled.
CSE: What are some of the challenges when designing electrical, power and lighting for data center projects?
Carré: Coordinating all the equipment needed for a higher density data center can be a tall task. Cable tray, power distribution and fire protection are all fighting for space. As rack power grows, the power feeds and cabling grow. Close coordination will prevent issues during installation. Another common issue is right-sizing versus oversizing. Costs need to be kept under control but running out of electrical capacity for the critical load will not be acceptable. Understanding the entire facility’s loads for IT, HVAC, lighting and future growth are necessary to allow for an installation that can accommodate the current loads, as well as the flexibility to add later.
Rener: There are many challenges. One is clearly defining power load requirements. We typically talk in terms or maximum power for any one rack and the maximum per “pod” for the group of racks. This is a challenge for some data center users as you need to define the needs of equipment that has not yet been procured.
Koukl: We recently designed a data center where the electric utility was not experienced with this sort of customer. The design team worked closely with the utility to alter policy to allow the service switchgears and backup power distribution system to operate in a flexible and seamless transfer. This highlighted the importance of involving the utility company during the concept design stage and clearly articulating how you intend to operate the power distribution system, sometimes detailing the design downstream of a service transformer.
Cleis: Design of the auxiliary systems for data centers often take more planning and coordination than other types of spaces. For example, the engineer needs to understand how the spaces above the ceiling and below a raised floor will operate and how they need to be tied to the fire alarm detection and fire suppression systems. Lighting systems need to be designed for small task work within cabinets, while taking into account shadows created by cabinet doors and people working in the aisle ways. Fixture locations need to be coordinated with environment containment equipment and lighting controls need to be designed to support the various tasks and requirements throughout the space.
CSE: What kind of maintenance guidelines are involved to ensure the data center is running efficiently after the project is finished?
Rener: There are several approaches. Manufacturers will have their recommended maintenance programs. Generators, if used for emergency systems, have specific code required on going testing. ANSI/NETA Standard for Maintenance Testing Specifications for Electrical Power Equipment and Systems(ANSI NETA MTS 2015). An interesting twist on all this: concerns that some periodic or routine maintenance may negatively impact reliability.
Carré: Preventive or planned maintenance contracts for the critical infrastructure are necessary. Something as little as clogged air filters can have a huge effect on how efficiently and reliably equipment will operate. Infrared scanning also can be a maintenance item that may allow for early indication of issues. The sooner a potential problem is identified, usually the easier and less costly it will be to repair.
CSE: How are data centers designed to ensure that the infrastructure can handle new, high-density equipment now and in the future?
Koukl: The definition of high density seems to continue to evolve in the industry. The ASHRAE Datacom Series of books and, specifically, IT Equipment Power Trends, Third Edition, provides further guidance on general IT equipment guidance and the impacts this will have to data center design. One of the outliers is going to be the chip power density and the ability to effectively cool some of the newer higher thermal design power central processing units and graphics processing units with air. Likely the answer will be the use of water or another fluid that will provide direct-to-chip cooling.
Cleis: Early communication to establish the maximum power and cooling requirements of the computing equipment at the cabinet level is critical. The engineer needs to then design cooling and power plants sized to support the overall maximum requirements of the higher density equipment. It is important to understand that this often includes targeted cooling equipment that is located physically closer to the computing equipment. The power distribution system needs to be large enough to support this distributed cooling equipment as well support the larger power requirements for the computing equipment mounted within the cabinets. It also is important for the engineer to discuss computing equipment migration. This should include the transition times, while both the existing equipment and new equipment are simultaneously installed and the systems are migrating to the new equipment.
Carré: On the data center floor, bus duct allows for flexibility to the cabinet. Depending on the voltage or amperage, the bus duct can accommodate the load through the plug connected into it. Especially in the case of high density where the power feeds may get quite large (for a single rack), bus duct can provide a large capacity of power to a single area. If multiple racks need large feeders from a panel, those feeders may require a much larger space than the bus duct. Additionally, bus duct plugs can be installed and removed safely while the bus is energized.
CSE: What are some key differences in electrical, lighting and power systems you might incorporate in a data center, compared to other projects?
Carré: For a standard commercial building, it is typical to have a single source of power or maybe a dual utility setup. However, there would not be redundancy or concurrent maintenance available. Much of the design for a data center is double a typical building. For backup power there are potentially larger UPS and generators. Many commercial buildings may not have anything more than a small life-safety generator or they may rely on a battery. If an office environment experiences a 10-second interruption of power, it is usually just an inconvenience. If a data center critical load experiences ANY interruption of power, it is a big deal.
Rener: The amount of power used and density of power is unlike most other projects. Also, most data centers are toward 400-volt systems for data equipment, which is unknown in the United States and requires dedicated transformers to provide this voltage. Also, unlike typical commercial facilities, the entire data center power system protection should be selectively coordinated. Lastly, the level of commissioning and testing is significantly higher for data centers.
CSE: How does your team work with the architect, owner and other project team members so the electrical/power systems are flexible and sustainable?
Carré: Understanding the owner’s requirements for load and load growth is important when making design decisions that allow for future plans. For example, as the load is expected to grow, how does each distribution level have to change to accommodate? Allotting space in electrical rooms for new equipment and verifying the pathways into the room will allow for future rigging. It’s important to build in connection points and/or concurrent maintainability such that when additional equipment must be added it can be done so without impact to the critical load.
CSE: Describe a facility metering or submetering project. What did it include and what best practices did you include for these facilities?
Rener: We typically provide an electrical power monitoring system with meters at most major points of the system. The meter capabilities and features will vary depending on their “level” within the electrical system. Also, the various equipment vendors such as generators and UPS will usually have their own internal monitoring systems and you need to specify the proport interface and protocol to tie them the EPMS.
CSE: Are you seeing more smart grid or microgrid aspects on such projects? If so, how have you served these needs?
Koukl: More of our clients are investigating becoming a distributed generation customer to both assist in a grid emergency as well as receive a rebate. The ability to parallel for an extended time and export to the grid can guarantee power contribution as opposed to islanding and only shedding what the facility is demanding at that time. This can be an effective way to use redundant generator capacity to assist the community and save on the monthly bill.