Data centers achieve a new level of high-tech: Codes and standards
Designing solutions for data center clients — whether hyperscale or colocation facilities — requires advanced engineering knowledge
- Bill Kosik, PE, CEM, BEMP, senior energy engineer, DNV GL Technical Services, Oak Brook, Ill.
- John Peterson, PE, PMP, CEM, LEED AP BD+C, mission critical leader, DLR Group, Washington, D.C.
- Brian Rener, PE, LEED AP, principal, mission critical leader, SmithGroup, Chicago
- Mike Starr, PE, electrical project engineer, Affiliated Engineers Inc., Madison, Wis.
- Tarek G. Tousson, PE, principal electrical engineer/project manager, Stanley Consultants, Austin, Tex.
- Saahil Tumber, PE, HBDP, LEED AP, technical authority, ESD, Chicago
- John Gregory Williams, PE, CEng, MEng, MIMechE, vice president, Harris, Oakland, Calif.
CSE: Please explain some of the codes, standards and guidelines you commonly use during the project’s design process. Which codes/standards should engineers be most aware of?
Starr: In addition to such building and energy codes as Occupational Safety and Health Administration, CFR, IBC, NFPA 1, NFPA 13, NFPA 37, NFPA 70 (NEC), IFC, ASHRAE and IECC, the three most common references for design of data centers are ANSI/TIA 942, BICSI and the Uptime Institute. These three documents drive many considerations from the system topologies to room design and performance requirements. The CBEMA/ITI curve is a useful tool for technical conversations on power quality. The project team should also spend time on maintenance conversations relative to NFPA 70E, NFPA 70B and manufacturer recommendations. The owner team should review NFPA 75. Others that should be referenced: Owner building insurance requirements (such as FM Global), NFPA 111 and 855 for battery storage, ANSI/IEEE (399, 493, 1188, 1679.1) and confirmation of listings such as UL 142, 924, 1642, 1778, 1872, 2085 and 9540.
Williams: From energy efficiency standpoint, when it comes to data center cooling, U.S. Green Building Council LEED v4 and ASHRAE Standard 90.4 are the first to keep in mind. LEED certified data centers usually take advantage of advanced cooling system that reduce energy consumption, improved cooling efficiency, renewable energy systems and renewable or other green backup energy sources and green construction. ASHRAE 90.4 provides the minimum energy efficiency requirements for data centers. ASHRAE TC 9.9 handbook is a useful tool for HVAC applications, energy efficiency practices, etc. This handbook includes thermal guidelines for data processing environments. ASHRAE 90.1 is another valuable guideline for energy efficient building design (excluding low-rise residential buildings). The new Open Standard for Data Center Availability from The Green Grid is a free tool to evaluate the availability of sustainable energy sources in data centers.
CSE: How has ASHRAE Standard 90.4-2016: Energy Standard for Data Centers affected your work?
Kosik: In 2013, ASHRAE Standard 90.4 was released as a code-intended companion to ASHRAE/IES Standard 90.1. The first decade of the 21st century saw tremendous growth in the construction of data center facilities, crossing all sectors of the economy. During this time several industry organizations had been developing guidelines and metrics for data center energy efficiency and while many of these metrics are quite useful and technically sound, over time it became evident that a single method to determine data center energy efficiency compliance was needed.
In Standard 90.4, compliance is based on mechanical load component and electrical loss component values in comparison to maximum values allowed per the design conditions. But it is important to understand that a data center design will comply with 90.4 only if it complies with 90.1. The design must satisfy the appropriate compliance path, which includes building systems such as envelope, service water heating, lighting and other equipment.
In late 2019, ASHRAE released the 90.4-2019 standard. While the overall structure of the standard remains the same, there are some big changes in the content. The maximum values for the MLC and the ELC have been lowered, resulting in a more demanding compliance process. Based on the people I have talked with, the consensus is that the minimum efficiency requirements in 90.4-2016 were too easy to achieve; it seemed inevitable that these changes would be made.
Peterson: In most cases we aim for a new data center and existing ones being updated to meet and exceed the standard anyway, so that when the standards are applied to the local/state codes the resolution will be to show the energy use in an acceptable fashion. In most jurisdictions there is not wide acceptance or expectation of compliance with the standard and accordingly not much effort has been needed to show calculations.
Williams: Our clients are always demanding lower energy costs, ASHRAE Standard 90.4-2016 gives them that. Because ASHRAE has begun to raise its acceptable data center operating temperature and humidity ranges in 2011, the trend toward free cooling optimization has grown to the point that its almost exclusively the design of choice for our data center clients.
CSE: What are some best practices to ensure that such buildings meet and exceed codes and standards?
Williams: Best practices are still evolving for these high energy use and high cooling requirement environments, so we are in an evolution process currently. Such techniques are increasing like real time thermal monitoring with detection of hot spots using sensors for temperature and infrared cameras, corrosion tracking using humidity sensors and, real time equipment temperature monitoring using embedded temperature sensors inside the servers and storage systems. More into the future we are seeing discussion of artificial intelligence and machine learning techniques being applied for assisted temperature, humidity, pressure and air flow monitoring. Frequent measurement and sampling as recommended by ASHRAE and the USGBC. These could provide valuable standard compliance information. Using virtual thermal analysis tools like computational fluid dynamics and continuous validation and calibration it will be possible to have an insight on how well a certain design meets or exceeds related requirements.
Starr: At a minimum, I suggest having data center projects commissioned. Further, make sure the commissioning team is brought onboard early enough to provide a review before 100% construction documents. Note a standard commissioning review focuses on “can equipment be…”: installed, operated, maintained and eventually replaced. A good commissioning review should also highlight any code violations and confirm that the project’s owner’s project requirements- and basis of design-noted requirements are met by the drawings and specifications. Beyond a standard commissioning review, there is great value in asking the commissioning team or third-party engineering firm, for an enhanced peer review. In the owner’s request for proposal, make sure the qualifications of the reviewer align with the scope they are peer reviewing.
CSE: How are codes, standards or guidelines for energy efficiency impacting the design of data centers?
Peterson: We have always approached projects with the intent of maximizing the energy efficiency, not just to meet code but to ensure that the owners and end-users can also count on lower operating costs. We have applauded how the codes have pushed the minimum performance of equipment and systems and manufacturers have stepped up to exceed those standards so that we can count on designing best-performing facilities.
Williams: ASHRAE Standards 90.1 or 90.4 are good starting points for designing data centers. However, because these only define a minimum requirement for data centers’ energy efficiency, designers need client input to push the envelope. When looking at the large tech companies such as AWS, Google or Facebook, we are seeing that these companies are developing their own innovative systems to overcome heat removal challenges in data centers. These systems are in many instances exceeding the minimum design requirements significant that are mandated by standards or rating systems. Hence in general the current standards are lagging the technological advancement in data center design that is available and being used by high-tech companies.
Kosik: Early in the timeline of data center efficiency, the primary focus was on cooling and power systems. This was logical because these systems consumed the most energy in a data center, just behind the IT equipment. Also during this time, a number of manufacturers were developing new, data center specific, HVAC and electrical equipment that would play a big part in reducing the overall energy use of the facility. However, many in the industry understood that a data center is more than computers, power and cooling systems and equipment. In particular, the USGBC added specific language and credits for data center facilities to the LEED rating system. In conjunction with improving the efficiency of the data center power and cooling systems, all of the other green building requirements apply, resulting in a more holistic approach.
CSE: What new or updated code, standard, guideline organization or association do you feel will change the way such projects are designed, bid out or built?
Starr: ASHRAE 90.4 is one to become familiar with, but otherwise, the most significant industry influence right now revolves around energy storage systems, such as batteries for uninterruptible power supply applications. This, compounded with the rise of lithium-ion over valve-regulated lead acide or flooded-cell battery options, has the industry learning about a new type of battery installation while also having significant code and standard updates, such as 2018 IFC 1206, 2018 NFPA 1 and NFPA 855. Lithium-ion battery form factors we see in the news (such as heating up cell phones) are different from those used in large energy storage systems.
Still, energy storage needs to be taken as a real concern for people and property. Design safeties must be included in products, the listing should be by a Nationally Recognized Testing Laboratory approved testing agency and the layout of the building systems should consider fire and battery maintenance safety. Most lithium-ion battery cabinets available are less than the 50 kilowatt hours code threshold, but still need 3 feet of separation from each other. This separation includes the 50 kilowatt hours chunks in pre-engineered systems up to 250 kilowatt hours. UL9450A testing documentation approved by the authority having jurisdiction may eliminate the need for separating battery cabinets. Over 600 kilowatt hours is also a new challenge, as incidental use classifications cannot exceed 10% of the building, thus driving some new construction to hazard occupancy classifications. Further, the newest energy storage requirements may require FMEA and enhanced permitting requirements.
Williams: The updated version of ASHRAE 90.4-2019, released in December 2019 and its future updates reflect the trend in data center design optimization happening in the industry. Another guide is also being prepared to look at a CFD guideline specific for data centers. This is looking to create a baseline practice for data centers’ thermal analysis and help to unify current thermal analyses methodologies. With organizations like Facebook and AWS taking initiatives to design their own custom energy and water efficient cooling systems and strategies, we hope their experiences and technologies will become open sources and that can serve as valuable additional guidelines and case studies that engineers can take advantage of in a wider role.
Peterson: There have been many groups leading the way on data center project performance goals and developing new metrics. Infrastructure Masons has a lot of top-notch contributors to the Data Center Performance Index white paper. ASHRAE funded, developed, published and updated 14 handbooks as well as upkeeping standards and investing in directly related research. The Open Compute Project has shown year-over-year increases in adoption and impact as they continue to absorb innovations from over two hundred organizations.
CSE: What are some of the biggest challenges when considering code compliance and designing or working with existing buildings?
Williams: Designing energy efficient facilities needs to account for both design goals of optimal data center cooling in this context and energy efficiency goals being partially satisfied. There is a narrow range that engineers could stretch their design parameters to maximize energy efficiency. For example, some data centers operate at 80°F while the upper limit specified by ASHRAE Standard 90.4 for air temperature in data centers is 80.6°F. This requires an intensive control strategy and backup plan to prevent system failure and malfunctioning during extreme climate or process events. In existing buildings, space limitation and existing infrastructures (outdated and inefficient infrastructure, limited capacity, integration difficulties, etc.) could be the limiting factors in code compliance and achieving the expected energy efficiency.
Peterson: Beyond the typical building code compliance challenges to modernize a data center, these facilities are often looking to maintain other certifications, such as ISO27001. We have phasing plans around the improvements with a thorough review of the applicable codes alongside the governing officials. Our approach includes prediction and ongoing assessments as renovations reveal the need to alter work to meet the latest compliance.
Starr: Some of the biggest challenges when considering code compliance are total energy storage kilowatt hours, maintaining original emergency power off design intent, fire-stopping new and abandoned penetrations and being cautious of building separations for campus installation when routing power between them. Some of the biggest challenges when considering design compliance are:
- Coordinating the likely sequence for outages for cutovers.
- Considering how the new systems will be fully commissioned.
- Working with design partners to evaluate floor ratings for new equipment.
- Finding floor space for new equipment layouts without having to isolate related equipment.
- Working with existing system architectures — such as centralized or distributed UPS static switches that may require feeder lengths match — may be a challenge in renovation projects when the outage window to perform de-energized work is minimal.
- High arc flash incident energy concerns prompting consideration of innovative solutions such as G&W’s CLiP.
Some of the biggest challenges when considering performing work within existing buildings are physical equipment routes through the building, strictly following standard operating procedures and writing detailed methods of procedure, keeping active spaces clean of construction debris and trying to troubleshoot with a clear mind while being subject to an outage window.