ASHRAE’s energy standard for data centers turns three
When specifying to meet ASHRAE Standard 90.4, examine the lessons learned to evaluate a path forward
- Understand how the ASHRAE Standards 90.1 and 90.4 differ.
- Learn some of the history of how ASHRAE’s compliance language for data centers changed over the years and eventually evolved into Standard 90.4.
- Identify possible changes to Standard 90.4 and understand why they are being proposed.
The late 1990s saw a tremendous growth in private and public sector use of large-scale computing systems. During this time and the subsequent tech industry bubble, national labs and other private-sector organizations were already studying energy use of computing systems and data centers. This seminal work, much of it fueled by the U.S. government, shaped much of our current-day thinking on data center energy use.
Fast-forward more than a decade, scores of formal and informal organizations are attempting to tackle the issue of energy use in data centers. There are many approaches: server efficiency and virtualization, cooling system optimization, reducing electrical system losses (focusing on uninterruptible power supply technology), modular data centers, climate-based efficiency and many others. Some organizations are explicitly vendor agnostic, while other researchers are a part of a manufacturing entity.
While mostly good-intentioned, the sheer amount of research and recommendations (sometimes divergent) created an atmosphere of the “wild west,” without any sheriff in charge. But, in 2015, after several years of development, ASHRAE released drafts of the Energy Standard for Data Centers for public review and comment. The following year, after the review process concluded, ASHRAE released Standard 90.4-2016: Energy Standard for Data Centers. The sheriff had arrived.
The predecessor to Standard 90.4
ASHRAE Standard 90.1: Energy Standard for Buildings Except Low-Rise Residential Buildings includes compliance provisions for data centers. However, up until 2010, ASHRAE 90.1 had little information addressing data center-specific requirements, even though the industry was rapidly evolving during that time period. ASHRAE needed to catch up on the substantial progress occurring in the data center industry, unify the different efforts underway and provide technical leadership for energy compliance in data centers.
Fortunately, the later versions of Standard 90.1 begin to remedy some of the difficulties that arose using earlier versions. The newer versions have more in-depth, tailored requirements written specifically for computer rooms. For example, the 2010 standard introduces a new efficiency metric (sensible coefficient of performance), which is used solely to determine efficiency of computer and data processing room air conditioning units.
The ASHRAE 90.1-2013 standard arguably has the most detail on data center energy use compliance. It includes clarifications and new requirements that, if followed, removes the provision for an economizer. Also included are sizing requirements for water economizers and a new alternative compliance path using power usage effectiveness. (This is controversial because PUE consists of many other components that will skew the results and has been viewed by some as an inadequate way to study data center energy use.)
And even with the progress made to clarify and improve the process and techniques for judging data center energy efficiency, the requisites for information technology equipment listed in Standard 90.1 still had a tendency for greater applicability to smaller data centers or server closets, not on large-scale, standalone facilities. The data center industry continued to mature and develop innovative, energy-efficient products and services. Similarly, the evolution of ITE continued, with (what seems like) constant release of new products.
In most cases, every new generation of technology equipment included better energy use performance. While ASHRAE 90.1-2013 showed promise in standardizing an energy compliance process for data centers, several aspects of data center design were not addressed in it. Once again, ASHRAE was under pressure to develop and publish a comprehensive energy standard dedicated to data centers.
ASHRAE uses a continuous maintenance program for its standards and works on a three-year review cycle, after which an updated version of a standard is released. Releasing an updated standard is a massive undertaking (even more so if it is a new standard). It is safe to say the ASHRAE’s committees were hard at work on the upcoming release of what would be referred to as ASHRAE Standard 90.4-2016 not too long after the release of Standard 90.1-2013. During late 2015 and early 2016, ASHRAE was releasing documents for public review anticipating the release of 90.4 later in 2016. Some of these documents were proposed addenda to Standard 90.1 to eliminate overlap with the yet-to-be-released provisions of 90.4. The primary document released for public review was called ASHRAE 90.4P and it ultimately garnered more than 1,000 comments; this is the basis for the current ASHRAE 90.4 standard.
Then, on July 27, 2016, ASHRAE published its brand-new standard called Standard 90.4-2016: Energy Standard for Data Centers. By design, 90.4 has much less content than the 90.1 standard, which is about four times longer. Instead of trying to weave in data center-specific language into Standard 90.1, ASHRAE wisely chose to create a separate standard that is only applicable to data centers and refers to Standard 90.1 for non-data center specific compliance items, such as building envelope.
It is important to understand that ASHRAE 90.1-2016 is the normative reference to ASHRAE 90.4. (The normative references to Standard 90.4 mostly consist of compliance language for building envelope, service water heating, lighting and other requirements.) This creates a system of documents designed to eliminate overlaps and minimize confusion between the two standards. This structure streamlines the ongoing maintenance process as well and ensures that Standards 90.1 and 90.4 stay in their respective lanes to avoid any misunderstandings relating to the technical and administrative boundaries of the two standards.
Finally, updates to ASHRAE 90.1 will automatically update ASHRAE 90.4 without significant changes to the text. In the same way, updates to ASHRAE 90.4 presumably will not affect the language in ASHRAE 90.1.
Standard 90.4 gives the engineer a completely new method for determining energy use compliance for data centers. ASHRAE introduced new terminology for demonstrating compliance: design and annual mechanical load component and electrical loss components. ASHRAE urges caution to not confuse these new metrics with The Green Grid‘s PUE and points out that MLC and ELC are to be used only in the context of Standard 90.4 and are not to be used for comparison or ranking purposes. The standard includes compliance tables consisting of the maximum load components for each of the 19 ASHRAE climate zones.
Design mechanical load component
To determine compliance, MLC can be calculated one of two ways. The first is a summation of the peak power of the mechanical components in kilowatts, as well as establishing the design load of the information technology equipment, also in kilowatts. ASHRAE 90.4 has a table of climate zones with the respective design dry-bulb and wet-bulb temperatures that are to be used when determining the peak mechanical system load. The calculation procedure is shown below. It must be noted that when comparing the calculated values of design MLC, the analysis must be done at both 100% an 50% ITE load; both values must be less than or equal to the values listed in the ASHRAE table 6.2.1 (design MLC).
= [Cooling design power (kilowatts) + pump design power (kilowatts)
+ Heat rejection design fan power (kilowatts) + air handling unit design fan power (kilowatts)]
÷ Data center design ITE power (kilowatts)
Annualized mechanical load component
The concepts used for the annualized MLC path are like the design MLC, except to use the annualized MLC path, an hourly energy analysis is required. This compulsory hourly energy use simulation considers fluctuations in mechanical system energy consumption, particularly in cases where the equipment is designed for some type of economizer mode, as well as energy reductions in vapor compression equipment from reduced lift due to outdoor temperature and moisture levels.
This approach seems to be the most representative of the energy performance of the data center, because it is based on industry standard energy analysis techniques (i.e., hourly energy use simulation techniques). Again, it must be noted that when comparing the calculated values of annualized MLC, the analysis must be done at both 100% an 50% ITE load; both values must be less than or equal to the values listed in the ASHRAE 90.4 table 220.127.116.11 (annualized MLC).
= [Cooling design energy (kilowatt-hour) + pump design energy (kilowatt-hour)
+ Heat rejection design fan energy (kilowatt-hour) + air handling unit design fan energy (kilowatt-hour)]
÷ Data center design ITE energy (kilowatt-hour)
Design electrical loss component
Using the Standard 90.4 approach to calculate the ELC defines the electrical system efficiencies and losses. ASHRAE 90.4 defines three electrical system segments that are used to determine ELC:
- Incoming electrical service segment.
- UPS segment.
- ITE distribution segment.
The segment for electrical distribution for mechanical equipment is stipulated to have losses that do not exceed 2% but is not included in the ELC calculations. All the values for equipment efficiency must be documented using the manufacturer’s data, which must be based on standardized testing using the design ITE load. The final submittal must consist of an electrical single-line diagram and plans showing areas served by electrical systems, all conditions and modes of operation used in determining the operating states of the electrical system and the design electrical loss component calculations demonstrating compliance.
ASHRAE 90.4 tables 18.104.22.168 and 22.214.171.124 list the maximum ELC values for ITE loads less than 100 kilowatts and greater than or equal to 100 kilowatts, respectively. The table shows the maximum ELC for the three segments individually as well as the total.
The efficiency of the electrical distribution system impacts the data center’s overall energy efficiency in two ways: lower efficiency means greater electricity use and a greater air conditioning load to cool the electrical energy dissipated as heat. ASHRAE 90.4 is explicit on how this should be handled: “The system’s UPS and transformer cooling loads must also be included in [the MLC], evaluated at their corresponding part load efficiencies.” (From section 126.96.36.199.1.1)
The standard includes an approach on how to evaluate single feed UPS systems (e.g., N, N+1, etc.) and active dual feed UPS systems (2N, 2N+1, etc.). The single feed systems must be evaluated at 100% and 50% ITE load. The dual active feed systems must be evaluated at 50% and 25% ITE load because these types of systems will not normally operate at a load greater than 50%. An added benefit to this process is a demonstration of how reliability affects energy use (see Figure 1).
Standard 90.4 uses a performance-based approach rather than a prescriptive one to accommodate the rapid change in data center technology and to allow for innovation in developing energy-efficiency cooling solutions. Some of the provisions seem to especially encourage innovative solutions:
- On-site renewables or recovered energy: According to the standard, if included in the data center design, an offset credit may be taken for on-site renewable generation or recovered energy. The advantage of using on-site renewables and energy recovery is that the daily peak will be eased, reducing greenhouse gas emissions and decreasing the demand for electricity.
- Derivation of MLC values: The MLC values in the tables in Standard 90.4 are derived from data on mechanical system energy consumption, based on different types of system design and equipment type (modeling and real world). The values are not created according to any system type or manufacturer; they are suitable for a broad range of mechanical systems. It is important to remember that the MLC values are the minimum efficiencies needed for compliance. Ideally the project would go beyond the minimum using highly energy-efficient designs and demonstrate even greater energy efficiency.
- Design conditions: The annualized MLC values for air systems are based on a delta T (temperature rise of the supply air) of 20 F and a return air temperature of 85 F. However, the standard has a proviso that states if the cooling system is designed to operate at higher supply air and/or water temperatures, the engineer has greater flexibility to innovate and propose nontraditional designs, such as water cooling of the ITE equipment (see Figure 2).
- Trade-off method: Sometimes certain aspects of the mechanical and electrical system design make it difficult to meet the requirements of the standard. To alleviate this situation, the standard stipulates that the mechanical and electrical systems can be analyzed together, allowing trade-offs. Where one system may not be compliant, the other may exceed the requisites, thereby offsetting the efficiency conditions.
Looking ahead to Standard 90.4-2019
The next version of Standard 90.4 will arrive later this year. In accordance with ASHRAE’s procedures for reviewing and updating standards, there have been several proposed addenda that have been released for public comment. Care must be taken not to assume that these addenda will be incorporated in the 2019 version; only time will tell how much, if any, of the language will be included in the final version.
However, some of the addenda items have a common theme. After a new standard is released to the industry for use on actual projects, parts of the standard will certainly need to be revised due to forces such as advancements in technology. Or some of the provisions didn’t result in what was anticipated. The introductions to the proposed addenda are very frank:
- Revision to ELC values: At the time Standard 90.4 was issued, values for ELC were based on electrical distribution equipment available in the marketplace. ELC was derived using manufacturer’s published efficiency data. These data were collected and analyzed three years before releasing Standard 90.4. In the subsequent years, electrical equipment technology, especially UPS systems, has improved at a significant rate. The primary gains in efficiency come from increased efficiency in UPS equipment and improved performance at part-load conditions.
- Elimination of design MLC: ASHRAE is proposing in the addenda to remove design MLC as a compliance option, as it was never meant to be a permanent indicator of energy efficiency or annual energy use. Plus, there are now readily available tools and techniques to calculate the annualized MLC.
- Raising the bar on MLC: ASHRAE admits that the requirements issued in Standard 90.4-2016 for MLC need to be more stringent and are “unnecessarily high.” They point out that other energy codes (such as California) have higher minimum energy-efficiency requirements.
- Still keeping MLC conservative: Interestingly, in the proposed addendum, the minimum compliance values are still conservative, according to ASHRAE. Considerably lower MLCs can be achieved using design strategies including airside economizers. while still adhering to ASHRAE’s recommended thermal guidelines.
- Controversial economizers: ASHRAE makes another candid assertion that the committee “did not want to tackle the use of air economizers for data centers.” The MLC values are achievable without using an air economizer; easily available cooling equipment and systems can meet the requirements.
- Change in how MLC is calculated: The 2016 version of Standard 90.4 requires that MLC be calculated at 100% and 50% ITE load. The results of those calculations must be less than the MLC value listed in Table 188.8.131.52. One of the addenda describes a new way of calculating MLC, based on 25%, 50%, 75% and 100% ITE load. This acknowledges that data centers generally run at less than 100% ITE load. The proposed mathematical process consisting of three new equations are shown in Figure 3.
Annualized MLC = (Mechanical-energy25% + Mechanical-energy50% + Mechanical-energy75% + Mechanical-energy100%) / (Data center ITE energy25% + Data center ITE energy50% + Data center ITE energy75% + Data center ITE energy100%)
Mechanical-energy (at X%) = Total annual cooling energy + pump energy + heat rejection fan energy + air handler fan energy at a constant ITE load of X% of the design ITE load
Data center ITE energy (at X%) = Design ITE load * 8,760 * X% (e.g., Data center ITE energy at % of design ITE load)
While this approach seems reasonable from a benchmarking perspective, actual operation will yield very different results. For example, servers and storage machines don’t have a linear heat output based on the computational output of the computer. So, when ASHRAE 90.4 asks for MLC calculations at different ITE loads, the engineer will have to assume that percent of ITE load is simply the total ITE power multiplied by 25%, 50%, 75% and 100%.
- Accounting for part-load performance of electrical systems: It might sound obvious, but when evaluating the MLC at part load, consideration must be given to the changing efficiency of the UPS and other segments of the electrical system. Certainly when the annual energy “consumption” of the electrical system is calculated, the efficiencies at part-load conditions will be lower, which adds to the annual kilowatt-hour. In the context of the MLC, there is an additional cooling load (if the UPS is located indoors) that must be calculated into the annual energy consumption of the mechanical system. The good news is that UPS manufacturers have made very good progress on improving UPS efficiency.
- Clarifying definition of small server rooms for easier compliance: For technology spaces in office buildings that are cooled via the house air conditioning system, such as intermediate distribution frame closets or small server rooms, there is a proposed alternate compliance path in the latest addenda. This approach simplifies compliance by reducing the effort of the user and authority having jurisdiction when it comes to creating and reviewing energy simulations, respectively. The goal with developing this proposed change is to improve the acceptance, use and enforcement of Standard 90.4.
When ASHRAE Standard 90.4-2016 was released, for some it represented a culmination of many years of research and analysis by many people in the data center industry. Certainly not all these people had direct input to the standard, but the groundwork they laid is integral to the development of the standard.
Now ASHRAE is on the cusp of releasing the 2019 version of the standard, which is based on three more years of analysis from data center energy use. The analysis includes information on data center operations and maintenance. The hands-on experience of running and maintaining highly efficient data centers, especially ones that are extremely innovative, is incredibly important to understanding the impacts of energy-efficiency measures.
From the proposed addenda, we have some insight into what changes will be included in the next release. One thing is for sure: high-performance, highly efficient data centers continue to be of great importance in the data center industry.