Data center design
CSE: Please describe a recent existing building retrofit you’ve worked on—share problems you’ve encountered, how you’ve solved them, and engineering aspects of the project you’re especially proud of.
Gillick: Our firm is in the process of commissioning a new UPS housed in an addition to a Tier IV data center. The project came about as a result of a catastrophic failure in the existing UPS. The data center had been running on emergency generator power. The root cause of the failure had not been identified. A site survey investigation determined that there had been a failure of a low-voltage fuse in a control circuit, which disabled the transfer back to utility power. The root cause was determined to be the low-voltage fuse failure, and a service restoration plan was developed. We conducted an MEP audit of the facility to determine if there were other single points of failure that might affect reliability. Our audit led to recommendations for commissioning and cutover from the legacy UPS system to the new UPS system. We provided peer reviews and commissioning of the new facility. Our team developed a plan to maintain operations for one year on the repaired UPS system during the retrofit project and a phasing plan to cut over to the new UPS in a live operating environment. We anticipate a successful completion of this project.
Dickens: Most of our firm’s projects are in the planning phase, and the conundrum continues to be determining what the future will look like and what infrastructure will be needed to support it. There is no answer to the first part of the question, because no one really knows what’s coming down the IT road. But the second challenge simply forces us to think about data centers, and their form factor in particular, in different ways. I’m proud of the fact that I don’t start a project thinking I know what that building will look like anymore.
Wesemann: We were recently involved with the upgrade of an existing health care data center, designed to 15-year-old standards, to meet the new capacity and energy efficiency standards that today’s data centers require. The challenges and solutions associated with upgrading older facilities include the following:
- Insufficient space to distribute mechanical piping and air. The facility had inadequate floor-to-floor heights to allow sufficient space for distribution. The existing 12-in. raised floor falls well short of being adequate for underfloor air distribution. While still in design, some considerations include moving to in-row air distribution, leaving the underfloor for only piping and electrical (data overhead). Another consideration was to remove the floor and ceiling altogether, leaving concrete floor and exposed structure with more space and accessibility for overhead mechanical and power distribution.
- Inadequate structural design: With greater capacity requirements and limited space for more, larger equipment, the roof became the best candidate for the equipment. However, the existing structural design limited the amount of equipment that could be placed on the roof. The original structural engineer was called back to “beef up” the structure to handle the equipment.
- Replace old equipment with new, while keeping the data center operational. The new system was designed to occupy limited but available space and become fully installed, tested, and commissioned before cutting over from old to new system. Unfortunately, the old system was not set up as an “A and B” redundant system, so the switchover will require a circuit-by-circuit sway of old power to new power. The new system will be A/B fully redundant to make future changes possible while reducing the risk of shutdowns.
- Energy efficiency requirements: Old UPS and mechanical equipment was installed in an era where little consideration was given to sustainable design. New UPS and mechanical equipment has been specified to meet today’s stricter efficiency expectations.
CSE: In a mission critical facility, what is the No. 1 challenge you deal with?
Lane: Data centers take an enormous amount of resources to operate. Energy efficiency of the electrical and mechanical systems is critical. Electrical meters can be used at the service entrance, at the mechanical equipment, and at the UPS systems to determine the amount of power used at various locations throughout the data center. One method used to determine the efficiency of the data center is PUE. PUE is essentially the amount of total power the data center consumed divided by actual power used by the servers. Older, less efficient data centers have PUEs above 2.0. Some very efficient data centers today can have PUEs in the 1.2 range. There are many design protocols that are used in today’s data center that have greatly increased data center efficiency. Some of these protocols include the following: more efficient UPS systems, outside air economizers, contained hot aisles, higher voltages, improved server efficiency, warmer cold isle temperatures, more efficient transformers, controls, variable frequency drives (VFDs), and lighting.
Gillick: Our firm is currently commissioning several of the military’s largest hospitals, so this is fresh in my mind. In a new hospital project, the most significant challenge is commissioning the central plant, which is typically constructed and commissioned before construction is completed on the hospital main building—that is, before there is an actual mechanical load or electrical connections to the hospital main and ancillary buildings. This creates significant challenges not only for phasing the commissioning process, but also for artificially loading the chillers, boiler plant, and electrical system. So we typically bring in compact trailer-mounted boilers to inject hot water into the cooling system to simulate chiller load and resistive load banks to simulate the load on utility services and the future 480- and 600-V electrical distribution system.
CSE: The U.S. Dept. of Energy (DOE) has launched an initiative to help increase the energy efficiency of data centers. Why is this a concern, and how have you dealt with it in your work?
Dickens: The concern is obvious because using more energy simply leads to more problems, including social, economic, and environmental. But as a mechanical engineer, using the least amount of energy to achieve our goals is simply part of the job description. So my approach to systems has not changed, but the relevance and importance of my objective is now shared by the rest of the design team. Because of U.S. Green Building Council LEED, Energy Star, and the U.S. Environmental Protection Agency initiatives, using less energy is not just the energy modeler’s problem anymore. It’s good to share.
Wesemann: Data centers are very large users of natural resources for energy and so any energy savings, even if just a few percentage points, will have a significant impact on overall energy conservation and reduction in the carbon footprint across the globe. It also makes good financial sense as the owner/operators of data centers reap the cost savings of energy-efficient measures.
Gillick: The DOE initiative has challenged engineers and manufacturers to develop more energy-efficient mechanical and electrical systems. On the HVAC side of the house, for example, many of the new computer room air-handling (CRAH) and computer room air conditioning (CRAC) systems comprise energy conservation measures such as variable frequency drives to control fan and pump speeds. On the electrical side of the house, manufacturers are pushing UPS and UPS distribution systems from 95% efficiency to 99% efficiency or greater. These advances have not changed the way engineers are designing the data centers, but they are changing the choices engineers have when designing electrical and mechanical systems and the ability to incorporate more DOE energy initiatives. Similarly, it has become very difficult to meet the federal, state, and local emissions standards and requirements for diesel generator emissions. As a result, we go to great lengths during the commissioning process to validate the emergency generator emissions compliance prior to regulatory testing and the energy efficiency of the products that are being put in place, and to validate that the engineer’s projected PUE meets the owner’s requirements.
Kosik: The DOE has had specialized programs to help industry lower energy consumption in buildings. The more standardization and guidance that is available to the data center community, the better. Many of the DOE programs have rebate incentives through the local utility when lowering energy use.
CSE: Are your data center clients requiring redundancy for all engineered systems?
Gillick: The industry has widely accepted the tier standards of mission critical facilities. Critical facilities are designed to meet Tier I through Tier IV guidelines. Our job as commissioning agents is to validate that the center meets the design standard. At almost every level, some redundancy is required for all engineered systems in data centers—increasing from little or no redundancy in a Tier I data center to 2N+ electrical and mechanical systems in a Tier IV data center.
Wesemann: Depending on the client and application, clients require varying degrees of redundancy. We arrive at a consensus with the client after discussion(s) of the risks and costs of redundancy in the various MEP systems. Another consideration is if the client has redundant backup sites, in which case the reliability requirements of any one site are less stringent since the entire data center itself is backed by another equal copy, which is located perhaps hundreds or thousands of miles away. If the client relies on one and only one data center, then redundancy is much more of a concern. Today we are seeing clients accept a minimal level of N+1 redundancy, while oftentimes going to “2N” or “system + system” redundancy at least in the UPS systems (even though other systems and components may still be N+1).
Lane: We are seeing varying levels of redundancy in modern data centers; 10 to 15 years ago, we would see enormous data centers built to the same redundancy level and the same power density throughout the entire facility. Today we are seeing single Tier II data centers with minimum redundancy for portions of the critical loads as well as Tier IV for other portions of loads. The redundancy level depends on the specific function of the computing task. Very critical loads will be built with 2N topology, while less critical loads will be built with N or N+1 topology. These loads could be in the same room. Additionally, we are building more data centers in a modular fashion—only building the power density required today, but providing for future expansion. This includes provisions for additional UPS modules, standby generators, chillers, and pumps.
Kosik: Yes, but it is done in a much more sophisticated way, which results in lower capital costs and uses modularity to increase future flexibility. Reliability is still paramount, but over the past five years or so, the industry has been focusing much more heavily on the effects of reliability on energy use. With that said, there is a segment of the industry that does not need high reliability in the power and cooling system. Reliability is achieved by shifting to other data centers that are also running the same workloads. In the supercomputing industry, typically only the data storage machines will be on UPS power and have some type of redundant cooling.
Dickens: Not necessarily, although the issue of concurrent maintainability is almost always a given. If we can achieve that with a thermal storage tank instead of an extra chiller, then we do. If there is a sufficient risk tolerance in place to allow a creatively valved and headered piping system to supersede the need of another pump, then we may do it. The key is in understanding the allowable level of risk and then designing to manage to that level. With that said, major equipment is usually redundant and if there are compromises, they are made in the less vulnerable distribution systems.