Driving data center design
In the information age, data centers can be the beating heart of not just a building, but an entire global corporation. Engineers with experience working on data centers offer advice on their complex design and getting all the various aspects to compute.
- Andrew Baxter, PE, Principal/MEP Engineering Director, Page, Austin, Texas
- Brandon Kingsley, PE, CxA, CEM Project Manager, Primary Integration Solutions Inc., Charlotte, N.C.
- Keith Lane, PE, RCDD, NTS, RTPM, LC, LEED AP BD+C, President/Chief Engineer, Lane Coburn & Associates LLC, Seattle
- Dwayne Miller, PE, RCDD, CEO, JBA Consulting Engineers, Hong Kong
CSE: Please describe a recent data center project you've worked on-share details about the project, including building location, size, owner's project requirements (OPR), etc.
Andrew Baxter: Page recently designed a Tier III data center for a confidential Fortune 100 company located in the Chicago metropolitan area. This is one of the most efficient data centers in the transportation sector, supporting the client's commitment to the environment. It was built to withstand severe weather conditions without compromising the integrity or security of its cooling system, which is anticipated to achieve an annual average power usage effectiveness (PUE) of 1.09. The new data center has been designed to achieve energy savings of approximately 50% above the required efficiency standards with state-of-the-art economizer systems for cooling critical electrical rooms and air-handling units, an energy recovery make-up air handling unit for ventilation, high-efficiency condensing boilers for heating, and highly efficient LED lighting. It is based on N+2 1000 kW units, and Phase 1 was designed to be a 4 MW information technology (IT) fit-up with a 12 MW total facility load at full build-out. Phase 1 of the 308,000-sq-ft project includes 25,000 sq ft of white space in an 180,000-sq-ft building. This initial phase includes the company's backup emergency operations center (EOC) in which it will control its entire worldwide operations in the event that the main operations center is ever down. The EOC contains conference rooms and 50 workstations for various user groups, which are focused on a video wall and a control room overlook for supervision. It is anticipated that approximately 85% of the facility will be free cooled.
Brandon Kingsley: Primary Integration is currently involved in commissioning a 128 MW cloud data center that is being deployed in four sites. The intent of the OPR is to design more reliability into the IT network and equipment and less into the mechanical, electrical, plumbing (MEP), and fire protection systems. The design consists of multiple buildings, which have hot aisle containment but no raised floors and no mechanical cooling. Cold aisle allowable operating temperatures can be as high as 90 F. Instead of designing the mechanical systems based on the IT equipment, the IT equipment was designed and selected in conjunction with the mechanical systems to operate within the mechanical system parameters. The biggest commissioning challenge has been staffing and scheduling to test the sheer quantity of mechanical and electrical equipment on each site, including 160 air handlers and 21 generators; however, because of the inherent simplicity of the mechanical and electrical system designs, we are not dealing with large central chiller plants that have complex control sequences.
Keith Lane: Lane Coburn & Associates has worked closely with Silent-Aire for more than 5 years enhancing the design of modular data center deployments around the country. There are numerous challenges and numerous benefits to the design, construction, and deployment of modular data centers. Modular data centers are designed and built as a complete system. The entire mechanical and electrical system is built around the client's IT infrastructure needs and requirements. Modular off-site construction, as opposed to the conventional brick-and-mortar data center, delivers speed, performance, and cost containment. Building off-site in a controlled, safe, and environmentally friendly space may quite often allow for much quicker deployment. In addition to the time savings in building the mechanical, electrical, and structural components, all components and systems are tested in the factory before shipping to the site. This saves time and money during the final integrated systems testing (IST) before handover to the client in the field. Optimal performance is more quickly achieved from a modular data center versus a solution built from scratch. The design specifications are tested and verified before the unit ships, thereby delivering immediate quality assurance. There are several advantages to building offsite, but the main reasons are cost control and speed to market. Delays related to weather, site conditions, unreliable or inconsistent labor forces, and labor inefficiencies are greatly reduced or eliminated in a warehouse/prefabrication environment. This leads to reduction in cost and schedule. Additionally, estimating the total cost of the project is typically more accurate. In the end, this represents less risk to the end user. Modular data centers that are factory built can come with either a UL or ETL safety certification label that certifies they have been factory tested and meet the required electrical safety codes and requirements. This safety certification allows the modular data centers to be classified as equipment and not modular buildings, which may often circumvent permitting and inspection requirements that would normally be demanded by the authorities having jurisdiction (AHJ) in a brick-and-mortar build, thus allowing for aggressive and expedient deployments. Other benefits to the modular data centers being classified as equipment are: they can be depreciated as equipment, and opportunities for leasing or financing of the modular data centers exist as well.
Dwayne Miller: Our most recent data center projects have been enterprise data centers for international integrated resorts. The properties are both in excess of 4 million sq ft. Both data centers tie into property infrastructure; hence the cooling, normal power, and generator back-up power are served from centralized systems. Owner requirements included on-site disaster recovery capabilities, which are addressed with primary and secondary data centers for each property. The centralized generator backup system is composed of multiple parallel engines, and the data center loads are second only to life safety systems with respect to the load priority. In addition, from a cooling standpoint, the data center is tied in to a large centralized chilled water system and is the highest priority load for the system. A combination of centralized and localized infrastructure is deployed to ensure continuity of services.
CSE: What are the newest trends in data centers in mixed-use buildings?
Kingsley: This really depends on the rack density, required reliability, and available utilities. For example, a research-based data center at a college or university may have a high rack density and high-reliability requirement. As a result, MEP designers tend to use a high-density cooling solution such as an in-row cooling. An independent cooling system may also be used rather than relying on the central chiller plant, which may be shut down in winter. Increasingly, we are seeing heat recovery systems being used in mixed-use buildings to recover waste heat from the data center and use it for the building heating system. This may make the most economic sense as an energy savings strategy in a building with a large white space data center and office space that represents a fraction of the overall cost.
Lane: Flexibility and modularity are the key features clients require in the market today. It is critical to design flexibility to modify the design for future phases and to ensure the infrastructure is in place to provide for changes. Modularity is critical to ensure incremental components can be added as density and/or redundancy increases are required. Baxter: The biggest trend is probably moving the data centers completely out of these types of buildings and having "purpose built" facilities. Heat recovery would be a newer trend, especially in regions where extended heating periods allow the heat generated by the data center to be used for building heating.
CSE: What are some challenges you have faced in coordinating structural systems with mechanical, electrical, plumbing, or fire protection systems?
Lane: Data centers are very unique facilities. The sheer amount of power and critical nature of the loads being served require significant expertise. Uninterruptable power supplies, large standby generators, fuel supplies, large conductors, medium-voltage services, large transformers, various voltages, harmonic distortion, metering, PUE, and energy efficiency all must be considered in the design of data center facilities. Because of the unique nature of the electrical load profile, the heating of underground electrical duct banks must be evaluated. This involves 3-D modeling of the underground feeders as well as a comprehensive failure mode analysis and Neher-McGrath heating calculations. The initial cost of building a data center is tremendous. The long-term costs associated with running a data center include the electrical and water services, which are very significant and must be considered during the design process. The electrical and mechanical engineers must work collaboratively to ensure the most reliable and cost-effective systems are designed and implemented. Enough design time must be built into the schedule to ensure value engineering ideas are fully vetted. Additionally, comprehensive commissioning of the data center should be provided by a third party to ensure all components of the MEP system work independently and as a system prior to actually serving critical loads.
Baxter: Coordinating all systems together-not just the MEP and structural-can be quite a challenge for these types of facilities. The structure can be very deep to carry the higher than normal weight densities a data center can impart to a structure, especially in multi-level facilities. This, along with the large number of cable trays, electrical raceways, mechanical systems, etc., can create significant space management challenges. Add in specific project requirements such as seismic restraint, excess wind loading capabilities, or electromagnetic pulse (EMP) shielding, and final coordination becomes critical.
Kingsley: Coordination with structural systems is especially challenging in existing buildings. When a data center is added in an existing facility, the MEP systems have to be designed around fixed existing structural systems. The available floor-to-floor height may dictate the types of systems that can be installed, such as in-row cooling instead of a more conventional system using raised floors and ceiling return plenums. As commissioning agents, part of our job is to make sure that all of the MEP equipment is accessible and maintainable. Without good coordination among all systems, we may find that a cable tray, for example, is inaccessible. The increasing use of BIM in system design, engineering, and construction is improving coordination and reducing these types of problems.