Designing with liquid-immersion cooling systems

Liquid cooling is an option in some data centers. Consider these best practices when looking at immersion cooling for your next data center project.

By Saahil Tumber, PE, HBDP, LEED AP, Environmental Systems Design Inc., Chicago September 21, 2016

Learning objectives:

  • Discover the various ways to cool data center equipment via liquid cooling.
  • Outline the different options available for liquid-immersion cooling.
  • Measure the pros and cons for using liquid cooling in a data center environment.

In simple thermodynamic terms, heat transfer is the exchange of thermal energy from a system at a high temperature to one at lower temperature. In a data center, the information technology equipment (ITE) is the system at the higher temperature. The objective is to maintain the ITE at an acceptable temperature by transferring thermal energy in the most effective and efficient way, usually by expending the least amount of mechanical work.

Heat transfer is a complex process and the rate and effectiveness depends on a multitude of factors. The properties of the cooling medium (i.e., the lower-temperature system) are pivotal, as they directly impact flow rate, the resultant temperature differential between the two systems and the mechanical work requirement.

The rate at which thermal energy is generated by the ITE is characteristic of the hardware (central processing units, graphics processing units, etc.) and the software it is running. During steady-state operation, the thermal energy generated equals the rate at which it is transferred to the cooling medium flowing through its internal components. The flow rate requirement and the temperature envelope of the cooling medium is driven by the peak rate of thermal energy generated and the acceptable temperature internal to the ITE.

The flow rate requirement has a direct bearing on the mechanical work expended at the cooling medium circulation machine (pump or fan). The shaft work for a reversible, steady-state process with negligible change in kinetic or potential energy is equal to ∫vdP, where v is the specific volume and P is the pressure. While the pump and fan processes are nonideal, they follow the same general trend.

For data centers, air-cooling systems have been de facto. From the perspective of ITE, air cooling refers to the scenario where air must be supplied to the ITE for cooling. As the airflow requirement increases due to an increase in load, there is a corresponding increase in fan energy at two levels: the air distribution level (i.e., mechanical infrastructure such as air handling units, computer room air handlers, etc.) and the equipment level, because ITE has integral fans for air circulation.

Strategies including aisle containment, cabinet chimneys, and in-row cooling units help improve effectiveness and satisfactorily cool the equipment. However, the fact remains that air has inferior thermal properties and its abilities are getting stretched to the limit as cabinet loads continue to increase with time. For loads typically exceeding 15 kW/cabinet, alternative cooling strategies, such as liquid cooling, have become worthy of consideration.

The case for liquid cooling

Liquid cooling refers to a scenario where liquid (or coolant) must be supplied to the ITE. An IT cabinet is considered to be liquid-cooled if liquid, such as water, dielectric fluid, mineral oil, or refrigerant, is circulated to and from the cabinet or cabinet-mounted equipment for cooling. Several configurations are possible, depending on the boundary being considered (i.e., external or internal to the cabinet). For the same heat-transfer rate, the flow rate requirement for a liquid and the energy consumed by the pump are typically much lower than the flow rate requirement for air and the energy consumed by the fan system. This is primarily because the specific volume of a liquid is significantly lower than that of air.

For extreme load densities typically in excess of 50 to 75 kW/cabinet, the liquid should preferably be in direct contact with ITE internal components to transfer thermal energy effectively and maintain an acceptable internal temperature. This type of deployment is called liquid-immersion cooling and it is at the extreme end of the liquid cooling spectrum. Occasionally referred to as "chip-level cooling," the commercially available solutions can essentially be categorized into two configurations:

  1. Open/semi-open immersion. In this type of system, the ITE is immersed in a bath of liquid, such as dielectric fluid or mineral oil. The heat-transfer mechanism is vaporization, natural convection, forced convection, or a combination of vaporization and convection (see Figure 1).
  2. Sealed immersion. In this type of system, the ITE is sealed in liquid-tight enclosures and liquid, such as refrigerant, dielectric fluid, or mineral oil, is pumped through the enclosure. The heat-transfer mechanism is vaporization or forced convection, and the enclosure is typically under positive pressure (see Figure 2).

For both types of systems, thermal energy can be transferred to the ambient by means of fluid coolers (dry or evaporative) or a condenser. It can also be transferred to facility water (chilled water, low-temperature hot water, or condenser water) by means of a heat exchanger.

A number of proprietary solutions are available for immersion cooling, and most providers can retrofit off-the-shelf ITE to make them compatible with their technology. Some technology providers are capable of providing turnkey solutions and require limited to no involvement of the consulting engineer.

Others provide products as "kit of parts" and rely on the consulting engineer to design the associated infrastructure. For the latter, collaboration between the design team and the cooling technology provider is critical to project success. The design responsibilities should be identified and delineated early in the project. Note that a comprehensive guide for designing liquid cooling systems is beyond the scope of this article.

Once the total ITE load (in kilowatts) and load density (kilowatt/cabinet) have been defined by the stakeholders, the criteria can be used in conjunction with the design liquid-supply temperature and anticipated delta T across the ITE, to determine the flow rate requirement and the operating-temperature envelope. Recommendations for a liquid-supply temperature and anticipated delta T are typically provided by the technology provider, and empirical data is preferred over theoretical assumptions. For example, a flow rate requirement of 1 gpm/kW, liquid-supply temperature of 104° F, and anticipated delta T of 10° F was used as the basis of design when deploying a specific technology. Requirements can vary significantly between different providers.

Selecting a liquid

The liquid properties impact major facets of the design and should be reviewed in detail. The mechanical industry is accustomed to working with typical liquids, such as water, glycol solutions, and refrigerants; deviations associated with unique liquids can create challenges. Properties such as kinematic viscosity, dynamic viscosity, specific heat, density, thermal conductivity, the coefficient of thermal expansion, and heat capacity can influence the design. Because the liquid is typically proprietary, the properties are not available in standard design guides or catalogs and are provided by the cooling technology provider.

Liquid properties have a direct bearing on heat-transfer characteristics and greatly impact the selection of heat-transfer equipment, such as coils, heat exchangers, and fluid coolers. As discussed, standard catalog data cannot be used for this purpose. However, major manufacturers are capable of providing estimated performance for unique liquids.

System pressure drop calculations can also be challenging. One option is to use the underlying principles of fluid mechanics. For example, the Darcy-Weisbach equation can be used to estimate the pressure drop through pipes when circulating Newtonian fluids. For sealed-immersion applications, the pressure drop through the ITE enclosure is typically supplied by the technology provider. When selecting pumps for liquid circulation, properties like density and viscosity will impact the brake horsepower, head, flow capacity, and efficiency of the pump. Major manufacturers can provide pump performance when circulating unique liquids. Another option is to estimate performance by using correction factors, equations, or charts developed by organizations, such as the Hydraulic Institute, and applying them to standard pump curves developed for water.

When the loop design is being established, the compatibility of materials that are expected to be in direct contact with the liquid should be reviewed. For example, pump seals and valve seats are frequently constructed of ethylene propylene diene monomer (EPDM), a type of synthetic rubber. However, EPDM is not compatible with petroleum-based liquids.

Similarly, the use of inert pipe materials, such as stainless steel and copper, and use of mechanical joints in lieu of welding, soldering, or brazing should be discussed with the technology provider. Any contamination of liquid due to incompatibility with materials of construction can have serious repercussions and can lead to catastrophic failure of the ITE.

Requirements for fluid maintenance should be discussed with the technology provider. Petroleum-based liquids, such as mineral oil, are susceptible to biological and water contamination over time. Liquid degradation can negatively impact the heat-transfer properties and can lead to premature failure of the ITE. The infrastructure should incorporate suitable means for fluid maintenance if deemed necessary.

For sealed immersion configurations, pressure limitations of the ITE enclosures must be considered. For a particular application, the pressure rating was less than 10 psig. The requirement impacts the elevation of mechanical infrastructure relative to the ITE, as the static head imposed on ITE needs to be kept to a minimum. Similarly, the pressure drop through the circulation loop, hence the pump head requirement, should be minimized. Pressure-relief valves or other means should be incorporated to prevent accidental overpressurization.

The right solution?

When dealing with extremely dense cabinets, immersion cooling is worthy of consideration. It is suitable for deployments ranging from a few kilowatts to several megawatts. Due to improved heat-transfer performance as compared with an air-cooling system, liquid-supply temperatures higher than 100° F are feasible. Higher liquid temperatures increase the hours of economization, offer the potential for heat recovery, and in certain climates can eliminate the need for chillers completely. The elimination of internal ITE fans reduces energy consumption and noise. In addition, pump energy for circulating liquid is typically lower than fan energy.

Despite the mechanical advantages, there are reasons for caution when deploying liquid-immersion cooling in data centers. The impact on infrastructure, such as structural, electrical, fire protection, and structured cabling, should be evaluated. In a typical data center, air-cooling systems are still needed as certain ITE, such as spinning drives, cannot be liquid-cooled. Immersion cooling is still in its nascent stage, and long-term statistical data is needed for detailed evaluation of ITE and infrastructure reliability, serviceability, maintainability, and lifecycle costs.

Saahil Tumber is a senior associate and lead mechanical engineer at Environmental Systems Design Inc., and is responsible for the overall design of HVAC systems for data centers, trading areas, and other mission critical facilities requiring high availability. His data center experience spans both enterprise and colocation projects.