Designing with liquid-immersion cooling systems
Liquid cooling is an option in some data centers. Consider these best practices when looking at immersion cooling for your next data center project.
- Discover the various ways to cool data center equipment via liquid cooling.
- Outline the different options available for liquid-immersion cooling.
- Measure the pros and cons for using liquid cooling in a data center environment.
In simple thermodynamic terms, heat transfer is the exchange of thermal energy from a system at a high temperature to one at lower temperature. In a data center, the information technology equipment (ITE) is the system at the higher temperature. The objective is to maintain the ITE at an acceptable temperature by transferring thermal energy in the most effective and efficient way, usually by expending the least amount of mechanical work.
Heat transfer is a complex process and the rate and effectiveness depends on a multitude of factors. The properties of the cooling medium (i.e., the lower-temperature system) are pivotal, as they directly impact flow rate, the resultant temperature differential between the two systems and the mechanical work requirement.
The rate at which thermal energy is generated by the ITE is characteristic of the hardware (central processing units, graphics processing units, etc.) and the software it is running. During steady-state operation, the thermal energy generated equals the rate at which it is transferred to the cooling medium flowing through its internal components. The flow rate requirement and the temperature envelope of the cooling medium is driven by the peak rate of thermal energy generated and the acceptable temperature internal to the ITE.
The flow rate requirement has a direct bearing on the mechanical work expended at the cooling medium circulation machine (pump or fan). The shaft work for a reversible, steady-state process with negligible change in kinetic or potential energy is equal to ∫vdP, where v is the specific volume and P is the pressure. While the pump and fan processes are nonideal, they follow the same general trend.
For data centers, air-cooling systems have been de facto. From the perspective of ITE, air cooling refers to the scenario where air must be supplied to the ITE for cooling. As the airflow requirement increases due to an increase in load, there is a corresponding increase in fan energy at two levels: the air distribution level (i.e., mechanical infrastructure such as air handling units, computer room air handlers, etc.) and the equipment level, because ITE has integral fans for air circulation.
Strategies including aisle containment, cabinet chimneys, and in-row cooling units help improve effectiveness and satisfactorily cool the equipment. However, the fact remains that air has inferior thermal properties and its abilities are getting stretched to the limit as cabinet loads continue to increase with time. For loads typically exceeding 15 kW/cabinet, alternative cooling strategies, such as liquid cooling, have become worthy of consideration.
The case for liquid cooling
Liquid cooling refers to a scenario where liquid (or coolant) must be supplied to the ITE. An IT cabinet is considered to be liquid-cooled if liquid, such as water, dielectric fluid, mineral oil, or refrigerant, is circulated to and from the cabinet or cabinet-mounted equipment for cooling. Several configurations are possible, depending on the boundary being considered (i.e., external or internal to the cabinet). For the same heat-transfer rate, the flow rate requirement for a liquid and the energy consumed by the pump are typically much lower than the flow rate requirement for air and the energy consumed by the fan system. This is primarily because the specific volume of a liquid is significantly lower than that of air.
For extreme load densities typically in excess of 50 to 75 kW/cabinet, the liquid should preferably be in direct contact with ITE internal components to transfer thermal energy effectively and maintain an acceptable internal temperature. This type of deployment is called liquid-immersion cooling and it is at the extreme end of the liquid cooling spectrum. Occasionally referred to as "chip-level cooling," the commercially available solutions can essentially be categorized into two configurations:
- Open/semi-open immersion. In this type of system, the ITE is immersed in a bath of liquid, such as dielectric fluid or mineral oil. The heat-transfer mechanism is vaporization, natural convection, forced convection, or a combination of vaporization and convection (see Figure 1).
- Sealed immersion. In this type of system, the ITE is sealed in liquid-tight enclosures and liquid, such as refrigerant, dielectric fluid, or mineral oil, is pumped through the enclosure. The heat-transfer mechanism is vaporization or forced convection, and the enclosure is typically under positive pressure (see Figure 2).
For both types of systems, thermal energy can be transferred to the ambient by means of fluid coolers (dry or evaporative) or a condenser. It can also be transferred to facility water (chilled water, low-temperature hot water, or condenser water) by means of a heat exchanger.
A number of proprietary solutions are available for immersion cooling, and most providers can retrofit off-the-shelf ITE to make them compatible with their technology. Some technology providers are capable of providing turnkey solutions and require limited to no involvement of the consulting engineer.
Others provide products as "kit of parts" and rely on the consulting engineer to design the associated infrastructure. For the latter, collaboration between the design team and the cooling technology provider is critical to project success. The design responsibilities should be identified and delineated early in the project. Note that a comprehensive guide for designing liquid cooling systems is beyond the scope of this article.
Once the total ITE load (in kilowatts) and load density (kilowatt/cabinet) have been defined by the stakeholders, the criteria can be used in conjunction with the design liquid-supply temperature and anticipated delta T across the ITE, to determine the flow rate requirement and the operating-temperature envelope. Recommendations for a liquid-supply temperature and anticipated delta T are typically provided by the technology provider, and empirical data is preferred over theoretical assumptions. For example, a flow rate requirement of 1 gpm/kW, liquid-supply temperature of 104° F, and anticipated delta T of 10° F was used as the basis of design when deploying a specific technology. Requirements can vary significantly between different providers.