Designing with liquid-immersion cooling systems

Liquid cooling is an option in some data centers. Consider these best practices when looking at immersion cooling for your next data center project.


Learning objectives:

  • Discover the various ways to cool data center equipment via liquid cooling.
  • Outline the different options available for liquid-immersion cooling.
  • Measure the pros and cons for using liquid cooling in a data center environment.

In simple thermodynamic terms, heat transfer is the exchange of thermal energy from a system at a high temperature to one at lower temperature. In a data center, the information technology equipment (ITE) is the system at the higher temperature. The objective is to maintain the ITE at an acceptable temperature by transferring thermal energy in the most effective and efficient way, usually by expending the least amount of mechanical work.

Heat transfer is a complex process and the rate and effectiveness depends on a multitude of factors. The properties of the cooling medium (i.e., the lower-temperature system) are pivotal, as they directly impact flow rate, the resultant temperature differential between the two systems and the mechanical work requirement.

The rate at which thermal energy is generated by the ITE is characteristic of the hardware (central processing units, graphics processing units, etc.) and the software it is running. During steady-state operation, the thermal energy generated equals the rate at which it is transferred to the cooling medium flowing through its internal components. The flow rate requirement and the temperature envelope of the cooling medium is driven by the peak rate of thermal energy generated and the acceptable temperature internal to the ITE.

The flow rate requirement has a direct bearing on the mechanical work expended at the cooling medium circulation machine (pump or fan). The shaft work for a reversible, steady-state process with negligible change in kinetic or potential energy is equal to ∫vdP, where v is the specific volume and P is the pressure. While the pump and fan processes are nonideal, they follow the same general trend.

For data centers, air-cooling systems have been de facto. From the perspective of ITE, air cooling refers to the scenario where air must be supplied to the ITE for cooling. As the airflow requirement increases due to an increase in load, there is a corresponding increase in fan energy at two levels: the air distribution level (i.e., mechanical infrastructure such as air handling units, computer room air handlers, etc.) and the equipment level, because ITE has integral fans for air circulation.

Strategies including aisle containment, cabinet chimneys, and in-row cooling units help improve effectiveness and satisfactorily cool the equipment. However, the fact remains that air has inferior thermal properties and its abilities are getting stretched to the limit as cabinet loads continue to increase with time. For loads typically exceeding 15 kW/cabinet, alternative cooling strategies, such as liquid cooling, have become worthy of consideration.

Figure 1: A flow diagram for an open-immersion cooling configuration is shown. The ITE is immersed in a liquid bath open to the atmosphere. Courtesy: Environmental Systems Design Inc.

The case for liquid cooling

Liquid cooling refers to a scenario where liquid (or coolant) must be supplied to the ITE. An IT cabinet is considered to be liquid-cooled if liquid, such as water, dielectric fluid, mineral oil, or refrigerant, is circulated to and from the cabinet or cabinet-mounted equipment for cooling. Several configurations are possible, depending on the boundary being considered (i.e., external or internal to the cabinet). For the same heat-transfer rate, the flow rate requirement for a liquid and the energy consumed by the pump are typically much lower than the flow rate requirement for air and the energy consumed by the fan system. This is primarily because the specific volume of a liquid is significantly lower than that of air.

For extreme load densities typically in excess of 50 to 75 kW/cabinet, the liquid should preferably be in direct contact with ITE internal components to transfer thermal energy effectively and maintain an acceptable internal temperature. This type of deployment is called liquid-immersion cooling and it is at the extreme end of the liquid cooling spectrum. Occasionally referred to as "chip-level cooling," the commercially available solutions can essentially be categorized into two configurations:

  1. Open/semi-open immersion. In this type of system, the ITE is immersed in a bath of liquid, such as dielectric fluid or mineral oil. The heat-transfer mechanism is vaporization, natural convection, forced convection, or a combination of vaporization and convection (see Figure 1).
  2. Sealed immersion. In this type of system, the ITE is sealed in liquid-tight enclosures and liquid, such as refrigerant, dielectric fluid, or mineral oil, is pumped through the enclosure. The heat-transfer mechanism is vaporization or forced convection, and the enclosure is typically under positive pressure (see Figure 2).

For both types of systems, thermal energy can be transferred to the ambient by means of fluid coolers (dry or evaporative) or a condenser. It can also be transferred to facility water (chilled water, low-temperature hot water, or condenser water) by means of a heat exchanger.

A number of proprietary solutions are available for immersion cooling, and most providers can retrofit off-the-shelf ITE to make them compatible with their technology. Some technology providers are capable of providing turnkey solutions and require limited to no involvement of the consulting engineer.

Figure 2: This flow diagram shows a sealed immersion cooling configuration. The ITE is enclosed in liquid-tight enclosures typically under positive pressure. Courtesy: Environmental Systems Design Inc.

Others provide products as "kit of parts" and rely on the consulting engineer to design the associated infrastructure. For the latter, collaboration between the design team and the cooling technology provider is critical to project success. The design responsibilities should be identified and delineated early in the project. Note that a comprehensive guide for designing liquid cooling systems is beyond the scope of this article.

Once the total ITE load (in kilowatts) and load density (kilowatt/cabinet) have been defined by the stakeholders, the criteria can be used in conjunction with the design liquid-supply temperature and anticipated delta T across the ITE, to determine the flow rate requirement and the operating-temperature envelope. Recommendations for a liquid-supply temperature and anticipated delta T are typically provided by the technology provider, and empirical data is preferred over theoretical assumptions. For example, a flow rate requirement of 1 gpm/kW, liquid-supply temperature of 104° F, and anticipated delta T of 10° F was used as the basis of design when deploying a specific technology. Requirements can vary significantly between different providers.

<< First < Previous 1 2 Next > Last >>

No comments
Consulting-Specifying Engineer's Product of the Year (POY) contest is the premier award for new products in the HVAC, fire, electrical, and...
Consulting-Specifying Engineer magazine is dedicated to encouraging and recognizing the most talented young individuals...
The MEP Giants program lists the top mechanical, electrical, plumbing, and fire protection engineering firms in the United States.
40 Under 40; Performance-based design; Clean agent fire suppression; NFPA 92; Future of commissioning; Successful project management principles
BIM coordination; MEP projects; NFPA 13; Data center Q&A; Networked lighting controls; 2017 Product of the Year finalists
Emergency lighting; NFPA 3 and 4; Integrated building systems; Smart lighting, HVAC design
Commissioning electrical systems; Designing emergency and standby generator systems; VFDs in high-performance buildings
Tying a microgrid to the smart grid; Paralleling generator systems; Previewing NEC 2017 changes
Driving motor efficiency; Preventing Arc Flash in mission critical facilities; Integrating alternative power and existing electrical systems
As brand protection manager for Eaton’s Electrical Sector, Tom Grace oversees counterfeit awareness...
Amara Rozgus is chief editor and content manager of Consulting-Specifier Engineer magazine.
IEEE power industry experts bring their combined experience in the electrical power industry...
Michael Heinsdorf, P.E., LEED AP, CDT is an Engineering Specification Writer at ARCOM MasterSpec.
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me