‘Cool’ Tower Dominates Heat Horizon

High Delta-T cooling, an innovative technology utilizing standard temperature air to cool electronic equipment, increases energy efficiency and reliability for data centers and other electronic-intensive facilities.

By R. Stephen Sinazzola, P.E., Vice President, and Zubin Menachery, P.E., Associate, RTKL Associates, Baltimore January 1, 2002

Despite a continuous engineering effort to save energy at almost all levels with all M/E/P systems, server-room cooling systems have remained virtually unchanged since the inception of the raised floor. And in the wake of ever increasing power- and heat-load densities in today’s data centers, traditional methods simply can no longer keep pace. Indeed, anecdotal evidence shows that conventional data-center cooling designs become ineffective as power density exceeds 150 watts/sq. ft.–approximately 3.8 kilowatts (kW) per computer equipment rack.

The lack of advancement, of course, stems from the fact that many data-center operators simply don’t want to sacrifice reliability for energy savings. With this in mind, RTKL Associates, in partnership with APW-Wrightline, recently developed a patent-pending technology, keyed by a process known as high Delta-T cooling (HDTC), that raises the effectiveness of air-conditioning systems for these facilities without sacrificing reliability.

How it works

The problem was tackled literally at the ground floor, with the solution surfacing in the form of a custom computer rack that conveys cool air directly to electronic equipment via specially designed doors. Dubbed the “Tower of Cool” (TOC), the rack unit is an integral part of the cooling system itself. Cooling air enters directly through the bottom of the rack via supply fans that are integral to a specially designed vertical panel in the front door (see Figure 1, p. 37). Air is distributed across electronic equipment, picking up heat and exhausting it through a specially designed back panel. Return air discharges through integral exhaust fans directly to a ceiling plenum, where it is conveyed to computer-room air-conditioning (CRAC) units, in essence doubling the effectiveness of air-conditioning equipment. This efficiency is the result of using standard temperature air to cool the electronic equipment while returning higher temperature air to the CRAC unit. By employing HDTC, heat transfer across the CRAC unit’s cooling coil can be raised to 40oF, based on a nominal supply temperature of 55oF and a return temperature of 95oF.

This differs from conventional data center air-conditioning design, in which 55oF air is delivered to a space to mix with the 110oF air typically discharged from electronic equipment. Such a mix usually produces 70oF air to cool equipment, and the air returns to the CRAC units at approximately 75oF, meaning heat transfer across the CRAC unit’s cooling coil is approximately 20oF.

Thus, with each CRAC unit providing twice the cooling heat transfer for the same airflow, the number of CRAC units in a particular facility can be cut in half. This leads to significant equipment and energy reductions–as much as 16% of the operating costs for a data center’s cooling plant, and as much as 7% in overall construction costs. Specifically, savings result from a smaller building footprint, less raised-floor area, and smaller central-plant-equipment sizing and distribution systems, including emergency and normal power distribution systems.

Note, the system is based on units utilizing chilled-water cooling coils and a down-flow configuration that discharges the cooling air into a raised floor. Additionally, the capacity of the chilled-water coil was doubled by employing a control valve change to increase water flow. Rack description

The key to the process is the TOC’s front door. Specifically, it features an air chamber with an opening in the bottom and a perforated panel on the front that conveys cooling air into the equipment compartment. This opening aligns with a set of fans that move cooling air from below the raised-floor directly into the air chamber in the front door. The chamber converts the vertical airflow into a uniform horizontal stream that flows through the equipment compartment.

The back door of the rack also has an air chamber, only its opening is at the top. The rear air chamber opening also aligns with a set of fans that draw warm air from the equipment out of the rack. A perforated panel draws air evenly from the equipment compartment, top to bottom. Warm air from the computer rack is delivered to the CRAC unit in the ceiling, where the isolated hot air discharged from the equipment ensures a higher heat transfer across the coil. Acid test

To verify the efficacy of the HDTC process, the TOC was loaded with about 7.4 kW of servers and fitted with several thermistor temperature sensors to monitor entering supply-air temperature, leaving exhaust-air temperature and the temperature in the back door air cavity at three elevations (see Figure 2). The readings were recorded by a BACnet controller that logged the data and converted it to a database file.

Airflow from the supply fans was verified by measuring velocity through the opening in the floor, and an ammeter was used to verify current drawn by the electronic equipment. This complement of temperature sensors and instruments effectively monitored the parameters of heat transfer through the computer rack.

With the assistance of data-center operators, six tests were developed to simulate failure modes and other issues specific to this concept. Test 1 measured “steady-state” performance, simulating the undisturbed operation of the TOC over the long term. Some 41 servers were run overnight for about 20 hours while data was logged; ammeter readings verified that the servers were drawing significant current.

Test 2 simulated failure of the middle supply and exhaust fan on the rack. The fans were disabled and the servers were then operated to verify that such a fan failure would not compromise the operating environment of the servers.

Because the front door is critical to the effective operation of the TOC, Test 3F was conducted to demonstrate that having the front door open for a period of time would not appreciably reduce the TOC’s effectiveness. For the test, the front door was opened and temperatures were recorded every 10 seconds for about six minutes. The back door was tested in the same fashion (Test 3B).

Test 4 was designed to assess the effect of the failure of one chiller in a bank of two or three that provide chilled water to the CRAC units that cool the data center. If a chiller fails, supply temperature from the CRAC units should rise 5 to 10 degrees, so supply air temperature was raised to about an average of 65oF for an hour.

Test 5 simulated a power failure, producing a CRAC unit shutdown. Because of the need to protect other equipment in the room, researchers were not permitted to disable the CRAC unit. In lieu of this, power to the supply and exhaust fans on the TOC was cut to create similar conditions. The test was conducted for approximately three minutes–the time required for generators to start and for power to be restored. Temperatures were recorded every 10 seconds.

The final test, No. 6, involved displacing raised-floor tiles in the vicinity of the TOC to simulate work being performed under the floor. This has the potential to significantly reduce the air pressure available for delivering cold air into the rack. One floor tile adjacent to the TOC and another in front of the TOC were removed for about 17 minutes, and temperature readings were recorded every 10 seconds. Test results

The test results confirmed that the HDTC concept performs under almost all conditions. The steady-state test, where temperature readings were taken at the supply and exhaust ports as well as at various locations in the back door air cavity, indicated that even with 65oF supply-air temperature from below the floor delivered to the servers in the TOC, equipment can effectively be cooled indefinitely. Arguably, this is a relatively high supply-air temperature for cooling a data center–large-scale, high-density facilities usually see a supply-air temperature of about 55oF. The tests, however, were conducted in a low power-density testing lab, where 65oF was adequate. But even at this elevated temperature, the highest temperatures recorded at the back door did not exceed 90oF. This is notable, because discharge temperatures from a similarly equipped conventional computer rack typically exceed 95oF to 100oF, even with additional airflow to the rack.

Test 2 showed that failure of one supply and one exhaust fan of the TOC does not appreciably compromise the operating environment (see Figure 3). The substantial drop in temperatures recorded in the back door was the result of the door being open briefly during the test.

The data from Test 3F showed that even with the front door open for about six minutes, the temperatures in the TOC did not rise significantly. Test 3B–operating the TOC with its back door open–was intended to show that the servers could continue to receive cold air and operate nominally. As the temperature sensors were in the back door, the data showed that upon opening the door, temperatures in the sensor compartment approached room temperature and returned to their steady state values when the door closed (see Figure 4, p.38).

In Test 4, where the supply-air temperature was raised to a maximum of 67oF, data demonstrated the elevation in temperature that might result from a chiller failure would not interfere with continued operation.

Test 5, which simulated power loss to CRAC units, showed a rise in the temperatures in the back door and at the exhaust fan (see Figure 5, p. 38). While not an exact analog to the conditions that would result from an air-handler failure, the test was as close an approximation as could be conducted without shutting down the CRAC unit. In fact, the test conditions were more extreme than a power failure to the air handlers, as fans on the TOC, which is backed by an uninterruptible power supply, would continue to operate.

The minor temperature rise noted would have been greater if the supply-air temperature had remained constant or rose instead of dropping, as shown by the data for the line representing “Cabinet 1, Supply” in Figure 5. In a data center the temperatures would not rise so rapidly. Furthermore, since these air handlers are typically on generator power, the units would restart within one minute, instead of the three-minute period that was tested.

Finally, Test 6, simulating work being performed around computer racks , found no adverse effect from removing raised floor tiles near the TOC. Temperatures, in fact, actually trended slightly lower. Test data translations

The conducted tests verify that the HDTC concept is effective in cooling approximately 7.4 kW of electronic equipment in one rack, and that the TOC can operate without adverse effects through the normal day-to-day operations of an active data center.

This, of course, begs the question as to how such testing translates into real-world numbers and benefits. With that in mind, a cost model was developed based on a server-farm prototype that RTKL designed for a large telecommunications company in 1999 (see table above).

For the purpose of this analysis, two circuits per cabinet–approximately 191 watts/sq. ft.–were used. This is an increase of 38% above the base prototype, reflecting the increased density HDTC should allow.

As is the case with any type of formal comparison, there must be at least one constant between the base and the alternate. In this analysis, the size of the electrical service for the electronic equipment is the constant, because electrical service is the highest cost element of the project. Given that the associated quantity of circuits with the service is also constant, the variables are the size of the building, the number of cabinets and the size of the cooling plant.

With the electrical service fixed at 6,600 circuits, and using two circuits per cabinet in the HDTC prototype, the cabinet count is reduced by more than 1,500 from the base prototype, meaning a smaller volume of raised floor. Additionally, the HDTC prototype reduces the quantity of CRAC units from 122 to 61. (Each CRAC unit has a nominal 10-hp motor operating on normal and emergency power). This not only reduces the cost of the mechanical system, but also reduces the cost of the electrical system by approximately 450 kW.

Furthermore, the reduction of CRAC units reduces the size of the chiller plant by a reduction in fan motor heat. This translates to a power requirement reduction of approximately 360 kW. Combined, these factors result in a total power reduction of 810 kW. The bottom line is that the TOC concept should allow data center owners and operators to confidently install more electronic equipment per rack as they will be cooled more effectively and efficiently.

HDTC Construction Cost Analysis* $=H007Base prototype specifications

HDTC prototype specs

Gross building area: 127,500 square feet

Gross building: 105,000 square feet

Raised-floor area: 91,700 square feet

Raised floor: 66,400 square feet

Equipment load: 12.7 megawatts (6,600 circuts, approximately 138 watts per square foot)

Equipment load: 12.7 megawatts (6,600 circuits, approximately 191 watts per square foot)

Cooling plant: 5,140 tons (5.0 megawatts connected electircal load)

Cooling plant: 4,620 tons (4.3 megawatts connected electrical load)

Airside cooling: 122 CRACUs on the raised floor

Airside cooling: 61 CRAC units units on the raised floor

4,250 cabinets (approximately) with 1.5 circuits per cabinet

3,150 cabinets (approximately) two circuits per cabinet

Base prototype cost

HDTC prototype cost

Shell at $ 35/square foot (127,500 square feet

$ 4,462,500

Shell at $ 35/square foot (105,000 square feet)

$ 3,675,000

Infrastructure

$ 52,048,000

Infrastructure

$ 48,822,000

Cabinets (4,720 at $ 1,025 per cabinet)

$ 4,838,000

Cabinets (3,150 at $ 2,010 per cabinet)

$ 6,331,500

Energy (Net present value for 10 years at 6%)

$ 62,712,000

Energy (Net present value for 10 years at 6%)

$ 59,080,500

Net present value

$ 124,060,500

Net present value

$ 117,909,000

NPV savings: $ 6,152,000 (10% of first cost)

$=H008