Keeping HAL Cool

Texaco's data center in Houston, Texas, serves several critical corporate needs, including research and development, analysis of field surveys and the "7 x forever" support of the global data center. The central energy building also supports executive offices and other business support areas.

By EDWARD C. KOPLIN, P.E., President, Jack Dale Associates, P.C., Baltimore, Md. November 1, 2000

Texaco’s data center in Houston, Texas, serves several critical corporate needs, including research and development, analysis of field surveys and the “7 x forever” support of the global data center. The central energy building also supports executive offices and other business support areas.

In 1995, with the pending addition of 60,000 square feet of office space at the data-center headquarters, Texaco facility engineers felt nervous that the existing setup would not be able to accommodate the additional load. In a recent interview, Glenn Hanson, facility operations supervisor, recalled how-new to the post-he had noticed that, “we weren’t getting the cooling [we needed].”

In fact, the 30-year-old plant had undergone a number of capacity expansions that resulted in a state of complex and unpredictable infrastructure control. A departure from legacy water-cooled mainframes to higher-power-density air-cooled midrange server environments posed new challenges to the existing infrastructure.

HVAC tops data-center concerns

The fact is, such challenges are endemic to operating mission-critical facilities. “Cooling is the biggest problem facing data-center designers today,” Dr. Roger Schmidt, head of air-cooled mainframe development at Armonk, N.Y.-based IBM stated at a recent presentation, adding that campus-wide cooling systems must evolve as data-systems designers migrate to state-of-the-art technology. Specifically, central chiller plants must adapt to nodes popping up in the infrastructure web.

In a perfect world, designers would always keep a sense of global perspective when designing cooling subsystems. Unfortunately, data-center hardware designers rarely understand the full impact of their decisions on existing cooling systems.

Subsystems at the Texaco facility were no different. As it was, problems abounded in the existing infrastructure, and Hansen found many faults in the addled and outdated cooling system. For example, “The return [chilled-water] pressure was so much higher than the supply pressure,” he recalls. “We were having to run more and more equipment just to get the heat transfer we needed.”

Wild fluctuations in chilled-water pressure, temperature and flow caused constant complaints about space temperature (see Figure 1, right) Flawed pump-control strategies eroded control valves. A destructive hydraulic water hammer blew the cast-iron impeller casing apart on a 2,000-gpm centrifugal pump. Frantic operators attempted to control critical space- and process-cooling loads with manual isolation valves, wasting capacity, energy and manpower. In addition, excessive chilled water returning from remote buildings wreaked havoc with central-plant capacity control. Temperature and humidity instability occurred as central-station air-handler coils dehumidified the data center while steam generators in computer-room air handlers operated continuously to keep the humidity levels up to setpoint. Moreover, these massive inefficiencies made utility costs difficult to budget. (See Figure 2, this page, for an overview of the original faulty chiller plant.)

Driving with the brakes on

There were further challenges in simply attempting to rectify the problems with the existing systems. For example, maintenance shutdowns and disruptions to critical processes were not allowed-and are still not allowed-according to Texaco’s corporate mandates. Sweeping changes to the chiller plant didn’t seem possible without taking the data center and all the support structure down for a period of weeks.

Texaco’s engineers faced a dangerous lack of both temperature and humidity control despite the fact that their chilled-water plant was pumping over 10,560 gallons per minute (gpm). It was as if they were pressing the accelerator all the way down while driving with the brakes on (see Table 1, this page). How could the company update the system to accommodate both new air-cooled servers and an additional 60,000 square feet without having to shut the entire chiller plant down?

Hanson recognized that prior engineering efforts lacked global perspective; he was managing a series of tactical improvements that were causing adverse conditions when operated concurrently. One system loosely tacked onto another led to fragmented temperature-control concepts and incompatible chilled-water pumping strategies for each building. Further, specifications for chilled-water coils were incompatible with chiller selections, and system design flaws precluded an effective operator-training program. Erratic temperatures and flows wasted available capacity and redundancy.

Jack Dale Associates, Baltimore, was retained to benchmark the campus systems and solve their dilemma. The biggest challenge seemed to be to develop a set of design documents that would allow Texaco to implement the engineered solutions without any disruptions to the global data center or to research and development areas. Led by the motto that “a problem well defined will seek its own solution,” Hanson and the consultants began seeking the guiding principles for producing the construction drawings and specifications.

Hydraulic decoupling

Prior to developing design concepts, existing systems were extensively tested and evaluated. A holistic system review identified various interactions confirming the operators’ experiences and explaining failure modes of recurring disruptions. In-the-field and over-the-phone interviews with the operators were conducted. Hanson preferred keeping facility operators involved in the design process because, in his words, “They’re the one’s that are going to make it or break it.”

The consulting engineers outlined improvements and procedures that boosted chiller-plant efficiency and are credited with saving Texaco at least $100,000 per year in utility costs (see Table 2, below). The course of action primarily dealt with existing operational problems. However, the design also offered “hooks” for expansion of the chilled-water reserve-and reduced energy consumption.

“Once we put in the bidirectional hydraulic decoupling bridges, that pretty much set the demand,” Hanson explains (see Figure 2, page 64 ). “Plus, reducing the pumping horsepower throughout the complex [by 950 hp] and then putting in the variable-frequency drives ended up so that whatever the demand was is what the central chilled-water system distributed, no more and no less.”

The design and construction phases of the project were completed quickly-with no facility shutdown-and at a reasonable cost of less than $600,000. This was significantly better than the estimated cost of $2 million and two-week shutdown of the entire 5,000-ton chiller plant that had been proposed by another engineering firm.

Five years after the project is complete, Texaco still relies on the redesigned system. With a satisfied grin, Hanson comments, “Aside from the fact that we’ve been able to train our operators much faster than before … the beauty of this system is that it basically runs itself.”