Cabinets, Cabling and Cooling

CSE: Is it safe to say that many of the notions the engineering community has about cooling data centers and similar mission-critical facilities are already outdated?

SPINAZZOLA: Without question. The focus must be on kilowatts per cabinet, not watts per sq. ft.

STACK: The reality is that newer rack-based servers generate much more heat than older systems, creating hot spots within the controlled environment.

SPINAZZOLA: Yet there is an unwritten standard developing in the industry that about 3.5 kW per cabinet is the high end of what I call the "old" methodology to cool a data center—raised floor, downflow air-conditioning units, performance tiles, etc. This theory is called "random cooling," with the idea that if enough cool air is moved, the equipment will stay cool. Unfortunately, at more than 3.5 kW per cabinet, thermal migration within the cabinet causes significant overheating issues in the upper third of cabinets, resulting in a tripling of equipment failures.

CSE: So what kind of technology or design strategies can offset this heat build up?

STACK: One solution for dealing with hot spots is to mount a fan to the back of the rack to pull heat out. A more scalable approach—one of the most significant developments in data center cooling in the past 15 years—is represented by new supplemental cooling systems that mount to the top of the rack or to the ceiling above the rack. They provide very efficient cooling because they are located close to the source of heat and use the energy absorption capability of a liquid changing state in the cooling process.

SPINAZZOLA: Besides air and refrigerant-based solutions, there are also liquid-based solutions that employ chilled water piped to each cabinet, which also increases heat rejection capacity. All three solutions can cool about 8 kW per 24-in.-wide cabinet and are considered to be precision cooling approaches, meaning that the more directed the cooling solution, the more efficient the cooling.

MORDICK: More consideration should be given to duct placement and increased cooling reserves.

CSE: How about on the power side?

STACK: Newer servers have been found to operate more efficiently at higher voltages. Enclosure-mounted power distribution units (PDUs) are now available to support these systems. This can also reduce cable size to the enclosure and improve utilization of the PDU circuit breaker pole positions.

MORDICK: Don't forget servers require access to two power sources, thus a high density power distribution system is needed. Such a system also requires remote monitoring and must be able to selectively shut down certain servers when required.

CSE: When it comes to specifying this kind of equipment, how should an engineer go about determining exactly what a building owner's needs are?

STACK: As a first step, it is important to get an understanding of the business' reliance on their IT systems, and the cost if the IT systems go down. The system design has a major impact on the level of expandability that can ultimately be achieved, so it is important to design it to meet current and future availability requirements. It is likely that many organizations will be seeking to increase availability in the years ahead as they move toward 24x7 computing.

MORDICK: I can't emphasize enough the importance of incorporating reserve cooling capacity, as much as 50% to 100% for future expansion, and clean power access with 100% to 200% power access reserve.

And something we've only briefly touched on is cabling, but in the matter of meeting an owner's needs, reserve cabling capacity should be at least 200%. One should also expect that cables installed today will be obsolete in less than three years and will be required to be replaced, according to new codes.

SPINAZZOLA: Many data center owners deal directly with computer equipment vendors and therefore tend to buy a packaged solution that is comprised of either power or cooling, or both. The owner first needs to identify what computing products best meet his or her IT programming needs and not necessarily buy packaged solutions. The role of the engineer and manufacturer is to then develop a product and solution to address the cooling, power and cable-management issues to support the IT requirements.

CSE: Since we've broached the issue of cabling, let's address that subject as it relates to cabinets. As communications equipment has evolved to become more expensive and extensive in data centers, so has the volume and sensitivity of the cabling connected to it. Has the functionality of server cabinets developed sufficiently to meet an owner's needs?

MORDICK: Space is the key to cable management. With equipment becoming smaller, a trend has developed to cram all the equipment into as few cabinets as possible. What has not changed is the number of cables that interconnect and drop to the work area/PC locations. More cables are being routed into a cabinet and are often routed in areas that block airflow, which leads to cooling deficiencies. The real cost comes when changes are made to the cables and they need to be re-routed. In some cases this is a weekly task—be it for new offices, upgrades, new equipment, etc. The added up-front cost of providing expansion and easy access to cables—although relatively small—often entices many IT administrators not to do it correctly to meet short-term budget targets. In the end, it costs more in upgrades, added locations and network maintenance.

SPINAZZOLA: Cable management is one of the most important issues to be dealt with when deploying a data center. Cabinet manufactures now need to include it in their design to be competitive and meet industry standards. Most manufacturers have developed cable management products that fit into the standard cabinet in order to make the cabinet more adaptable to end users facing cable-management issues.

In fact, the leading cabinet manufacturers have developed integral cabinet cable management beyond industry standards. These manufacturers are including features that eliminate using specialty products, and are also developing products that are highly flexible and accommodate moves, adds and changes.

CSE: How about from a power quality perspective? Do any cabinets today provide shielding from EMF? How about cables?

MORDICK: Power coming into a cabinet can very easily pick up external noise or EMI/RFI energy. This radiation must be removed from the power source and oftentimes is grounded to earth.

Even though EMF is less of an issue within data centers, it is a real issue with industrial spaces, and metal cabinets alone are not the answer. The cabinet must provide conductive surfaces along all seams and door/cover converging areas. This can be done using conductive gasket material that is laid over a conductive surface. This approach provides what's called a "Faraday Cage."

SPINAZZOLA: There are a myriad of power solutions on the market, and many are compatible with most cabinets. With increased load per cabinet, power cable management is a major issue. Many computing products on the market are dual cord, or even tri-cord. Therefore, it is important to know what is going into each cabinet and match the power solution to that need.

But power quality really happens outside the cabinet through advances in UPS and PDU technology. The challenge in the short and mid term is putting the pieces together to be able to deploy blade server technology now starting to receive acceptance in the community.

CSE: In general, what kinds of technological advances do you see coming down the line?

STACK: In terms of cooling, we don't see the traditional computer room air conditioners going away, but we do see these systems increasingly being supplemented by spot cooling solutions targeted at specific racks. Some suppliers will introduce enclosures with integral water cooling capability, but the more advanced will provide this integral cooling capability without introducing water into the data center.

Power systems that support these high-density systems will not only need to be scalable, but also capable of supporting high levels of availability, meaning redundancy.

We also expect to see more powerful and expensive systems being utilized outside the data center, increasing the need for enclosures that can integrate power, cooling, monitoring and physical security in a self-contained system.

MORDICK: Cooling equipment will drive toward more integrated designs directly connecting the cabinet to the HVAC system to maximize efficiencies. I do not see cabinets with water or other fluids being plumbed to the cabinet as an efficient means to ensure acceptable cabinet environments. Although this process is being investigated and offered by several manufacturers, it is too expensive and introduces greater complexity to the overall system. I also question the reliability and flexibility of such systems.

Finally, in the area of cable management, we anticipate an increasing use of fiber from the data center to the work area. Wireless may also become more prevalent as issues addressing bandwidth and security are improved. Ultimately, this will help with the cable management issues.

SPINAZZOLA: I'm skeptical in that until the design community wakes up and smells the coffee, and accepts that the cooling approach we have been using for the last 30 years is no longer effective, new technology will sit on the shelf. That being said, it is important to know that there is great technology on the market just waiting to be deployed.