More "Nines" Not Always Better

A mission critical facility (MCF) provides an environment where processes occur which are integral to an organization's viability. The MCF for a stock exchange includes its trading floor and related clearing houses. A bank's MCF may consist of a data center that processes billions of dollars of credit card transactions; the most critical portion of a hospital may be its operating and communication rooms.

Owners, engineers and contractors discuss the power availability of an MCF in terms of "nines," which defines the probability that a system will function at a future time. For example, a 4-nine facility has a 99.99% probability of power availability at a future instance, while a 5-nine building has a 99.999% probability of availability.

To achieve these levels, organizations spend considerable sums to enhance mechanical and electrical systems that sustain MCFs. In fact, some owners spend hundreds of dollars per sq. ft. to improve reliability because an infrastructure failure can lead to an unacceptable loss of revenue, or more importantly, a loss of life. Yet many owners miss the big picture in that many MCFs, even with a lot of "9s," will fall short of desired expectations because single points of failure are overlooked.

For example, a designer may have multiple feeds from the local utility in parallel with backup generators, but the feeders may terminate at a single switchboard. A less obvious example involves control components, such as programmable logic controllers and input/output modules, which are often specified without redundancy. Various components can fail within these devices and create an extended power outage.

At the same time, the various subsystems of an MCF should be designed to a commensurate level of reliability, because the overall availability of the facility is no greater than the weakest link.

The other major factor in MCF power failure has to do with operational deficiencies. The decisions and interactions of humans greatly affect the availability of systems. And while human error is the single biggest reason for MCF failure, this variable is often neglected by engineers when a system's theoretical availability is estimated, as it's difficult to quantify.

MCF owners are also guilty. While flipping for a costly infrastructure, they inadequately budget for the training of personnel and the development of coherent and consistent procedures and processes.

What should owners and engineers do? First, the acceptance testing and operator training process should commence at the early stages of the project. The design performance criteria, which include all failure scenarios, should be tested as a complete system prior to acceptance.

After the acceptance of the facility, strict procedures regarding change control, periodic predictive maintenance and facility work rules should be required. The lack of these measures can lead to outages, which are otherwise preventable. For example, it's not unreasonable that UPS systems may become overloaded over time; or that essential equipment, such as generators and backup pumps, may not start because of poor maintenance practices.

In sum, many 9s can still result in naught without reliability from more mundane system components and proper training. So when budgets factor, maybe one less 9 may ultimately mean greater reliability.