Critical look at power

This month's panel discusses what facilities need mission critical power and at what levels, successful design schemes, technological advancements, and commissioning for these facilities. CSE: The term “mission critical” applies to data centers, telecom centers, and healthcare facilities.

By Melissa Hillebrand, Associate Editor August 1, 2008

This month’s panel discusses what facilities need mission critical power and at what levels, successful design schemes, technological advancements, and commissioning for these facilities.

CSE: The term “mission critical” applies to data centers, telecom centers, and healthcare facilities. What other types of facilities could be defined as mission critical?

BRIAN RENER: Mission critical could apply to semiconductor manufacturing (clean rooms); pharmaceutical, hazardous biological, or chemical labs; certain security or public safety related projects including 911 call centers, emergency or disaster response centers; military or government agencies; airport air traffic towers; and other critical transportation centers. The common theme of these facilities is the need for highly reliable and stable power systems to support critical functions.

BILL KOSIK: There are other obvious choices such as electronic trading operations, laboratories, and military and defense facilities, but mission critical really refers to any enterprise that needs total continuity of operation regardless of disruptions due to loss of grid power, severe weather, equipment failure, or even such mundane things such as equipment maintenance and upgrades. It really boils down to any business that relies on technology to generate revenue and wants to keep its customers (and shareholders) satisfied.

ED SPEARS: Additional mission critical applications include: air traffic control, broadcast TV, industrial applications, water treatment/water quality monitoring facilities, and, of course, nuclear power generation.

CSE: A few years back, we used to talk about mission critical power reliability in terms of the number of nines: 99.99% uptime meant four nines, etc. Do all mission critical facilities need the same level of reliability? How do you decide what level is necessary for a specific facility?

BRAD WALTER: No. The level of reliability required depends on a number of factors including the economic impact of disrupting the critical process, and the redundancy among facilities and within the process itself. Determining the level of redundancy requires a value analysis in conjunction with the customer and hopefully extending to the end-user clients concerning the processes and technical analysis of the effect of infrastructure failure on the processes.

CARL COTTULI: Maintainability and expected service level are a few key drivers. The combination of these numerical values will produce a targeted nines value, based on the amount of time the system needs to be available in a given year. The nines value will drive decisions around the topology combined with the maintenance requirements, which in turn drive decisions about the level of reliability around N, N+1, 2N, etc.

RENER: I think both uptime numbers and basic tiers should be used as a guideline rather than a guarantee. There are so many variables associated with each site and project. Systems are not always easy to fit exactly into a specific type of tier or uptime statistic. Reliability is never a static number over the life of a facility. Human factors play an important role in ongoing reliability. Further, if a power system is designed to meet some high level of complexity and statistic, but you need a doctorate to figure out how to operate and maintain it—you will face problems. Ultimately, however, it comes down to a cost/risk benefit analysis.

KOSIK: Having a very robust electrical distribution system serving the critical load will do nothing if other systems, such as cooling, ventilation, exhaust, and water supply systems that are necessary for the mission to be successful, do not have equally robust system topologies. It usually comes down to the age-old adage of the weakest link in the chain.

SPEARS: End users define the levels of reliability required. We now see regular requests for “six nines” designs. More commonly, though, four to five nines is the design goal. The following types of enterprises are driving all of us toward the “higher nines” targets:

  • Life safety/public safety (airlines, communications, public health)

  • Defense and government

  • Applications where operational loss would interrupt commerce (banks, retail, online commerce)

  • Applications where operational loss would interrupt revenue streams.

CSE: What are some of the more successful design schemes you see that guarantee maximum uptime in mission critical facilities?

COTTULI: The install, building, and commissioning phases of the project cycle are critical to success in terms of achieving a system’s maximum impact against a design intent. The commissioning phase particularly is important because it represents the point where the system can be put through all of its operational procedures at rated load. Taking the time in this phase is vital to understanding the system behavior and in turn ensuring the capability to support the load.

KOSIK: There really is no guarantee against an interruption to the operations, even with the most robust mechanical and electrical design scheme. This is based on one critical fact: Humans are the single greatest reason that critical facilities have outages. This is why it is important that the systems are not overly complicated to operate regardless of the required uptime, especially during times when a component has failed or during a maintenance operation. This is when stress levels are high and people can make mistakes. At times like these, having clear, easily understandable, and intuitive control and monitoring systems will go a long way to help avoiding unintentional mishaps that can cause a loss of operational continuity.

SPEARS: Dual powered/multi-corded information technology (IT) gear allows the use of dual bus, or A/B bus powering architecture, and this is becoming generally accepted as the most reliable scheme for tier III/IV requirements. Its benefits are best realized in newer facilities, where all of the equipment is dual-corded. In legacy installations, the A/B bus arrangement requires the additional complexity of inline static switches and the need to synchronize independent generator and UPS systems. A carefully designed system will include provisions to minimize the need for series switching, and enhance the ability to perform concurrent maintenance, and allow for battery and load bank testing.

RENER: From a system design scheme, I have seen multiple power distribution units (PDUs) serving dual corded racks as well as PDUs with multiple sources (transfer switches) resulting in multiple, independent pathways to the end-use equipment. Equipment selection and design can only go so far. I believe the biggest reliability issues depend on human factors where independent acceptance testing and commissioning, and operator training play a major role.

CSE: What specific power quality issues do you encounter in designing power systems for mission critical facilities?

KOSIK: It is not a power quality issue as much as it is a power efficiency issue. Without careful analysis and planning, highly reliable power and cooling systems also can become highly inefficient and use unnecessary high amounts of electricity. This is happening because of the inherent efficiency points that each piece of power and cooling equipment will have. An example of this is when multiple, large centrifugal chillers are used as a part of a cooling plant design. Generally, these chillers will have an optimal efficiency at approximately 70% of the total load. So if the plant configuration is 2N or N+2, it is very important to use control strategies that allow the chillers to run efficiently but also ensure that there is no loss of cooling in case of a failure or during a maintenance procedure.

RENER: No. 1 issue: getting reliable accurate information out of the utility provider. Very few utilities companies have detailed, accurate power quality information on their power services at a site. The second issue is getting reliable power nameplate information to do load calculations for the planned data equipment. Nameplate data provides peak information and often does not provide enough information on normal usage. The peak numbers are needed to plan the distribution system, but normal usage data are needed to effectively evaluate the power required at the facility service level. Many times equipment has yet to be selected, or is not even available from the manufacturer, during the early planning and design phases of the mission critical facility. Power quality, especially harmonics, within the facility also is a significant issue.

COTTULI: Power quality problems engineers typically encounter include brownouts and blackouts. As the demand on the utility grid continues to climb on aging utility infrastructures, these occurrences are expected to increase. Consequently, customers from many market segments are looking to some power quality products.

SPEARS: Proper grounding and handling of the neutral circuit continues to be a challenge in many installations, and this is a matter of vendors recognizing the need to provide clear, concise, communications regarding proper methods for installing and connecting power protection infrastructure.

Harmonics (created either by the UPS or the connected load) often are an unexpected problem, because the impedance of the generator or utility source varies for different sites. In general, this issue is becoming much less common with the advent of improved UPS rectifier designs, and the use of power factor (PF) corrected IT equipment.

Speaking of PF-corrected IT equipment, we are seeing increased incidences of “leading” or capacitive power factor loads, especially in newer data centers where the processing loads are relatively light. This often is a problem for UPS systems, as they can become unstable when faced with a leading load. Only a few vendors have successfully addressed this issue without requiring de-rating of their equipment. Generators, however, may respond to a leading load by shutting down unexpectedly. It has become more important to ask, early in the game: “Is there a chance that my data center will exhibit a leading power factor?” The answer may significantly affect the UPS and generator purchasing decision, and this is not a pleasant issue to deal with after equipment is already installed and operating.

WALTER: In the developed world, the issues are generally transients, sags swells, and outages. Frequency variations from utilities are exceedingly rare, and when they do occur, they are generally an immediate precursor to a major power outage. The situation where frequency stability may be a problem is when the primary power source is an on-site cogeneration plant. In addition, modern IT and communications equipment is very tolerant of frequency variations and relatively tolerant of voltage variations compared to industrial control systems.

CSE: Looking into the future, what types of future advancements could improve power reliability for these facilities?

RENER: I think that reliability issues are being joined by another connected issue—energy efficiency. Rising energy costs are forcing a new perspective on mission critical facilities. An unexpected benefit of implementing energy saving measures can be an increase in reliability. One example might be careful air flow modeling. Delivering air only where it is most needed affects reliability but also can help minimize oversizing air systems and save energy. Properly sized and loaded electrical and mechanical systems are more efficient and provide better performance. Alternative power sources such as geothermal also are being explored. Another significant challenge is providing cooling to rack-mounted equipment with very high power densities (i.e., more than 4 kW per rack).

WALTER: One improvement might be distribution of IT power at 400 V/230 V to eliminate the two-step conversion from medium voltage to 480 V and from 480 to 120 V/208 V. More and more infrastructure power equipment is becoming “world class” that operates at 716 F at 400 V in some cases, or 415 V at 50 or 60 Hz as well as 480 V at 60 Hz. Converting directly from medium voltage to 400 V/230 V reduces the amount of equipment in the critical path while simultaneously reducing losses in the IT equipment power path and increasing the capacity of feeders going to the racks at a given current rating.

CSE: What do engineers need to know about Article 708: Critical Operations Power Systems, in the 2008 National Electrical Code and how it applies to mission critical power?

COTTULI: The big take-away for the engineer on any new code of this sort is to understand the current position of the authority having jurisdiction where the facility is installed. Although the NEC has issued and approved this standard, the local authority having jurisdiction may ignore this code section or adopt it. In fact, only a few local authorities have even reviewed and issued a position, some have adopted the new section, and some have rejected it stating existing code and local interpretation cover the items to and adequate level. So call your local building inspector office.

RENER: This section was added to address homeland security related issues. Practices such as wiring separation and protection that have been required for systems like life safety or fire protection now are applied to power systems that affect public safety and national security. There are other interesting requirements in this new section that have been considered good engineering practice in other types of mission critical facilities, like multiple levels of ground fault protection, surge protection, commissioning, and acceptance testing. All in all, it is an excellent new addition to the code and hopefully a source of ideas for other types of mission critical facilities, which are not covered by this new article.

CSE: What role does commissioning and acceptance testing play in today’s mission critical facilities/power systems?

KOSIK: The activities that take place during the life of the project that have to do with equipment/system performance testing and verification can make or break a project. It is the weakest link metaphor at work again. If the equipment is delivered to the site without proper factory acceptance testing, it is not known how that equipment will perform in a failure mode. Similarly, if system level testing is not done in a rigorous manner with detailed up-front planning, what happens during an incident will be unpredictable, possibly causing a major failure. Finally, debugging the interdependencies between the power, cooling, and control systems during integrated systems testing is really the culmination of all the efforts in making the facility as resilient and robust as it was meant to be.

RENER: A very important role, perhaps the single most cost-effective part in improving reliability in today’s mission critical facilities. The two really are separate but connected services. Acceptance testing using InterNational Electrical Testing Assn. standards and qualified technicians ensures that the various components in an electrical system are independently and rigorously tested in the field. Commissioning is a systems process using National Environmental Balancing Bureau standards and independent third-party professionals to review everything—from design, to components, to installation, and testing—to ensure that the entire electrical and mechanical system operates exactly the way it was intended. These services may only add 1% to 2% of upfront costs, but result in significant reductions in change orders, time lost, operational problems, and outages. Commissioning also can include gathering and providing a substantial amount of documentation on the equipment and operating sequences for the facility, especially the HVAC and electrical system. This makes it much easier for the on-site O&M staff to better understand the systems and respond more effectively in the event of problems.

COTTULI: It is incredibly important to conduct a full commissioning in as close to a real operating condition as possible. Commissioning comes as the last item in a long, expensive journey as it faces pressures to shorten and expedite. This is a critical mistake. The commission phase needs to be one of the most comprehensive to ensure operability of equipment, familiarization with operating personnel, and final acceptance of a long expense capital expenditure project. In other words, did you get what you paid for from your vendor?

SPEARS: Commissioning and acceptance testing are extremely important to ensuring the successful operation during a powering disturbance. All too often, due to cost, logistical complexities, availability of test equipment, or just plain negligence, this testing is not done comprehensively, or not done completely. This presents a significant risk for the end user, who may not be aware that mission a critical power system has not been tested for every contingency. The participation of experts, whether they be consultants or vendors, should be a part of every installation. The critical equipment should not be turned over until this testing has been completed and reviewed, with all outstanding issues addressed.

WALTER: Commissioning and acceptance testing are crucial in validating the overall design; ensuring that various pieces of equipment, as installed, function as intended; and ensuring that the equipment and subsystems integrate properly to accomplish the desired functionality. These activities span everything from the obvious, such as breaker coordination and correct wiring of control interfaces, to more subtle issues of timing, threshold settings, and dynamic response to transient conditions. Documented test results also provide a benchmark against which future performance may be compared to highlight potential problems that may develop before they increase risk to the critical mission.

Participants

Carl Cottuli

Vice President of Data Center Science Center

APC

West Kingston, R.I.

Bill Kosik, PE, LEED AP

Chicago Managing Principal/Energy and Environmental Initiatives Leader

EYP Mission Critical Facilities

Chicago

Brian A. Rener, PE, LEED AP

President, Central Region

Sebesta Blomberg & Assocs.

Chicago

Ed Spears

Marketing Program Manager for Technical Sales

Eaton

Cleveland

Brad Walter

Dir. of Applications Engineering and System Development Active Power

Austin, Texas

Ask the experts: overcurrent devices

Every month, Consulting-Specifying Engineer editors ask a distinguished panel of experts for information about how to best solve your problems, challenges, and new engineering issues. At www.csemag.com/asktheexperts, CSE gives its readers and Web visitors the opportunity to pose questions directly to the panelists. Below is a question for August’s topic, specifically about NEC 700.27 overcurrent devices.

“How does NEC article 700.27: overcurrent devices affect the selectively coordinated system using circuit breakers? Is it possible to use circuit breakers in a selectively coordinated system under NEC article 700.27? ” —Donald Zasada, Construction Official—Electrical, New Jersey Dept. of Community Affairs, Trenton, N.J.

BRIAN RENER: It depends. There are a wide variety of circuit breaker types. Specifically, many engineers overlook selecting breakers with various adjustable tip features. Advanced features such as zone interlocking also are available on larger circuit breakers. Further, how a system is designed can help or hinder the coordination processes. If properly designed and specified, circuit breakers are an excellent choice for a selectively coordinated system. Another factor is selective coordination of ground fault protection. NEC requires ground fault protection for 277 V/480 V circuits 1000 amps and larger. There are many instances where a ground fault on a branch circuit has resulted in tripping the 1,000 amp feeder, thus affecting a significant portion of a facility rather than isolating the ground fault closer to the fault. It may be necessary to apply ground fault protection to lower amperage feeders and branch circuits to provide effective selectivity for ground faults.