Providing power for high-tech facilities
High-tech buildings as well as mission critical facilities must have reliable electrical power.
For the purpose of this article, high-tech buildings such as pharmaceutical or manufacturing facilities can vary greatly with respect to how reliable power is defined and addressed in the context of the specific business needs. Many operate 24 hr per day, every day of the year. These buildings always need a reliable power source because downtime could result in millions of dollars per hr in production losses.
A data center is an example of a mission critical facility (see Figure 1). It could be a stand-alone building or a space within a building where mission critical computer equipment operates. A mission critical area or facility could reside within a high-tech facility, and it is not uncommon for some variation of a mission critical area to be present in many larger buildings.
The use, function, and customers for various high-tech facilities can cover a broad spectrum. The commonality among these facilities is the need for constant, reliable power. The tolerances in high-tech manufacturing can be so precise that a slight voltage or frequency deviation can destroy products.
In high-tech facilities, the consequences of power fluctuations and outages are very significant and in extreme cases catastrophic. Each facility type has unique challenges based on the specific equipment required for its business or process and issues specific to conditions for its geography. Weather, natural disasters, existing utility infrastructure, and similar factors external to the facility are unique for every location and must be analyzed on a case-by-case basis to determine design measures to mitigate associated threats.
As consulting engineers, the requirements for the high-tech facility power system are driven by industry standards and codes (such as NFPA 70, NFPA 110, IEEE 141, IEEE 142, IEEE 242, and IEEE 446) and often more importantly by client expectations and the needs of the client’s particular business model. The impact of an outage drives code requirements. Codes must be followed to mitigate life safety concerns and processes that maintain the safe operations of the business—both of which mandate a level of investment.
High-tech facilities rely heavily on sensitive electronic equipment. Equipment manufacturers typically define operational parameters such as minimum acceptable voltage, frequency, harmonic content, and similar characteristics required for proper equipment operation. Reliable power must be provided to ensure this sensitive electronic equipment operates properly.
One of the challenges associated with power quality is the use of terms that lack universal or consistent definition. Terms such as “clean power,” “spikes,” and “clean ground” fail to provide clear definition of the requirements for proper operation. In the 1970s, the Computer Business Equipment Manufacturers Association developed the CBEMA curve, which evolved into a common guideline for defining and analyzing power quality. The CBEMA curve was also adopted by IEEE 446 and is used as a guide for designing standby and emergency power systems. Fundamentally, power quality is the ability of the power system to serve the load in a manner that facilitates proper equipment operation.
To deliver reliable power for high-tech facilities, the power system design approach must be holistic with a strong focus on the following fundamental design elements to achieve a robust electrical infrastructure:
· Grounding system
· Lightning protection
· Surge protection
· Utility service
· Generator backup
The foundation for a high-tech facility electrical design is a grounding system that complies with IEEE 142: Recommended Practice for Grounding of Industrial and Commercial Power Systems and NFPA 70: National Electrical Code (NEC). The Codes address safety requirements whereas the standards incorporate features necessary for required performance. The service entrance, separately derived systems, and equipment grounding conductors are parts of the grounding and bonding system. NEC Article 250.50 requires the grounding electrode system to be bonded to all grounding electrodes throughout the building. The intent is to ensure the entire building has a system that minimizes the difference in potential for the grounding system and also provides a low impedance path to facilitate overcurrent protective device operation in the event of a system fault.
In high-tech facilities, sometimes challenges occur when an equipment manufacturer or vendor requests a stand-alone ground that is isolated from the building grounding system. However, unbonded, separate grounding systems are not permissible because they cause safety hazards and violate NEC Article 250.50. The grounding system must be maintained as a safe code-compliant system and for some sensitive electronic equipment must also address performance needs in the form of an isolated ground, as described in NEC Article 250.96(B).
The isolated grounding conductor connects to an isolated grounding receptacle at the equipment and extends to the transformer secondary bonding point, which limits the connections to the normal equipment ground bonding system. An isolated grounding conductor is intentionally designed for high impedance to reduce the electronic noise (electromagnetic interference) for the grounding system to a specific piece of equipment. Appropriate grounding is a basic building block for a safe, robust, and stable power system.
Lightning protection is another fundamental building block of a robust power system design—particularly in storm-prone geographies. The location of the facility and the average quantity and density of lightning strikes are considerations for the risk assessment that should be conducted specific to the project. A lightning protection system is usually comprised of a UL Master Labeled system installed in compliance with NFPA 780: Standard for the Installation of Lightning Protection Systems. Although less conventional, early streamer emission type systems are another approach to facility protection. The intent of the lightning protection system is to protect the facility and ensure that if a lightning strike occurs, the entire building potential rises and falls in unison. By ensuring the system is appropriately bonded and functions as a single system, the likelihood of flashover occurring is minimized and the facility is able to safely address the energy the lightning strike imposes on the system. The frequency of lightning strikes to the facility may increase, but the damage from each strike is lessened.
Surge arresters (NEC Article 280) and surge protective devices (SPD) (NEC Article 285) are critical elements in protecting high-tech facilities from externally and internally induced voltage anomalies. To protect the system and equipment, the possible sources of surges for the system must be analyzed with appropriate SPD technology applied at strategic points for the system. SPDs are installed at numerous levels in the electrical distribution system, and the ratings of the devices must be analyzed in the context of the potential energy levels that must be addressed at the various locations in the distribution system.
The highest energy events typically originate external to the facility; hence, any location at which power is delivered to the building from an exterior source such as utility service or exterior generator service should be protected. The intent is to capture externally generated surges at the point they enter the building so the anomaly is not allowed to migrate into the building distribution system. The building service entrance is the first line of defense, with a second level of protection at the main distribution switchboards downstream from the service entrance. Internal to the building, there is often equipment such as motor starters or contactor-controlled loads that impose voltage transients that can also be mitigated with SPDs. For internally induced surges, prudent segregation of loads in conjunction with separately derived systems for a nominal level of buffer and SPDs at the panelboard and receptacle level results in a system that is effective in addressing surge conditions inside the building.
The quality of the available utility service is a key consideration in the design of a power distribution system serving a high-tech facility. Estimating the electrical load for the facility and determining the proposed serving utility’s system constraints for service to the building must be evaluated to ascertain quality of the utility source. The engineer should request an outage history from the serving utility including the quantity and durations of outages—preferably for at least a 2-year period. Also request the utility’s definition of an outage. For example, some utilities log an outage only when a customer calls or a service crew has to be dispatched. Overhead or underground distribution for the site should also be reviewed with the serving utility and owner. Although an overhead distribution is often self-healing (clears faults), it is typically more prone to voltage anomalies than an underground system. Depending on the magnitude and critical nature of the load, it may be prudent to research with the owner and utility to determine if the project justifies utility services from two separate utility-owned substations for improved availability.
Large high-tech facilities may lend themselves to medium-voltage service. For reliability, multiple substations are typically used to step the power down to usable voltages with main-tie-mains for flexibility in substation operations. When designing these larger facilities, consideration must be given to separating substations from each other and from generator backup in the unlikely event that one may fail, causing a fire. Additional consideration must be given to the availability of maintenance personnel needed to repair and test the medium-voltage equipment.
Using medium-voltage power with either utility or privately owned substations increases the reliability of power service and introduces the ability to parallel generators at the medium-voltage level and the potential of looped medium voltage for ease of expansion. Coordination with the utility to determine the available fault current is a key design element to ensure short-circuit current ratings and maximum available fault currents are addressed when selecting overcurrent protective devices and equipment. The designer should consider numerous switching configurations of different systems to give the most flexibility for the distribution system. The system configuration must consider the necessary testing and operation of overcurrent devices on a systematic testing plan, as well as minimization of common mode failure points. The available fault current will also influence equipment arc flash calculations, which must be performed for distribution coordination to minimize the incident energy available at any point in the system.
The quality of the electrical utility service usually has direct impact on the backup generation system design. The requirements for emergency (NEC Article 700) and legally required standby (NEC Article 701) are baseline design requirements. The optional standby (NEC Article 702) loads are those loads critical to maintain operations and driven by the needs of the business as opposed to NEC 700 or NEC 701 requirements. The engineer must coordinate with the owner to determine what is required to ensure operations are maintained and an acceptable level in the event of a utility outage. The business risk associated with a power outage will drive the design requirement.
Parallel generation can reduce the risk associated with failure of a single genset engine. With large systems, the aggregate bus associated with parallel generation provides a robust backup source because of the stability of a larger capacity source as loads are added and subtracted from the system. Paralleling with the utility and providing closed transition (make-before-break) functionality creates system flexibility for testing and also allows for proactive/preemptive operations. In other words, the ability to transfer loads to the generators without disruption as a preemptive measure in the event of a storm or a planned system utility outage enhances system operations and minimizes disruption for high-tech facility loads.
An important item often overlooked in parallel generation system design is the need for redundant equipment station batteries. Station batteries provide power to control and monitor the paralleling system. If the batteries fail, the entire system fails. When designing large complex systems, engineers should seek out single points of failure and determine the most at-risk elements of the system. In other words, the system is only as robust as the weakest link. In addition to providing both code-required and business-critical power in the event of an outage, a prudently designed generation system can greatly enhance testing, maintenance, and recovery capabilities if there happens to be a utility event or a problem with the on-site distribution.
Even in high-tech facilities, the need for mission critical UPS systems has become more commonplace. At many offices, small local UPS units may reside at every desk and in each rack of equipment. Larger centralized UPS units typically protect the enterprise data center serving the facility. Different types of UPS equipment are used for power system protection. They include standby, line interactive, standby ferro, double conversion, delta conversion, and rotary. Applications vary from desktop to very large mission critical N+2 or greater systems.
Maintenance should be considered to ensure the system can be serviced with limited or no disruption of the critical load. An external maintenance bypass should be provided to allow UPS removal and replacement without downtime. Consider an N+1 UPS arrangement and external maintenance bypass for maintenance and future replacement with minimal impact to the loads. Review battery supplies to the UPS versus flywheel backup or a combination of both. In order to reduce the likelihood of common mode failure, serve the UPS from two alternate sources if possible. In order to ensure a single neutral to ground bond reference, the UPS input, internal maintenance bypass and external maintenance bypass should be configured for three phase, three wire. The UPS system output feeder should be three phase, three wire with separately derived systems created downstream.
The UPS technology used is a function of required run time and whether or not generator backup is provided. If the objective is sustained operation with a generator working in conjunction with a battery-based UPS system, the battery size can be limited to time required for generation to be on line. If there is no generator backup and the objective is orderly shutdown, the batteries must be sized to provide sufficient time. Providing sufficient run time for an orderly shutdown results in battery run time that may be longer than if the UPS is intended to just provide ride-through until generators can assume the load. If frequent short-duration utility outages or voltage events are the challenge, rotary UPS technology may provide sufficient ride-through time for the utility to stabilize. There are numerous options and engineers must work with their clients to define the challenges to be addressed. Redundancy—both internal to the equipment and for the external system—should always be investigated.
UPS systems are complex and, although current technologies are robust, contingencies in the event of equipment failure should be analyzed and defined for the end user. The typical online UPS will filter and condition the incoming power to the sensitive electronic loads as well as reduce harmonic implications for the line side system. Mission critical facilities typically require UPS power for ride-through until generators can come online and sometimes for power conditioning.
Since the invention of switch-mode power supplies for electronic equipment, there has been much concern about their impact on electrical distribution systems. Rectifiers in these power supplies produce the equivalent of square waves from which the dc output is developed. These simulated square waves are actually created by multitudes of harmonic frequencies reflected back into the electrical distribution system. These harmonics are typically most prevalent in the third, fifth, seventh, and ninth orders. Without using harmonic mitigating methods, transformers and electrical conductors could overheat—particularly on the neutrals where the harmonics typically add instead of canceling.
This overheating is typically a result of the high harmonic currents present at these odd orders of harmonics and can sometimes create a requirement for derating the fundamental rating by as much as 50%. Additionally, the transformers will still see these harmonic currents in the form of circulating currents in the delta windings causing inefficient operation and the potential for transformer overheating.
Harmonic mitigation and cancellation techniques can be used to minimize harmonic content within the electrical distribution system. Vendors should be directed to provide equipment with minimal reflective harmonics, such as keeping the total harmonic distortion below 10% for both voltage and current. Harmonic filters at the panelboard level can minimize the negative effects on long feeders that serve panelboards. If the panelboards are close to a distribution board, the filter can be applied at the distribution board to achieve the same results. Finally, the application of zigzag phase-shift transformers can help cancel harmonic currents, minimize transformer overheating, and limit harmonic reflection back to the upstream components. High harmonic content impacts energy consumption because the harmonic currents increase VARs. Harmonic current mitigation via passive-zero-sequence filtering and phase shifting for cancellation of harmonic currents are strategies that must be evaluated at the outset of design to enhance both power quality and system energy performance.
The simplified single line power diagram shown in Figure 2 represents a small data center suitable for a midsized company. The building has a single 480/277-V, 3-phase, 4-wire, 1,200-A service entrance. This service feeds mechanical equipment, lighting, and stepdown transformers for 208/120-V, 3-phase, 4-wire loads such as receptacles and small equipment. A dual-input UPS is provided with a 480-V input and 480/277-V output. The backup input to the internal static bypass is configured with a 480/277-V input and also serves as the maintenance bypass. The UPS synchronizes with this input so that the external maintenance bypass can be used without disrupting the loads. The UPS is a high efficiency (97% or greater) unit and uses batteries as the backup source. It is constantly inverting the incoming power and charging the batteries. The batteries’ direct current power is inverted to alternating current to supply power to the load.
The single line diagram shown in Figure 3 reflects the addition of a backup generator power source (NEC Article 702) with an open-transition automatic transfer switch for the UPS and selected mechanical loads. Deration for the generator must be evaluated due to the UPS and the inrush of the mechanical equipment. Also pay attention to the potential 3- or 4-pole transfer switch and the grounding and bonding requirements. In this application, when utility power is lost, the bypass input to the UPS and the external maintenance bypass are no longer available.
The single line diagram shown in Figure 4 reflects generator backup for the entire facility with an open transition circuit breaker auto transfer application. Note that NEC Article 702 also applies to this application. This generator must be sized for the entire facility, and considerations must be made for the inrush currents and deration of generation capacity based on the UPS. A more complex version of this system could be designed to incorporate electrically operated circuit breakers to shunt loads when power is supplied by the generator, thus minimizing the genset size.
Alternate designs may be considered that include redundant paralleled UPSs, redundant paralleled generators, and alternate utility transformers with main-tie-main breakers or switches. Numerous configurations can be derived from these alternatives, but the designer must always beware of grounding and bonding, ground fault protection schemes, and coordination studies for overcurrent protection. When paralleling generators in a redundant configuration or paralleling with the utility, consideration must be given for the possibility of increased available fault current.
One of the most overlooked items when designing facilities such as these are that many operate 24/7/365. The designer must consider the need to exercise switches and circuit breakers and minimize arc flash energy. Other required routine testing includes annual (minimum) infrared scanning and comparing the results to those of the previous year, running the generators weekly or monthly to confirm they are ready to function properly when needed, standard maintenance routines for all batteries (UPS, generators, and paralleling equipment), routine maintenance of generators fluids and filters, testing fluids in medium-voltage transformers, and testing air transfer systems that are needed when the generators operate. Periodic, consistent testing also provides opportunities to discover system deficiencies in a controlled setting with appropriate staff on hand to react in a timely manner.
Reyburn is the director of electrical engineering at JBA Consulting Engineers in Las Vegas. He is a professional engineer registered in 31 states and is a member of NFPA, NSPE, IAEI, IES, and ICC.