Optimizing Power System Protection for Data Centers

Data center power systems are constructed for premium reliability at a significant cost. Achieving an adequate return on such an investment requires that the system provides many years of trouble-free operation. Power system protection is designed to assure two seemingly contradictory purposes: maintaining a continuous (uninterrupted) flow of power; and acting rapidly to isolate power system f...

By Mike Mosman, P.E., Vice President and Director of Electrical Engineering, CCG Facilities Integration, Baltimore, and Reza Tajali, P.E., Square D, Nashville, Tenn. September 1, 2003

Data center power systems are constructed for premium reliability at a significant cost. Achieving an adequate return on such an investment requires that the system provides many years of trouble-free operation.

Power system protection is designed to assure two seemingly contradictory purposes: maintaining a continuous (uninterrupted) flow of power; and acting rapidly to isolate power system faults.

The most prevalent—and at times, insidious—type of fault is the ground fault. By understanding fundamental issues pertaining to ground fault application on uninterruptible power supply (UPS) equipment, a consulting engineer can preempt problems, and end users can find solutions.

Data center power systems can be solid grounded or high-resistance grounded. While both methods have their merits and disadvantages, this story focuses on solid grounding. Where this type is employed, a low impedance return path is provided for ground currents. National Electrical Code (NEC) article 215.10 requires ground fault protection on feeders of 1,000 amps or more on a solidly grounded four-wire system between 150 and 600 volts to ground. NEC 230.95 requires the same on service entrance equipment.

Figure 1 depicts a modern microprocessor-based ground fault protection system. This illustrates a radial system, where power can flow in only one direction. Under normal conditions, the currents in the three phases add up to zero. If a ground fault exists, the sum of the three phase currents will be equal to the ground current (or the zero sequence current). The sensor measures this current by summing the individual phase currents. The microprocessor will trip the circuit breaker when the zero sequence current exceeds its ground fault setting.

The power distribution in a data center, however, is not typically radial. Multiple power sources are tied together to increase system redundancy. This is very apparent in UPS output switchgear equipment. Figure 2 (p.18) depicts a parallel redundant UPS architecture. Multiple UPS modules are used to increase both capacity and redundancy. A static bypass switch provides a path for power to flow around the UPS modules when they cannot handle the load. A manual bypass feature bypasses the entire system for maintenance.

The UPS modules are separately derived sources and are grounded per NEC code, as are the other power sources—the utility source and the generator. In Figure 2 utility and generator have separate ground connections. This setup is acceptable as long as the neutral conductors are not extended, as in a three-wire system. Where four-wire systems are used, the transfer mechanism must be four-pole, or the generator must share the same grounding electrode conductor as the utility main.

For short periods of time during power transfer (about eight to 10 cycles), the static switch will run in conjunction with the UPS modules. This condition creates a path for zero-sequence circulating currents to flow. Figure 3 illustrates this condition. Note that one UPS module is removed to simplify the diagram.

Why do these circulating currents exist? The UPS modules and the utility—or generator—are voltage sources. Similarly, if two voltage sources are not identical, a net voltage difference will exist. This could be due to harmonics (voltage distortion) or the different characteristics of the sources.

Additionally, the UPS modules are controlled to lead the utility source by a few electrical degrees. If a low-impedance closed path is provided, a circulating current will flow. Solid grounding provides a very low impedance closed path for all zero-sequence currents. IEEE standard 142— The Green Book —provides a discussion of the circulating currents in parallel sources.

The ground fault relays on circuit breakers recognize these circulating currents as ground fault currents. If the magnitude of the circulating current is higher than the setting of the ground fault protection relay, and the time delay on the ground fault trip is set very low, the circuit breaker will trip, sacrificing the critical load.

Simplistic Spells Damage

One simplistic “solution” is to eliminate the ground fault protection on the UPS output switchgear equipment. This practice will actually reduce the reliability of the system. Consider a ground fault on the UPS System output bus in Figure 4 (p.19). Under normal conditions the static switch is open, and the power is supplied through the UPS modules. Ground faults often start as low magnitude arcing faults, and they sometimes progress slowly. Because ground fault protection is eliminated, this fault will continue to “burn” until the magnitude of the fault is higher than the overload setting—typically 300% of full-load amps—of the UPS system.

The ideal condition would be to detect this fault quickly and isolate it before it causes extensive damage. This has an effect on the equipment’s mean time to repair (MTTR). Quick identification and isolation of faults will reduce damage. This translates to shorter repair time, allowing the data center to be put back into service more quickly.

Ideal Protection

The ideal protection system is the differential system, and zones of protection are the essential elements of this system. Figure 5 (p.19) shows an example zone of protection. Any current that enters a differential zone must also leave that zone through one of the protected circuit breakers. Otherwise, all of the circuit breakers on the boundary of the zone will trip. Therefore, a fault inside the zone of protection will cause tripping of the circuit breakers that are on the boundary of that zone. But a circulating current will not set off a differential scheme.

If a differential system were employed in Figure 5, the fault on the UPS System output bus would only be detected by circuit breakers MOD1, MOD2 and MODS. These circuit breakers would open with little delay and isolate the fault.

Defining the zones of protection is the task of the consulting engineer. The degree of protection specified is a function of the reliability expectations of the system. Figure 6 (p.19) illustrates one method of defining the zones of protection.

Consider a ground fault in Zone 2. Only circuit breakers BP1 and BP2 detect this fault. The “Main” and “Gen” breakers will not. Because the system is selective regardless of the ground fault settings on the circuit breakers, it is not necessary to increase the setting on the main utility circuit breaker to obtain coordination between circuit breakers MAIN, BP1 and BP2.

This system is far more advanced than zone-selective interlocking, which neither resolves the circulating current nuisance, nor does it provide zones of protection.

Unnecessary Equipment Cost

Differential relays are expensive and costly to implement. The ideal system does not include them. A properly designed low-cost differential ground fault system would have the following features:

  • The system uses the ground fault protection circuitry in the circuit breakers to create the differential scheme. In the proposed system, ground fault sensors will have to be wired in a special configuration. Full testing and commissioning must also be included as part of the specifications.

  • Each bus is protected independently. A fault in one zone would only trip the circuit breakers connected to that bus.

  • The system is impervious to any and all circulating currents that can cause nuisance tripping of the circuit breakers

  • The system operates correctly under any arrangement of the circuit breakers. With any and all interlocks (mechanical and electrical) removed, the system would sense ground faults correctly and trip only the correct circuit breakers.

These four rules can be incorporated into almost any project specifications. With such systems, it is no longer necessary to artificially set the ground fault relays to high levels in order to avoid nuisance tripping. Ground fault settings can be set low enough to provide maximum protection and minimize the damage at the point of fault.

Challenge the Manufacturer

Competent engineers can design such a system. Some manufacturers may employ differential ground fault schemes in their products, but many do not. The UPS system design engineer should be aware of the circulating current phenomenon and challenge the equipment manufacturer to develop appropriate differential schemes to accommodate it. It is not necessary to degrade UPS system reliability to satisfy the NEC and provide ground fault protection.