Tricky Trouble, Simple Solution
After portions of Frederick Memorial Hospital's new central energy plant came online, the hospital took responsibility for the regular testing and maintenance of the different systems. One of those tests included running the generators under the available building load on a weekly basis. The staff would start both generators manually and transfer each switch from the normal service to the gener...
After portions of Frederick Memorial Hospital's new central energy plant came online, the hospital took responsibility for the regular testing and maintenance of the different systems. One of those tests included running the generators under the available building load on a weekly basis. The staff would start both generators manually and transfer each switch from the normal service to the generator one by one.
One morning, shortly after the testing began, a call came in to the hospital facility department saying that the intensive care unit had lost devices connected to critical power. A quick check found that the automatic transfer switch (ATS) system was still getting power, so the trouble was downstream. Further investigation revealed that a 600-amp circuit breaker in the main critical branch distribution board had tripped. In addition to the ICU, this breaker served the lighting and appliance branch circuit panels for most of the operating rooms.
Over the next several weeks of testing by the hospital facility staff and TLC, the breaker would trip sporadically, sometimes going from normal to emergency transfer and other times going back. With no consistency to the pattern of tripping, deciphering the root cause became a trial-and-error challenge. The settings on the breaker were all turned up to maximum with no improvement. Since the new monitoring system was not yet online, a set of local meters was brought in to monitor the testing. Although spikes associated with a typical transfer were measured, the breaker did not trip despite efforts of switching it back and forth multiple times. However, right on cue, the breaker tripped again the following week.
This breaker happened to re-feed an existing portion of the critical electrical distribution, and something, obviously, had to be causing a large inrush current for the breaker to trip. Our first suspicion was a large transformer fed from the distribution board immediately downstream. The transformer fed mostly lighting and receptacle panels. A plot of the transformer inrush current vs. the upstream breaker protection curve quickly ruled this out as the sole possible source of trouble. Digging deeper into the loads on the existing critical electrical distribution revealed the culprit, as the distribution panel contained numerous three-phase motors including some pumping systems. The uncommon condition of motor loads transferring between unsynchronized power sources was at times creating inrush currents of 15 to 20 times normal running amps. This, coupled with transformer inrush, was causing the breaker to trip.
Now that the source of the problem was revealed, the solution became obvious: All of the transfer switches had been specified with in-phase monitors to allow the generator to sync up with the utility prior to transfer. This feature is especially useful for this type of testing where transfer from energized source to energized source is done. Although transfer times are increased under this scenario to allow for synchronization, under a normal power outage, transfer times are immediate since there is no source to sync to. The increased transfer time during testing has no effect on the hospital operation since the switch is energized in either position. The in-phase monitor function had already been enabled on the equipment branches because of the apparent motor loads, but not on the critical branch, since three-phase motor loads on this branch are not allowed for new installations. Enabling the in-phase monitor on the ATS upstream of the breaker immediately eliminated the nuisance tripping for testing. The breaker was reset to original design coordination settings. In the future, the motor loads will be taken off the critical branch.
It should be noted that the hospital maintenance and facility staff deserve a lot of credit for maintaining patient safety and hospital operations during the diagnosis of this problem. After the first incident, in which they quickly located and reset the tripped breaker, they stood beside it and immediately reset it when necessary on all the diagnostic testing until it was clear the problem was solved.