Diagnosing, resolving ATS timing issues
Generator load transfer timing is critical during power disruptions.
Doug Calhoun, PE, LEED AP BD+C, and Mark Montgomery, PE, LEED AP, ccrd partners
In healthcare facilities across the U.S., a very critical component of the essential electrical system (EES) is often overlooked. While generators are tested and fuel delivery systems are maintained, appropriate time-delay settings for automatic transfer switches (ATSs) in the EES are often overlooked (see Figure 1).
In both new and existing facilities where new ATS loads are added to the EES, the timing of load transfers to and from the generator during power disruptions is critical to assure proper system operation. National Electric Code (NEC) 517 describes the allowable criteria necessary to use time delays for the appropriate loads. NFPA 99 and NFPA 110 describe further design requirements including acceptable environmental considerations for ATSs, delay settings, load shed criteria, and critical definitions that should be understood when applying ATSs in a healthcare environment.
One such definition to consider is the difference between Level 1 and Level 2 systems. A Level 1 system is determined by the risk that human life could be lost if the emergency power system fails to perform, as stated in NFPA 99-184.108.40.206.10. A Level 2 system is determined by the local authority having jurisdiction (AHJ), and includes facilities with a lower risk to human life from an emergency power system failure. Most of the time in a healthcare facility requiring emergency power, a Level 1 system is required.
Another important distinction is the emergency power supply system (EPSS) type, which determines the time within which the EPSS must restore power to the life safety and critical loads. This is determined by NFPA 99-220.127.116.11.6.1 and NPFA110 Table 2.2-2. By developing a good understanding of NEC 517, NFPA, and the transfer switch time-delay settings, the generator system loading can be staged and large block loads dropping onto the emergency bus can be avoided, thus averting the potential for generator shutdowns, paralleling problems, and nontransfer of loads critical to facility operation.
ATS operation and settings
There are four commonly used types of transfer switches in the electrical design industry:
- Standard open transition
- Delayed transition
- Closed transition
In healthcare applications, open transition and delayed transition ATSs are the most commonly used for loads required to be on the EES. Open transition switches transfer from the primary source to the secondary source with no overlap of the two sources. Delayed transition switches provide the same features with the addition of an adjustable time delay in a center “off” position. These two types of switches are commonly used for all three ESS branch loads.
Closed transition switches allow an overlap of the two sources during the transfer, basically paralleling the two sources together for a brief period. Note that the use of closed transition switches must be carefully coordinated with the local authority and utility provider. They are not allowed in some applications due to the potential for back feeding power onto the utility grid. Closed transition switches provide the benefit of eliminating “transfer blink.”
Bypass isolation capabilities can be added to the standard, delayed, and open transition switches to provide another level of maintainability and reliability. For optional loads, those not required by NEC 517 but desired to be on the EES, separate manual or ATSs can be used. These switches shall monitor the EES load to determine available capacity and controlled transfers at coordinated times to assure the system continues to operate, according to NEC 517.26.
The operating sequence of a standard ATS typically includes:
- Sensing the interruption of utility power
- Sending a start signal to the generator or generators
- Sensing that generator power is available
- Transferring load to the generator
- Sensing the return of utility power
- Retransferring load to the utility
- Sending a stop signal to the generator.
Per NEC 517 and NFPA 99, the transfer of power for life safety and critical loads must occur within the first 10 sec of the loss of utility power. Loads on the equipment branch are allowed to be delayed beyond the first 10 sec. These loads include air handling equipment, chillers, boilers, and pumps for chilled and/or heated water. Equipment branch delays can become very important when considering the overall system operation in large facilities or facilities that have had many system expansions.
Using switch delay capabilities for loads that are allowed to be delayed can provide engineers with opportunities to minimize large loads from hitting the generator bus at the same time. A large load influx can cause fluctuations in generator voltage, current, and frequency, which can lead to false ATS transfers, generators dropping off the emergency bus, or—in a multiple-generator paralleling system—generator synchronizing problems. If either the voltage or frequency is out of tolerance, the paralleling gear will not allow any generators other than the first to close on the bus. As a result, the first generator on the bus could become overloaded as more ATSs attempt to transfer. These cascading transfers and overloads can cause a catastrophic EES failure (see “Parallel-friendly ATS features and options”).
Common system designs
NEC 517 describes the required separation of loads in a healthcare environment into three branches of power: life safety, critical, and equipment. Each of the branches must have an ATS designated for serving loads on that branch. Depending on the load requirements for the project, more than one generator may be necessary to meet the needs of the project. Figures 2 and 3 differentiate a single generator system from a paralleled multigenerator system.
Another important consideration is that the life safety and critical branch ATSs must transfer load within 10 sec of a loss of the normal power source. This includes generator start time, which can range from 5 to 7 sec on a single generator system, to 6 to 8 sec on a large multigenerator system. In some cases, the use of the delay-to-engine-start signal to avoid momentary utility losses while starting the generators can exceed the 10-sec requirement.
If the project area doesn’t experience momentary utility losses often, it can be counterproductive to induce delay settings on the engine-start signal. In single-generator systems, the careful use of delay can make a big difference in the size of the generator. Proper load staging will give a smaller generator the ability to maintain proper frequency and voltage that would have otherwise required a bigger generator to maintain. It can also reduce the risk of generator failure while taking on ATS loads. As engines get more hours on them, they aren’t able to take large blocks of load as well as they did when they were younger.
A good application for load-shed relays on the emergency power system is when there is a fire pump in the system. Putting a load-shed relay on the least critical equipment branch or optional ATS allows more of the generator’s capacity to be used. If the fire pump is required to start, it will trip off the least critical ATS with a load-shed relay to provide the capacity to run the pump.
In large multigenerator paralleled systems, generator start and sync times are major system design considerations. It is important to consider the system functionality during start-up from a utility outage. Paralleling two or more large generators (1,000 kW and above) can take longer than 10 sec before energizing the emergency bus. This is especially true of generators with aftertreatment on their exhaust systems.
For this reason, it’s important to consider how the system operates with the first generator connected to the bus while the others are trying to “catch it.” This first generator provides the voltage that allows transfer switches to transfer. Any switch with a zero-delay setting will immediately transfer to the emergency source, adding load to the system in the middle of the synchronizing process. This added load onto a single engine can induce enough variation in the connected generator’s voltage and frequency that it slows down the process of synchronizing the remaining generators. This real-world scenario means that rarely will a paralleled system have two generators synchronized and connected to the bus in less than 10 sec.
When designing systems with multiple generators, engineers should consider designating one generator—sized to accommodate all of the life safety and critical loads—to be the first to energize the emergency bus. This allows the transfer of life safety and critical loads within 10 sec, while allowing additional generators to reach the required speed and voltage, and synchronize to the emergency bus.
Additional delayed transfer switches are then sequenced to transfer as the additional generator units are added to the bus. The equipment transfer switches must have a load-add/load-shed relay to achieve proper control from the paralleling gear in case of a generator failure. A key factor in this strategy is to maintain life safety and critical loads that are less than the single generator’s block-load capacity.
A common consideration for larger systems is to divide the life safety and critical branches among multiple smaller ATSs. This can decrease the block load that the generator system will see when normal power is lost. Even delays of 1 sec in these ATSs can result in a staggered starting process that a single genset can handle without dipping voltage and frequency. This careful use of load-transfer delay settings can create a systematic, controlled transfer of all essential system loads to the emergency bus within the code required time frame (see Figure 4).
Engineers should use additional ATSs to handle any other user-requested loads that the generator system must accommodate. These loads are typically imaging equipment; HVAC systems (other than those required on the equipment branch); and central plant equipment such as chillers, air compressors, and pumps. Many times the additional loads consist of large block loads that will hit the emergency bus all at once when transferred. These large loads can cause system voltage sags, which may affect downstream loads or cause ATSs to attempt to transfer if the tolerances are not set correctly. If possible, the loads should be separated onto multiple ATSs to allow staging of the loads onto the emergency bus by using the aforementioned delay functions.
Manual transfer switches can also be used to allow the engineering staff to monitor system operation and load, and to transfer noncritical loads under controlled circumstances. As the emergency power system size gets larger in a paralleled system, it is very important to ensure the loads are properly segregated in an organized manner. Allowing too many optional loads to creep onto the critical branch can create problems in the start-up sequence. A much wiser approach is to keep what absolutely needs to be on the emergency power system in less than 10 sec as small as possible, then add optional loads at delayed times after all the generators are synchronized and connected to the system.
Evaluating existing systems
A thorough evaluation is required when diagnosing and resolving ATS timing issues. A good place to start is to collect data in a spreadsheet of all ATSs, their age, make and model numbers, pole configuration, voltage, peak demand load, and all available settings. It is also important to understand the context of these ATSs. Questions to ask include:
- Are all of these ATSs connected to the same emergency power system?
- Are there multiple emergency power systems?
- Are any of these emergency power systems a paralleled generator system?
After gathering this data, evaluate the total load for each system, how that load stages into the system during start-up, and if there are any potential issues with the current state of the system. In most cases, it will take only a few seconds into the step-by-step load transfer process to understand how the system reacts as each ATS attempts to transfer. The same process should be followed for each failure mode within the system to understand how the system reacts to the failure of any one component. For example, consider what happens to the emergency power system if the fire pump starts, or if a generator fails in a paralleled system.
Testing, commissioning ensure proper system operation
Testing an EES at initial start-up of a new system is a necessity. All failure modes should be evaluated and tested to ensure the system operates as designed. This includes all the ATS timing. Whether the system is a single engine generator or a large paralleled system, there are many details that can be overlooked in the installation and programming that can lead to issues if not tested to ensure proper operation.
Testing the system while under load is the best way to simulate true operating conditions. The load can be imposed using a load bank for systems with load bank connection provisions, or with actual building load. Using the building load is preferred and gives a true indication of how the system will respond during a loss of utility power. Otherwise, it has not proven proper operation or been tested in all the failure modes. This is particularly important in paralleled systems.
When modifying an existing system or tying into an existing system, care must be taken to evaluate the changes and how they impact the entire system. As loads change in an existing facility, it is imperative that the system is commissioned periodically. It is easy to overlook the small renovation to a facility that may increase the emergency power load by small 10- to 50-kW increments, but repeated renovations over several years can add up, causing significant changes to the system. While it is impractical to re-commission an emergency power system for every small load added, you should not underestimate how these small changes accumulate over time into significant change.
Not only do you need to consider how the system performed during monthly testing, but you need to evaluate the testing procedure to see if it is truly simulating what happens in a utility power outage. Simulating a true utility power outage is very difficult to do in a fully functional hospital. The testing procedures cannot put any patients at risk. However, a procedure that does not periodically check all the system functions can give false confidence in the reliability of the system.
Too often the monthly testing procedures don’t put all the ATS settings to the test. A common procedure in a large paralleled system is to start all the generators manually, let them synchronize, then go to each ATS and put them into test mode one at a time. While this tests the functionality of the generators and paralleling gear, it does not simulate what happens in those critical seconds of start-up when ATSs are staging on the first generator and the other generators are trying to synchronize. It does not show if load is being delayed properly to prevent an overload condition or other potential failure modes of the system.
Simulating a true utility power outage each month may be excessive, but a full testing procedure should be developed for quarterly or annual tests. Often this can be done as part of a bigger disaster preparedness test for the facility. The Annex at the end of NFPA 110 provides individual component testing guidance that should be considered along with the manufacturer’s maintenance recommendations. However, testing of the system as a whole is monumentally important to ensuring it will perform when called upon.
It is easy to overlook important details on how systems operate in the moments following a loss of utility power. There are many considerations to weigh when determining proper ATS settings and timing considerations in an EES. The details must be coordinated and tested to make sure that patients and staff are not put at risk. This starts with a basic understanding of ATS features and their importance in the particular application.
The first step in a successful outcome is to determine the system configuration, loads on each ATS, and then decide the ATS settings you wish to implement. When working with an existing system, evaluating the entire system is necessary to ensure the project at hand does not necessitate the adjustment of existing system components.
The final and probably most important step is to test the system thoroughly and often. For hospitals, running the system weekly and loading the system monthly are required. These minimum testing standards required by code and AHJ may not be adequate to ensure proper operation when called upon to do so. The more complex the system, the more important it is to test all its required sequences.
Calhoun is a principal and senior electrical engineer in the Dallas office of ccrd partners. He has been with the firm for 17 years and has extensive experience in designing and constructing highly reliable electrical systems for healthcare, corporate, and data center facilities across the U.S. Montgomery is an associate principal and electrical engineer in the Denver office of ccrd partners. He has 13 years of experience designing highly reliable power systems for healthcare facilities and data centers. He is part of the electrical specification writing committee for the company.
Parallel-friendly ATS features and options
In addition to standard operating requirements, some ATSs offer delay- and load-shed functionality that can provide stability to the EES with multiple ATSs. The most common functions include:
Delay to engine start signal: This delay can help stop unnecessary generator starts if frequent momentary interruptions or voltage spikes and sags occur on the incoming utility power to a facility. The typical setting for this delay is 1 to 2 sec.
Delay to emergency: This delay coordinates the transfer of loads onto the emergency bus. It helps provide stability by easing load onto the generator system. The typical settings are less than 3 sec for life safety and critical loads and more than 10 sec for equipment loads and other nonessential loads on the EES.
Delay to normal: This delay is used to ensure the normal power source is stable before transferring from the emergency power source back to the normal (utility) power source. This delay can also help by easing load back onto the utility power transformers. The typical setting can be from 30 sec to 2 min or more.
Load shed: This function protects the generator system from overload conditions. If the load exceeds the generator capacity, low-priority ATSs such as optional equipment branches can be shed from the system.
Cooldown: This feature protects the engine and ensures proper engine cooldown after the load is transferred back to the normal power source. Typical generator cooldown is 5 min.
Voltage tolerances: Voltage tolerances protect downstream equipment that may have sensitive voltage requirements. For example, in UPS applications, voltage tolerances that are not set tightly enough can discharge the batteries without ever starting the generator.
Center off: This feature protects the equipment branch or ATSs that serve large motor loads from the inrush caused by residual transformer magnetism or out-of-phase rotating equipment. Close coordination with large mechanical system motors is imperative to ensure switching back and forth to out-of-sync sources does not damage mechanical or electrical equipment. Typical settings range from 5 sec to 2 min.
Typical ATS options include:
Auxiliary contacts: These can be used to determine ATS position remotely
Neutral conductor options: These options can include:
- Solid neutral, which is used for applications where the generator does not have a separately derived ground
- Switched neutral, which is used for applications where the generator is grounded separately from the utility
- Overlapping neutral, which diminishes voltage transients on the generator during a switching event
- Onboard ammeter and volt meters to determine instantaneous voltage and load
- Onboard power monitoring to determine instantaneous and historical data on voltage, frequency, and load.