Commissioning electrical systems in mission critical facilities
The overall goal of commissioning must be to ensure that a facility meets the design intent and the owner’s requirements. For critical facilities, this goal is generally achieved by proving to the owner that the reliability, redundancy, and resiliency that he or she paid for is indeed present and operational in the finished facility.
Because there are so many failure scenarios and variables, it is rarely possible or cost efficient to reasonably test each one, but the commissioning authority has an obligation to provide a level of testing that will allow the owner to feel confident that each system is working and capable of maintaining a proper planned operational state during common external events.
As expected, the owner will want to use the commissioning process to be certain that the installation, performance, and operation of new equipment is acceptable before it supports critical load, and he or she will strive to do this as cheaply and as quickly as possible.
This article explores the best practices for testing several electrical systems, as well as some of the challenges encountered. It also presents selected case studies observed during the functional performance testing phase of the commissioning process, as detailed in ASHRAE Guideline 0. Implementing these best practices and lessons learned on future projects will improve the quality of the product provided to the owner.
Including the generators in the commissioning scope for a critical facility is imperative because they are the only source of long-term standby power when the utility becomes unavailable (see Figure 1).
When testing a generator, it is best practice to ensure that the load for step loading and endurance testing has a power factor rating that matches the nameplate power factor on the generator, as the generator will be tuned and calibrated to operate best at its rated conditions. The manufacturer also will not likely be able to provide documentation on how the generator is expected to perform if the load used for testing deviates from the name plate conditions. The tuning and calibration is especially important when attempting a 0% to 100% step load, and often the system will not respond properly within acceptable tolerance if the power factor of the load does not match the nameplate rating.
Due to new EPA regulations, generators are now limited regarding the amount of pollution that they can emit under all running conditions, including when responding to step loads. This has been a challenge for generator manufacturers who in the past simply allowed the system to call for more fuel, which resulted in billows of black smoke entering the environment. In an effort to minimize pollution, manufacturers have had to finely tune the generators, resulting in the increased importance of testing the generators at rated power factor. In addition, because the generators are typically exercised under load for routine maintenance and testing, the owner often buys a permanent resistive load bank (unity power factor) sized for the rated capacity of the generator. It is important to explain to the owner that the permanent load bank that will be used for future load testing may not be appropriate to use during commissioning if it is rated at unity power factor.
Generator commissioning case study: Two 13.8 kV 3 MW generators that were rated for 0.8 power factor were each tested using a 3 MW unity power factor load bank. In each case, when conducting the 0% to 100% load step, the generators were able to support the load during only one out of seven attempts. The load was resistive, but the voltage drop induced by the step load caused the load bank controller to lose power, which shut down the load bank. Even when the load was maintained, the voltage and frequency deviated beyond the published criteria for 100% of the step load because the generator performance data was not based on a unity power factor load. This problematic operation was not observed for the same generators when they were tested at the factory using a 0.8 power factor load bank.
Automatic transfer switch (ATS)
The ATS is an important component of the critical facility because it is used commonly in critical facility designs to transfer power from a primary source to a secondary source after the loss of the primary source.
Open transition ATSs are designed to allow for an interruption to the load using a break-before-make transfer. Because of this, loading the ATS during open transition transfers during functional performance testing is not required. Load is also not required when testing an ATS’s ability to perform closed transition transfers. During closed transition transfers, the ATS will parallel the primary and secondary sources prior to transferring. It is important to ensure that the ATS can properly conduct closed transition transfers and will handle the transition in the same manner, regardless of whether it is carrying load or not. A power quality meter must be connected to the output of the ATS to confirm that the transfer is completed within the specified time for closed transition applications. It should be noted that load is required for all ATSs when conducting infrared scanning. It is recommended that all components of the ATS are infrared scanned under full load on all primary, secondary, and bypassed power paths after final installation is complete. Load is also required for closed transition applications when the secondary source of the ATS is a generator. This testing is usually conducted as an integrated system test to prove that the generator and ATS work properly together under full load. The integrated system testing is conducted after functional performance testing for the ATS, generator, and other integral systems is completed.
In most cases, for an ATS to be functionally tested, both sources must be available because the ATS will usually inhibit any transfer if there is only one source. This problem can arise in situations where ATSs are added to existing live facilities. Because of their integral role in the power distribution system, they often can’t be tied into the electrical system without bringing down the loads that they will serve. In an effort to minimize disruption to the live facility, the ATS testing will likely occur prior to connecting it to the live facility. However, the ATS can be connected to the secondary source if the secondary source is a generator. When the primary source serving the load is restored, there is usually limited time for testing the ATS as it will immediately be required to provide power to critical loads.
ATS commissioning case study: An ATS manufacturer was required to start up and test the ATS on a project before it was tied into the electrical system. To do this, the ATS vendor required both the primary and the secondary sources to be available for the start-up. The electrical contractor added a jumper between the two sources and connected the secondary source of the ATS to the generator. When the generator was started, the ATS saw both the primary and secondary sources as available. A major drawback was that there was no way to disconnect only the primary source during start-up without also simulating the loss of the secondary source, so it was not possible to verify automatic transfer operations without simulation techniques. The ATS also had a much easier time performing closed transition transfers because the two sources were perfectly synchronized, as they both came from the same generation point. All of the functionality was retested after the final tie-in during functional performance testing to ensure the system was operating properly in the actual design configuration.
The UPS is probably the most important piece of equipment in the critical facility because of its ability to maintain power to critical loads, regardless of the operation of all of the other supporting systems (see Figure 2).
Monitoring the inputs to the rectifier of the UPS, the static bypass within the UPS, and the UPS output bus is considered best practice during functional performance testing. After each transient, step load, or battery discharge test, the waveforms recorded by the power quality meters set up on the system should be reviewed to confirm that no events were triggered and that the output waveforms stayed within tolerance and recovered within the specified time frame. UPS systems are often placed into service quickly after functional performance testing, so it is best to check the power quality meter results—including waveform captures—during on-site testing rather than waiting for a report from the meter technician. This way, any problem discovered during UPS testing can be quickly rectified as the manufacturer often has to consult the factory on problematic internal UPS operation.
Full load endurance tests should be conducted on UPS systems after the system has been installed on-site, even if full load testing was conducted in the factory. Many components need to be disconnected for shipping and are then reassembled on-site. Electrical equipment can also be affected by problems that develop during shipping and may not be detected without performing the endurance test on-site. Generally, an 8-hr duration for a full load test is considered adequate to confirm that the system will be capable of functioning at full rated load without problems.
In some cases, it can be difficult to monitor the logic used by the UPS to handle various operations because the actions are carried out by microprocessors installed on circuit boards. This emphasizes the importance of properly setting up power quality monitoring equipment prior to testing the UPS. If a problem is detected during testing, the manufacturer will have a much easier time solving it if it is provided with significant data generated both by the UPS’s internal monitoring system and the external power monitoring equipment used during testing. When a failure occurs, it can be very difficult to understand what is happening inside the equipment. Captured test data almost always improves the issue resolution process.
UPS commissioning case study: While setting up the system configuration for a battery discharge test, both battery string breakers opened when load was applied to the batteries. The event that caused this response was retested twice with no anomalies noted. During further testing, the failure could not be recreated. The manufacturer replaced parts within the UPS that could have failed and caused the initial problem. After the replacement, the UPS was tested at a variety of load step changes and was transferred to static bypass, maintenance bypass, and back to inverter. An additional 2-min battery discharge at 65% load was then conducted while UPS screen calibration was performed. The manufacturer indicated that the repairs were successfully executed and that the system was operating properly, but was not able to explain why a crucial function within the UPS dramatically failed.
Generator paralleling switchgear
Generator paralleling switchgear is a crucial component to a critical facility in situations where the generator supported load exceeds the capacity of one generator (see Figure 3).
Generator paralleling switchgear systems should be tested at the rated power factor of the generator paralleling switchgear system—typically 0.8. This is important to show that each generator properly shares the kW and kVAR loads. Just because paralleled generators evenly share kW while serving a resistive load does not always mean that they will evenly share kVAR when serving a reactive load.
A major challenge with testing generator paralleling switchgear systems is that they are often rated for very heavy loads due to the number of generators that can be connected to them. In some cases, it may not be practical and may also be very expensive to load generator paralleling switchgear systems to rated capacity. It is recommended that enough load be provided so that it exceeds the capacity of one generator. Ideally, the load banks provided will be sized to the expected operational capacity of the generator paralleling switchgear, but not necessarily to its full design capacity.
Generator paralleling switchgear systems rely heavily on programming within the programmable logic controller (PLC) for operation. Knowledge of how this program operates is often limited to a handful of experts. Changes to PLC programming must be documented in a PLC programming change log. The log should include the date of the change, the reason for the change, a description of the change, and the new version number of the program that includes the change. Older versions of the program should be saved in the event that updates create additional problems and reverting back to an earlier version of the program is required.
Generator paralleling switchgear commissioning case study: After a generator paralleling switchgear system was tested, programming changes were made in response to issues discovered during the testing. Retesting was conducted, but only a portion of the tests were repeated. Later, during owner training, additional programming problems were discovered as a result of the changes made prior to the previous retesting. The PLC programming was changed again after additional tweaks were required. To be sure that both the PLC programming and the system were working properly, a retesting procedure was conducted including every possible user initiated transfer and automatic transfer in both open and closed transition scenarios. The retesting was video recorded and documented, and the PLC event log was extracted to show the transfers that occurred. The retesting was completed successfully with no additional programming changes required.
Main electrical switchgear
Main electrical switchgear is an important component to a critical facility because it distributes power to all of the downstream electrical distribution equipment.
Circuit breaker settings must be inputted, coordinated, tested, and verified throughout all main electrical distribution equipment. If there is a fault in the system, it is imperative that selective coordination is implemented so that the fault is isolated as far downstream as possible. Main circuit breakers must be properly set up to ensure that they will stay closed during fault conditions and wait for downstream equipment to clear the fault. This will be ensured by implementing proper National Electrical Testing Association-recommended circuit breaker testing including instantaneous pickup, short time pickup, short time delay, long time pickup, long time delay, ground fault pickup, ground fault time delay, contact resistance tests, and insulation resistance tests.
While main electrical switchgear is an integral part of the electrical distribution system, the system’s current carrying capacity may increase the arc flash hazard. To avoid injury, main electrical switchgear should be disconnected before it is opened or worked on. Because the owner will often not own a means of disconnect ahead of this equipment, it usually requires involvement from the utility provider, which can be problematic and difficult to schedule.
Main electrical switchgear commissioning case study: Modifications were required to be made to the main electrical switchgear that serves a data center site. To ensure that all modifications were made correctly, infrared scanning had to be conducted. Due to the current carrying capacity of the main electrical switchgear, it was not safe to be within 6 ft of the equipment when it was open, and opening it could be done only when the main electrical switchgear was not energized. This required the lengthy process of shutting down all of the loads in the building, opening the main electrical switchgear, and restarting all of the systems so the main electrical switchgear could be scanned at a safe distance under load. The same procedure had to occur to replace the covers on the main electrical switchgear after the infrared scanning was completed.
Static transfer switch (STS)
An STS is an important and useful component for a critical facility because it provides the ability to seamlessly transfer load during both failure and maintenance situations (see Figure 4).
STSs behave similarly to ATSs, but because they are designed to transfer within a few msec, there are several settings that must be coordinated. STSs are commonly fed from UPS systems. These UPS systems are present to prevent interruptions to the downstream STSs. During a planned maintenance event or during a utility power failure, the UPSs are designed to perform transfers to bypass or battery within a certain time frame. Because the STSs are set up to transfer on a loss of the primary source for a certain duration, the time frame must be longer than the allowable interruption seen from the UPS. If not coordinated properly, a routine transfer to bypass at the UPS level can cause the downstream STSs to transfer to their secondary source.
On several occasions, phantom voltage and current readings have been observed at the STS screens with no connected load. Rebooting the system typically corrects this problem. While the manufacturers generally indicate that there are no operational risks, this anomaly is puzzling.
STS commissioning case study: At a site containing eight STSs, one unit displayed current values on a single phase with open load breakers and no current was measured using portable power monitoring equipment. Another STS unit showed 160 A in this scenario while 0 A was measured with portable power monitoring equipment. The manufacturer assured the team that simply rebooting the screen would correct the problem and would not jeopardize the load in any way. Rebooting the screen did correct the problem, and the unit was monitored to ensure that the problem did not return.
Electrical power monitoring system (EPMS)
The EPMS allows all of the electrical systems within the critical facility to be monitored from a single location, giving the operator visibility to ensure that all systems are not generating any alarms and are operating properly and efficiently (see Figure 5).
When confirming that the EPMS is monitoring systems correctly, multiple states must be checked for each point. Points must be modified in the field and checked to ensure that the same values or statuses observed in the field are properly reported back to the EPMS.
One difficulty encountered in this area has to do with discrepancies with points. Design engineers typically specify points to be monitored by the EPMS, but they often approve equipment submittals that are unable to provide these points. To avoid this problem, it is best to meet with the design engineer and the equipment manufacturers prior to the acceptance of the submittals to ensure that the points that are important to the design engineer can be provided by the equipment.
EPMS commissioning case study: Many points monitored by the EPMS, including voltage spikes and sags, are very difficult to simulate. To simulate real voltage sags on a project, the electrical system was placed on generator and large step loads were added with a load bank. The generator struggled to maintain the voltage when required to carry the large step load, which resulted in voltage sag alarms and generation of waveforms captured at the EPMS.
The equipment in the electrical distribution system of mission critical facilities must operate dependably. After commissioning challenges have been resolved and best practices have been employed, these systems will meet the original design intent and owner’s requirements, ensuring the owner that the facility embodies reliability, redundancy, and resiliency.
Joshua J. Gepner is a senior associate at Environmental Systems Design Inc. He has more than 10 years of engineering experience focusing on design, consulting, and commissioning. He specializes in commissioning mission critical facilities and is knowledgeable in commercial, residential, and industrial electrical design as well as LEED and building energy code standards.