Water proof

By Rajan Battish, P.E. Principal, RTKL Assocs., Baltimore October 1, 2006

When Hurricane Isabel hit Baltimore in 2003, the downtown was flooded, crippling the area’s infrastructure and severely damaging many buildings. The Candler Building, a historic structure located just one street from the bay, was one of those victims. In fact, its basement was flooded with four and a half feet of water, resulting in a total loss of power.

The building owner, HRPT Properties Trust, turned to RTKL’s Applied Technology Group to get the building back online as quickly as possible. While we were able to do so in 24 hours by bringing in temporary generators and substation transformers, a longer-term solution was definitely required to avoid future interruption.

Part of the Baltimore harborside landscape since the early 1900s, the Candler Building once housed one of the first Coca-Cola bottling plants and, in fact, is named after one of Coca-Cola’s founders, Asa Candler.

In the mid-30s, the 13-story building was converted into an office building to serve as the first headquarters of the newly-formed Social Security Administration. In fact, the building was chosen because, at the time, it was among the few existing facilities in the Baltimore/Washington, D.C. area that was large enough to store the SSA’s enormous amount of equipment and vast number of records.

Today, the 560,000-sq.-ft. facility serves as a technology hub for an impressive list of tenants, including MCI, Qwest, Constellation Energy, Expedia and Verizon.

High-tech expectations

With these companies requiring 24/7 power, the expectation is a higher level of reliability than for a standard office building. Consequently, in the aftermath of the flood, a $4.5-million electrical upgrade was initiated. By implementing the program in phases, work could be completed without affecting existing operations.

Even though individual tenants have a multitude of backup generators and UPS systems to support their businesses, the building’s main power source already included a redundant 13.8-kV service from the utility. In 2003, however, when the flood waters reached the loading dock of the building, the water poured into the lower level through vents and conduits. The water level quickly reached as high as 50 in., causing the primary switchgear and associated distribution transformers, as well as the building’s secondary switchgear, to shut down. To maintain continuity of operations for the mission-critical spaces, backup generators were used for temporary power until the electrical infrastructure could be re-energized.

Engineering support for temporary power was subsequently provided, as was a report summarizing recommendations and options available for dealing with the damage. Since most of this infrastructure was exposed to bay water, concerns were raised regarding the reliability of the existing equipment and the long-term effects on the supporting electrical substations, busway, transformer, switches, contacts, relays and other electrical equipment.

Furthermore, at the time of the flood, the age of the facility’s electrical substation dry-type transformers was approximately 15 years old, so the probability of failure had already increased simply by virtue of age. As a point of reference, a liquid-filled transformer’s failure rate typically increases by approximately 40% when it reaches a period of 11 to 25 years of operation.

Consequently, the equipment’s age, combined with exposure to bay water, created concerns that degradation of the equipment life may have been accelerated. There was also an inherent risk that further issues could surface over time, such as failure of contacts and control wiring and an increased risk of a breakdown in bus insulation. This could cause a fault on the entire system.

Finally, the flood water caused grime to form inside transformer windings, weakening the equipment’s ability to withstand faults, no doubt affecting its IEEE-required rating for a basic impulse level.

Out with the old…

Previously, primary electrical service in the Candler Building had been provided as a main-tie-main configuration with 15-kV load interrupter switches, as well as distribution to building secondary manual-operation double-ended (1A and 1B) and single-ended substations. The primary switchgear also provided power to a single-ended substation transformer (2A) via 2,500/3,325-kVA dry-type transformers to step down the voltage from 13.2 kV to 480/277 volts.

The double-ended substation was rated for 4,000 amps, 480/277 volts, 3-phase and 4-wire with 2,500-kVA dry-type transformers designated as 1A and 1B. The double-ended substation was designed as a main-tie-main with fused switches as overcurrent protection devices. The mains of the double-ended substation were rated at 4,000 amps along with bus of the substation. The tie, however, was only rated at 2,000 amps, thus limiting the load transfer between bus A and B of the switchboard.

The single-ended unit substation was rated at 4,000 amps, 480/277 volts, 3-phase and 4-wire with a 2,500-kVA dry-type transformer designated as 2A. The unit substation had an isolation disconnect switch rated at 2,000 amps tied into double-ended substation 1A via a local 2,000-amp disconnect switch in the double-ended substation.

The double-ended substation provided power to the north section of the building. A 3,000-amp, 480/277-volt distribution plug-in busway provided power to floors one through seven from the 1A side of the double-ended substation. Section 1B of the double-ended substation provided a 3,000-amp, 480/277-volt feeder bus to the seventh floor and then became a distribution plug-in busway for floors eight through 12. The single-ended unit substation (2A) provided power to the south side of the building via a 3,000-amp, 480/277-volt plug-in busway from floors one through nine.

The existing life-safety generator, at 750 kW, was damaged by the flood waters along with the transfer switches. Consequently, a temporary trailer-mounted generator was rented to provide life safety for the duration of the clean-up and construction until a new system was installed.

…in with the new

Thanks to the upgrade following the flood, the capacity of the facility’s new generator, installed on the first floor with a remote radiator in the loading dock area, was increased from 750 kW to 1,000 kW with a longer fuel runtime—24 hours at full load. Sound-attenuated intake louvers were also designed, along with a sound baffle in the generator room, to minimize sound transmission to the rest of the building. The generator exhaust was run through the loading dock and out to the street at an elevated height. The remote radiator was hung from the loading dock ceiling, and the exhaust was likewise ducted out to the street at an elevated level.

New primary and secondary switchgear for building loads were provided in compartmented rooms for increased reliability. New primary power, from the utility manhole to the new first-floor electrical rooms, was also provided. This complemented the building’s four 15-kV service feeders, two of which are still located in the basement to serve the existing switchgear, and a second set that provides power to the new switchgear on the first floor. The primary service feeders are rated at 13.2 kV with main-tie-main configuration and motor-operated load-interrupter switches. Load-interrupter switches provide power to four redundant cast-coil transformers for 480/277-volt building distribution. The transformers are sized for 150% overload capacity and installed in compartmented rooms.

Two new main-tie-main substations are provided on the first floor to carry the base building loads at a distribution voltage of 480/277 volts. The substations were installed in compartmented rooms and configured for redundant bus, where one transformer serving the switchgear bus can support the entire switchgear load (bus A and B) in an overload mode.

As far as existing feeders, they were tapped and extended from the basement, above the observed water level, and run to the new switchgear on the first floor. As for the work schedule, the cutovers were limited to weekends and nights, and temporary generator power was provided for critical loads.

The transfer scheme between the A and B bus for each switchgear is an automatic and open transition. The transfer scheme interfaces with six automatic transfer switches, associated with a new generator, to provide the best source to the building load. It also ensures that the life safety requirements are met. Thus, a thorough factory testing of the sequencing and full level 4 and 5 commissioning were performed to ensure proper operation of the switchgear. The emergency feeders are also tapped and extended to new automatic transfer switches and associated controls.

Key challenges

As for the construction itself, challenges ranged from physical problems to technical concerns to management issues. For instance, during the initial flooding, the water needed to be pumped out. Multiple 50-hp pumps were brought in for this purpose. However, since the bay was only one street away, the pumps could not keep up with water inflow. Consequently, power had to be shut down, and the building evacuated until the storm departed and water subsided. Even so, the clean-up effort was Herculean due to all of the debris.

Not only did it have to be removed, but at the same time, the electrical equipment had to be cleaned and retested for power restoration. Tenants utilized emergency generators to keep their mission-critical loads operational. Temporary generators were brought in to back them up due to concerns of extended runtime.

After retesting and cleaning up the electrical equipment, new panelboards and a temporary generator for life safety were provided due to extensive water damage to the life-safety generator. The temporary generator stayed at the building until the new genset was installed and commissioned for proper operation.

To get the damaged equipment operational, the NEMA standard for handling water-damaged equipment was used as a guideline, along with recommendations from the testing agency and vendors, to determine which equipment could be reused. Temporary transformers were also provided as backup to the base building transformers.

During the design process itself, many concerns arose regarding the location of new switchgear and equipment. Urgency for the new construction became more apparent when electrical equipment that appeared to be operating properly started to fail. As an example, the temperature on the dry-type transformers increased with the fans on the transformers failing.

Consequently, various options were investigated to address the water-damaged equipment. Items discussed were replacing the existing equipment and reliability of the systems proposed. To increase the reliability and redundancy of the electrical systems, new dual 13.2-kVA service was routed to the first floor of the Candler Building, above the 100-year flood plain. Concerns such as floor loading—typical commercial buildings are about 70 lbs. per sq. ft.—accessibility, noise, vibration and the like needed to be addressed due to the fact that the building is a commercial facility with office tenants above the future electrical room.

Fortunately, the floor-loading issue was simplified. Because the original structure was a bottling plant, floor loading was 120 lbs. per sq. ft. This being the case, the structure was sound for installation of transformers, and special vibration isolators were installed for the generators. New egress and maintenance paths were also installed for future replacement of the electrical equipment. The new equipment was installed through knockout windows on the first floor along with the loading dock area.

Additional concerns were generator exhaust, intake louvers, radiator exhaust and fuel for the generator. The existing single-wall fuel tank was removed, and a new concerted encased fuel tank was installed in the basement. Also, the fuel tank was anchored in the event of the basement flooding again. To make the fuel available to the generator on the first floor, duplex pumps were installed on the first floor and away from the potential water in the basement.

In order to make sure the facility operated as designed, level 4 and 5 commissioning was performed, and interfaces with existing building life-safety systems were tested. The testing was performed on weekends to minimize impact on operations of the tenants. Temporary generators provided power to the critical loads that could not be impacted by commissioning of the switchgear and interface of the switchgear with life-safety systems.

Taking responsibility

At the end of the day, the Candler Building’s owner recognized the critical need to take steps to ensure that business would continue in the event of another weather surge and the potential damage that another flood could cause to the infrastructure. By relocating the entire electrical infrastructure to the first floor, which is very uncommon, the building’s high-tech tenants can now place more faith in the electrical reliability of their historic facility, a stone’s throw away from the bay.

Rack-Mounted Transfer Switches

One of the newest trends in computer-intense environments is the use of dual-corded power supplies to servers. While this provides added reliability, one of the issues, according to Daren Shumate, P.E., LEED AP with the Washington D.C., office of RTKL, is what to do in a retrofit situation when you have single-corded legacy equipment and the customer wants redundancy. This is an especially sensitive issue, he says, as these systems are also more susceptible to a single point of failure if they’re being used with traditional static automatic transfer switch schemes, because the loss or malfunction of an SATS means everything downstream is lost.

A solution to this problem, he says, is to employ rack-mounted static transfer switches at each server rack. What’s really advantageous about these devices, according to Shumate, is that they can be procured through facilities during construction or through IT during normal server installation.

—By Jim Crockett

The Birthplace of Social Security

The Social Security Administration began operations in 1936 at the Candler Building, on the harbor in downtown Baltimore. Converted from a Coca-Cola bottling plant, the facility served as the site of Social Security’s operational headquarters until 1960, when a new headquarters facility was opened in the Baltimore suburbs. SSA chose the building because, at the time, it was the only facility in the area large enough to hold its massive amount of equipment and stored records.

In November 1961, on the 25th anniversary of the building’s opening, SSA issued a special issue of the Bulletin of the Division of Accounting Operations. The special issue, available online at