Guidelines for data center grounding and bonding

Data centers have some very specific and unique requirements for grounding and bonding that differ significantly from the typical electrical distribution system in other types of facilities. These include:

• Grounding and bonding of equipment in the data center

• Grounding of the building distribution system

• Connection of the two systems as required by the National Electrical Code (NEC) Article 250

NEC REQUIREMENTS

NEC Article 250 requires that the main electrical service be connected to a grounding electrode. Where available, this grounding electrode must be the metallic, incoming domestic water service line and this connection must be made within 5 ft of the water line entrance into the building. If the water usage is metered, a metallic jumper must be installed using listed grounding clamps so the water meter can be removed without affecting the integrity of the building grounding. In addition, driven ground rods and connection to the building steel structure or reinforcement rods in the concrete structure must be provided as supplementary grounding electrodes. All of these grounding electrodes must be connected together at a single point, and the composite of all ground electrodes must have an impedance of less than 5 ohms to earth ground. If this impedance exceeds 5 ohms, then additional, driven ground rods or a chemically assisted grounding electrode must be added to reduce this value to below 5 ohms. The neutral of the incoming electrical service must be connected to the building grounding system at a single point, using a removable connection, which will allow service personnel to remove this jumper and ascertain if the neutral system is connected to ground at any other point at the incoming electrical service. (Multiple connections between the neutral and ground can cause ground loops and associated operating problems beyond the scope of this article.)

Many equipment manufacturers (of data processing, medical diagnostic, and other sensitive electronic systems) insist that their equipment have an isolated ground, separate from the rest of the building and not connected to the grounding electrode for the main electrical service. While having an isolated ground will reduce system noise, it can create a hazardous condition. For example, if an equipment malfunction causes a voltage differential between the building grounding system and the isolated ground of the specialized equipment, there is a potential for shock when someone contacts the two, supposedly grounded surfaces.

Therefore, the NEC has always required that all grounding systems within a facility be connected electrically to prevent hazardous conditions. It requires a solid, electrically continuous connection between the building ground and the “isolated” ground of each special system.

THE BASIS FOR GROUNDING

The NEC requires grounding for three main reasons:

• Safety. Assuring that all non-current-carrying, metallic components of the electrical distribution system are grounded reduces the opportunity for a conduit, panel door, or other item to be above ground.

• Overcurrent operation. A solidly grounded system contributes to quicker activation of the overcurrent device; when there is a short-circuit from a phase conductor to ground, the breaker or fuse sees a greater fault current and opens more quickly. (To see how this works, get a time-current for a fuse or circuit breaker and look at how quickly the device operates at four times its current rating versus how quickly it operates at 10 times its current rating.)

• Equipment operation. Grounding facilitates the operation of equipment where the ground is used as a signal reference ground. If there is a problem with having a voltage on the ground, the signal reference point is at an elevated potential and equipment operational problems can arise.

SPECIAL CONSIDERATIONS

Grounding is critical for the proper operation of data processing equipment, whether the system is connected directly to the building’s electrical distribution system or connected to a sophisticated power backup system consisting of UPSs, generators, power distribution units (PDUs), isolating transformers, and harmonic filters. A solidly grounded system eliminates static electricity, which can damage sensitive electronic components, and provides a means for preventing personnel injury due to voltage variances among various types of data processing equipment and their enclosures. An equi-potential grounding system creates a noise-free, signal reference point so that data transfers among the many data processing components have the same ground reference, which greatly reduces data transmission errors among the components.

Because most data processing systems use some type of standby generator or UPS system, they are considered to be separately derived systems (see the definition in NEC Article 100.I) and, therefore, must be separately grounded in accordance with NEC Article 250-30. However, the quality of the grounding electrode required by the NEC is inadequate for the requirements of a data processing system. (Connection to the building structural steel is considered to be adequate.) The impedance of building steel to ground is highly variable and affected by how the foundation is connected to the steel, whether the joints are bolted or welded, how much oxidation is on the steel at the joints, how tightly the joints are bolted, and many other factors.

Therefore, the first goal for a data processing grounding electrode is to assure that the impedance to ground is as low as possible. Methods for attaining low impedance include:

• An array of grounding electrodes spaced at least 10 ft apart

• A buried ground loop with multiple ground rods

• Chemically enhanced grounding electrodes

• Ufer grounds or other types of grounding systems.

Once the grounding electrode has been established (location, low impedance, minimum length, configuration, etc.), all electrical distribution components associated with the data center should be connected to a central ground bus. This ground bus is an insulated, isolated ground bus and is intended for a signal reference ground—not an equipment, safety ground. It will be connected to the isolated ground bus in the UPS, the PDUs, the branch circuit panels, and the isolated ground conductors going to the individual server racks. This bus will be completely isolated within the data center room from the green-insulated or bare, safety-grounding conductor that is connected to the conduit, boxes, panel board cabinets, server rack frames, UPS enclosures, generator frames, and the like. It is critical to the data center grounding system’s integrity that there are no connections between these two systems within the data center and, in fact, no connections anywhere—with the exception of a single bonding jumper specifically located to minimize the interactions between the electrical distribution system and the data center ground system.

SYSTEM INTEGRATION 

It may seem as if there is no way to satisfy the NEC grounding requirement that all of the grounding electrodes within a facility be bonded together and still provide a reliable, low-noise, low-impedance, virtually isolated, signal reference ground for electronic systems, but there is. The key to satisfying this dichotomy is to use a simple current divider from “Introduction to Electricity 101” (see “Functions of a current divider,” page 10).

By providing two separate low-impedance grounding electrodes (one for the main electrical service and one for the sensitive electronic system), and connecting them with the smallest bonding conductor permitted by the NEC (i.e., the conductor with the highest impedance), you’ll achieve this goal.

Because more of the current will flow toward the lower-impedance ground, and less will flow toward the higher-impedance bonding conductor connecting the electronic system grounding electrode, the harmonics and associated voltage variations will flow away from the electronic system. Likewise, a low-impedance grounding electrode in the electronic system will give a better signal reference ground for the electronics, while the influence of the harmful voltages on the electrical service ground will be reduced to a minimum.

Pinpointing UPS failure

One of our long-time clients asked us to help determine why several of the UPS systems serving its local PBX (a group of servers connected to operate the on-site phone system) failed.

The systems were divided into four server cabinets; each was fed by a separate UPS and all four were fed by a single, 120/208 V branch circuit panel located within the space. On the line side of the panel was a completely original transient voltage suppressor.

The installation had been commissioned within the prior 12 months; the UPS feeding one of the server cabinets had failed twice, each time causing the phone system to fail. In each case, the UPS failed during a local thunderstorm. The UPS supplier’s technician suspected the failures were due to high-voltage spikes brought into the room on the main power feeder to the panel. In addition, he observed a high-voltage condition (over five volts above earth ground) on the ground bus of the server rack associated with the failed UPS, which he felt supported his conclusion that a defective power distribution system was the cause of the UPS failures.

As we investigated the installation we found:

The surge suppressors were more than 25 years old and had never been replaced.

The grounding conductor that was contained within the feeder conduit to the branch circuit panel was not connected to the ground bus within the panel or to the transient surge suppressor.

There were four driven ground rods spaced some 12 in. on center in a row along the wall below the panel. These ground rods were driven through holes drilled through the concrete floor and into the undisturbed earth below the building.

All ground conductors for the UPS systems, the server systems, and all other equipment within the PBX room were connected to these four ground rods. The rods were connected with listed ground clamps and then to a copper ground bus located to the side of the ground rods, on the wall.

The main electrical service to the building was some 1,200 ft away, and there was no connection at the main electrical service to either driven ground rods or the incoming cold-water service.

The step-down transformer feeding the branch circuit panel had a single connection from the secondary neutral point to the 0.5-in. cold water line three floors below the PBX room (about 75 ft in total circuit feet).

It was evident that the surge suppressor was not functional, the grounding was not in accordance with the NEC, and the isolated ground system was inadequate for the data system. We recommended the following:

• Test the surge suppressor and repair or replace it based on the findings
• Bring the grounding system up to current NEC requirements by adding both driven ground rods and a cold-water service connection at the incoming electrical service
• Reconnect the panel ground connector that was routed to its upstream distribution panel.

We also concluded that the four driven ground rods in the room were virtually useless. The building they were located in had been constructed around 1910, and the room was more than 100 ft from the closest outside wall. In addition, the ground water table was low in the area. As a result, the rods had been driven into earth that was virtually devoid of moisture. With the rods being only 12 in. on center, they acted as though there was only a single ground rod, not four (ground rods should be at least 5 ft—and preferably 10 ft—apart in order to achieve a resulting decrease in the impedance to ground).

The manufacturer’s rep made all of our recommended changes within the room (but the service grounding was not corrected at that time). In the process, he discovered that the locking receptacle that connected the failed UPS to the server rack had a damaged neutral conductor that allowed a direct connection from the neutral to the grounded receptacle box. This connection between the neutral and unattached ground allowed the harmonics from the server power supplies to be impressed onto the data center’s ground system, thus elevating the ground system voltage. If the power system had been properly grounded and bonded, this high-impedance isolated ground system could not have existed and the harmonics would have been properly routed to S a low-impedance ground point, thus preventing the operating problems.

Functions of a current divider

A basic current divider works as follows: The voltage source enters a T circuit with a low impedance on the left and a high impedance on the right. Current entering the leg of the T will divide in accordance with the inverse of the impedances (i.e., more of the entering current will flow into the low-impedance circuit and less will flow into the higher impedance side).

The Ufer ground

In the early 1940s, Herbert Ufer developed the Ufer ground, a method of obtaining a low-impedance ground in more arid soils without the use of one of the relatively costly, chemically enhanced grounding systems. Ufer found that connecting a grounding conductor to the rebar within the cast-in-place, concrete foundation created a better ground (one with lower impedance) at a lower cost than that which could be achieved by using multiple, driven ground rods. Because concrete has a tendency to be relatively hygroscopic, the foundation retains more moisture than the surrounding soil, producing a lower impedance ground than most other grounding methods.

Disadvantages of the Ufer ground are that it is not as effective when used in shallow, spread-footing foundations, and the rusting of the reinforcing bars causes an increase in the impedance to ground. (Iron oxide has a much higher resistance than the reinforcing bars.)

One way to address these problems is to use a bare copper conductor in place of bonding to the reinforcing bars in the concrete. Here’s how it works: Prepare a coil of several feet of 3/0 bare copper wire and place the coil of wire at the bottom of one or more of the drilled piers that are part of the building’s foundation, prior to the pier being poured. Extend the conductor out of the concrete, protecting it from damage during and after the concrete pouring process, for future connection to the ground bus.

For a building that has another type of foundation, an effective alternative is to simply drill an 8- to 12-in. diameter hole, 20- to 30-ft deep, drop the coil of copper wire to the bottom of the hole, and fill the hole with less viscous concrete up to grade, leaving the copper tail protruding for future connection to the ground bus.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.