Safeguarding the Nation’s Critical Power

If it wasn't obvious before the 2003 Northeast blackout that high security requires highly reliable, uninterruptible electric power, it's certainly clear now. Fortunately, in the name of homeland security, there has been an unprecedented push to develop security measures that ensure operational continuity and protect the nation's critical infrastructures from natural and human-induced hazards.

By Paul LaPierre, EYP Mission Critical Facilities, New York March 1, 2004

If it wasn’t obvious before the 2003 Northeast blackout that high security requires highly reliable, uninterruptible electric power, it’s certainly clear now.

Fortunately, in the name of homeland security, there has been an unprecedented push to develop security measures that ensure operational continuity and protect the nation’s critical infrastructures from natural and human-induced hazards.

To meet the challenges presented both by cyber and physical threats, government and businesses are demanding more sophisticated technologies that will protect the critical nodes upon which these infrastructures depend. But as these new security and supervisory control and data acquisition (SCADA) technologies grow in effectiveness, so does their dependency upon a stream of reliable, uninterruptible electric power—critical power.

Within public policy circles, however, the issue of reliable power is considered primarily a utility concern and is largely assumed as a given in the critical infrastructure protection process. A February 2003 report from the federal government, “National Strategy for the Physical Protection of Critical Infrastructures and Key Assets,” identifies the country’s critical infrastructures and defines the energy sector as being comprised of two segments: electricity and gas/oil. Within the energy sector, the focus is on generation, transmission and distribution, and controls and communications. Oddly, in the event of a power outage, mitigation strategies within the critical nodes of the other infrastructure sectors are not addressed.

Taking stock

But with the increasingly widespread recognition of critical power as a third, crucial element to critical infrastructure protection, this is starting to change. The Northeast power blackout of August 14, 2003, illuminated the strengths and weaknesses of those industries that have invested in critical power and those that have not. The financial sector came through relatively unscathed, the transportation sector showed a real dichotomy in air travel and the telecom sector was spotty. Also, there were some near misses in the defense industrial base, and because the energy sector failed, so too did major systems in the water sector. What became abundantly clear is that although there is interdependence among all infrastructures, the backbone is not just power in general, but reliable, uninterruptible electric power.

The financial sector figured this out long ago and invested heavily in backup power. As a result, the U.S. Treasury Dept. congratulated the financial sector on its efforts and capabilities to survive the ordeals of the blackout.

In the transportation sector, the Federal Aviation Administration’s investment in redundant power capabilities allowed air traffic control to remain operational. Unfortunately, even though flights could move in and out of airports, the terminals did not have sufficient backup power. Air traffic ground to a halt as security systems were rendered inoperable and passengers could not be processed.

Communication via landlines was largely unimpaired, but communicating via cellular was a problem. AT&T, Verizon, Cingular and T-Mobile all reported problems, not due to capacity, but rather, to power. At many of the cell base stations and antennas, the backup battery power was designed to last only a few hours at best. And the increased calling volume reduced that capacity even more. According to T-Mobile, 30% of its cell towers were still out 24 hours later. “You don’t engineer a network for a massive power failure,” said a spokesperson for AT&T.

It’s been proven time and again that the grid is inherently vulnerable. That today’s high security environments cannot rely solely on it is not a new revelation. However, it is one thing to make this claim, and another to actually assess the issue and look for solutions. It was precisely because of this issue that a study on critical power was commissioned in April of 2003. (For the full text of the “Critical Power” white paper, go to www.eypmcf.com .) By coincidence, the paper was released two days before the 2003 Northeast blackout and gained wide circulation within the industry, as well as in policy circles of the critical infrastructure community.

The premise of the study is that while the public grid must certainly be hardened and protected, most of the responsibility for guaranteeing critical power supply at large numbers of private grids and critical nodes ultimately falls on the private sector, as well as on local tiers of the public sector.

A new risk profile

Many essential services and businesses have critical power needs that have not been properly addressed, often because they have never been systematically assessed. Even organizations that prepared properly for yesterday’s power interruptions may be unprepared for today’s . The risks of the past were relatively benign threats of routine equipment failures, lightning strikes on power lines and small-scale hazards such as squirrels chewing through insulators or cars colliding with utility poles.

The possibility of a deliberate attack on the grid, however, changes the risk profile fundamentally, and that possibility sharply raises the risk of relatively long-term outages that extend over wide areas.

Tiers of power

Even before 9/11, it had become clear that the digital economy requires a level of power reliability that the grid alone simply cannot deliver. Utilities have traditionally defined an outage as an interruption of five minutes or more, but digital hardware cannot tolerate power interruptions that last more than milliseconds. Backup generators, uninterruptible power supplies and standby batteries have already been widely deployed. About 80 gigawatts of off-grid backup generating capacity already exist, which is an installed base equal to about 10% of the grid’s capacity. And roughly 3% to 5% of the public grid’s capacity is currently complemented and conditioned by UPS equipment, with about 25 GW of large UPS capacity in businesses and government buildings, and another 10 to 15 GW of capacity in smaller desktop-sized units located in both businesses and residences. End users as well have installed more than 30 million large standby batteries.

Until recently, the deployment of much of this hardware has been directed at power quality —smoothing out spikes and dips that last for only fractions of a second—or short-duration issues of power reliability, with grid outages lasting from minutes up to an hour or so. In the new geopolitical environment, however, planners must address the possibility of more frequent grid outages that last for many hours, days or even longer. Assuring continuity during extended outages requires a different approach and a different level of investment in local power infrastructure.

Much of the critical infrastructure literature refers to the grid as a single entity, and thus, implicitly treats it as critical from end to end. But the first essential step in restoring power after a major outage is to isolate faults and carve the grid into much smaller, autonomous islands. From the perspective of the most critical loads, the restoration of power begins at the bottom: On-site power instantly cuts in to maintain the functionality of the command and control systems that are essential to coordinating the step-by-step restoration of the larger whole.

The bolstering of the grid begins at the top tier, which is in the generation and transmission facilities. Much of the modern grid’s resilience is attributable to the simple fact that “interties” weave regional grids into a highly interconnected whole so that any individual end user can receive power from many widely dispersed power plants. Incidentally, this architecture also increases everyone’s vulnerability to far away problems.

Large end users rely on similar intertie strategies to help secure their specific critical power needs. Substations, deeper in the network and closer to critical loads, can also serve as sites for deployment of distributed generating equipment. With the addition of its own generating capacity, the substation is “sub” no longer, but rather, becomes a full-fledged mini-station.

Opportunities for deploying new generation at this level of the grid—either permanently or when emergencies arise—are expanding, although still greatly underdeployed. Utility-scale, mobile “generators on wheels”—either diesels or turbines—offer an important additional option. Some substations already play host to small parking lots worth of tractor-trailers, each carrying 1 MW to 5 MW of generators. In the longer term, other sources of substation-level generation and storage may include fuel cells and massive arrays of advanced batteries.

For the most critical loads, however, none of these options is an adequate substitute for on-site backup power. On-site power begins with on-site supplies of stored electrical, mechanical or chemical energy—typically mediated and controlled by the high-power electronics and controls of a UPS. Rechargeable batteries remain the overwhelmingly dominant second source of power. However, batteries store far less energy per unit of volume or weight than do liquid hydrocarbon fuels.

Thus, to cover the threat of longer grid outages, the backup system of choice is the standby diesel generator. Sized from tens to thousands of kilowatts, diesel gensets can provide days, or more, of backup run time with the limits determined only by how much fuel is stored on-site, and whether supplies can be replenished. Diesel generators are strongly favored over other options, because they strike the most attractive balance between cost, size, safety, emissions and overall reliability. And the far-flung, highly distributed infrastructure of fuel-oil storage tanks is effectively invulnerable to the kinds of catastrophic failures that could incapacitate power lines or gas pipelines across an entire region.

To complement the hardware, monitoring and maintenance play a key role in maintaining power reliability from the gigawatt-scale tiers at the top of the grid down to the UPS and individual loads at the bottom. Real-time control plays an essential role in the stabilization of still-functioning resources, and the rapid restoration of power to critical loads after a major failure in any part of the grid. At the grid level, SCADA are used by utilities and transmission authorities to monitor and manage distribution. Likewise, at the user level, all the power hardware depends increasingly on embedded sensors and software to monitor and coordinate. This is a non-trivial challenge as problems happen at the speed of electricity.

Reliability-centered maintenance, which is common in the aviation industry but still a relatively new concept for power, is becoming more important with the rising complexity of systems. Some of the most useful critical power investments thus center on routine upgrades that replace older equipment with state-of-the-art hardware, which has built-in digital intelligence and monitoring capabilities. Changes as seemingly simple as speeding up the performance and automating of circuit breakers can greatly lower the likelihood of serious continuity interruptions precipitated by the power-protection hardware itself. Also, sensor- and software-driven predictive failure analysis is now emerging and will certainly become an essential component of next-generation, reliability-centered maintenance.

Resilient Design

One of the most important but least appreciated challenges in the critical power arena is determining just how robust and resilient supplies of power actually are. It is easy to declare a power network reliable but difficult to ascertain the actual availability metrics. The aviation and nuclear industries have spent many decades developing systematic, quantitative tools for analyzing the overall resilience of alternative architectures and are continuously improving the best ones.

But tools of probabilistic risk analysis—essential for any rigorous assessment of reliability and availability—are still widely underused in critical power planning. Employed systematically, they require power engineers, statisticians and auditors to physically inspect premises, analyze multiple failure scenarios, draw on hardware failure-rate databases and incorporate both human factors and external hazards. Proper critical design takes into account the key, though frequently overlooked, distinction between power reliability and the actual availability of the system thus powered. The analytical tools and technologies required to engineer remarkably resilient, cost-effective power networks are now available. The challenge is to promote their intelligent use when and where they are needed.

Private investment, public interest

Significant parts of the private sector were making substantial investments in backup power long before 9/11, because they realized that electricity is essential for operating most everything in the digital age and because the grid cannot provide power that is sufficiently reliable for many very important operations. Backing up a building’s power supplies can be far more expensive than screening its entrances. But improving power improves the bottom line by keeping computers lit and the assembly lines running. Likewise, in the public sector, secure power means better service.

Even though such investments are made for private or local interests, they strengthen the public grid as a whole. In the event of a major assault, power restoration for all is sped up and facilitated by the fact that some of the largest, most critical loads can take care of themselves for hours, days or even weeks.

Even more importantly, the process of restoring power system-wide has to begin with secure supplies of power at the critical nodes. Coordinating the response to a major power outage requires functioning telephone switches, 911 centers and police communications. The grid itself can’t be re-lit unless its supervisory control networks remain fully powered. The most essential step in restoring power is not to lose it, or at worst, to restore it almost immediately at key nodes and subsidiary grids from which the step-by-step restoration of the larger whole can proceed.

Finally, in times of crisis, not only can private generators reduce demand for grid power, with suitable engineering of the public-private interfaces, they can feed power back into limited parts of the public grid. Options for re-energizing the grid from the bottom up are increasing as the high-power switches and control systems improve.

The most effective way for government to secure the nation’s critical power infrastructure is to encourage private sector investment in critical power facilities—and not just by the relatively small numbers of quasi-public utilities and large federal agencies, but by private entities and state and local government agencies. Dispersed planning and investment is the key to building a highly resilient power infrastructure.

In sum, it has become abundantly clear that the enormity of the Northeast blackout of 2003 drove home two points. The first is that without power, even the best defenses are rendered useless. Secondly, the reliability requirements of increasingly complex technical facilities and environments cannot be met by the country’s inherently vulnerable electric power grid. These realizations present opportunities for the deployment of critical power hardware that supplies exceptionally clean and reliable power to the critical nodes of the digital economy, and that guarantees operational continuity for the duration of the extended grid outages that both deliberate and accidental assaults on the infrastructure might cause.

The Road to a Better Power Grid

There are eight major areas in which policy makers, industry associations and end users in the public and private sectors can look to improve the nation’s power infrastructure.

Assess vulnerabilities. Policymakers should lead and coordinate the efforts of user groups, critical power providers and utilities to conduct systematic assessments of critical power vulnerabilities for specific industries, utility grids and configurations of backup systems.

Establish critical power standards for facilities used to support key government functions. Federal and local organizations should work with the private sector to establish guidelines, procedures and, in some cases, mandatory requirements for power continuity at private facilities critical to government functions.

Share safety and performance-related information, best practices and standards. Utilities, private suppliers and operators of backup power systems should develop procedures for the systematic sharing of safety- and performance-related information, best practices and standards. Policymakers should take steps to facilitate and accelerate such initiatives.

Interconnect public and private SCADA networks. The SCADA networks operated by utilities and the operators of backup power systems should be engineered for the secure exchange of information in order to facilitate coordinated operation of public and private generators and grids. Policymakers should take steps to facilitate and accelerate that development.

Secure automated control systems. The necessary integration of SCADA networks operated by utilities and the owners of backup power systems requires high assurance of cyber-security of the networks in both tiers. Policymakers should take steps to advance and coordinate the development of complementary security protocols in the public and private tiers of the electrical grid.

Share assets. Policymakers and the private sector should take steps to promote the sharing of “critical space” for on-site generation and power-conditioning equipment, and to advance and coordinate the establishment of distributed reserves and priority distribution systems for the fuel required to operate backup generators.

Enhance interfaces between on-site generating capacity and the public grid. Improve technical and economic integration of on-site generating capacity. The public grid can also back up critical loads, lower costs and improve the overall resilience of the grid as a whole.

Remove obstacles. Private investment in critical power facilities creates public benefits. Thus, policymakers should explore alternative means to remove obstacles that impede private investment in these facilities.