Developing a SCADA master plan framework

The significant role that technology plays in today’s water and wastewater industry is widely known. The proper choices and implementation of the supervisory control and data acquisition system’s components to ensure the continuity of the facilities operation and success is essential.

The obsolescence of automation infrastructure, the shrinking of technology life cycles, the ubiquity of cybersecurity threats and the surge of a new generation of users who demand smarter technology are continuously forcing changes to be implemented on existing SCADA systems and are the common drivers for SCADA master planning in the public and private sector.

SCADA master plans are typically shaped by plant processes, end users’ specific necessities, budget constraints, the ascension of new disruptive technological paradigms and unexpected challenges such as the current pandemic, which calls for more unsupervised intelligent controls and secure remote access.

What is a SCADA master plan?

An SMP is an organized set of proposals, engineering documents and guidelines to generate the design framework and roadmap for capital expense projects related to SCADA and automation assets to be executed during a specified period.

One of the first well-documented SMPs was developed by the water department of the city of Philadelphia in 1975. It consisted of a series of feasibility studies to develop a five-year plan to implement a modern automation system for water treatment and distribution to ensure the best quality water at the lowest cost. The primary plan included many methods that are still used, including the multidisciplinary approach and evaluation of the automation asset inventory.

The American Water Works Association Utility Management Manual recommends that an SMP should be developed to pursue the following goals:

• Establish and document vision, strategies and goals for effective operations control.
• Find and prioritize business, operational and technical requirements.
• Establish a coordinated, prioritized program to meet operational goals.
• Define short- and long-term projects, including costs, resources and schedules.
• Define policies, procedures organization, technologies and change management.
• Establish buy-in among stakeholders and executive sponsors.

The primary factor in achieving these goals is to comprehend the client’s needs, ensuring a clear understanding of the stakeholder’s expectations of the SCADA system from the preliminary phase of the project. The overall objective is to align the process, operational and business needs with technology capabilities to develop what is known as a “best fit” solution achieving a shared vision among stakeholders to develop a plan for an expandable system with a sustainable framework that supplies a comprehensive set of services to all users.

Goals and planning period

Decisions about the size of each project depend on several factors, including process requirements, constructability factors, budget, resources, operation, business impact and project scope. Effective SMPs evaluate the project needs to estimate the schedule to meet the desired outcomes.

Long-term projects typically have a significant level of complexity and investment. LTPs usually pursue multiple independent goals. Many LTPs address growing water treatment demands, which translates into a necessity for the improvement of the automation system. These areas of improvement include data sharing, communications, hardware and software standards and cybersecurity. A successful SMP should prioritize and segment the LTP into short-term projects to achieve manageability.

Short-term projects characteristically focus on specific objectives that require less budget and time to complete. The exception might be for unplanned events that require immediate attention, like disaster recovery or remediation efforts. In most cases, STPs imply changes to the SCADA infrastructure that minimally affect the existing operation. Parts of STPs can include critical services or equipment that influence long-term goals, like requiring specialized vendor packages or improving network or cybersecurity infrastructure.

Pilot or trial projects are the execution of a small-scale application to prove the viability of an idea while managing the risks and identifying deficiencies before committing the resources to a plantwide implementation. The pilot or trial will confirm the proposed concept’s feasibility and scalability, allowing end-users to plan safety precautions, methods and resource allocation to reduce the risk during full implementation. In some cases, pilot or trial projects can be a tool to compare commercial solutions.

The water and wastewater sector

There are three primary areas within the water/wastewater sector: water treatment, wastewater treatment and water resource management. Water treatment is the collection, purification and distribution of water for human use or consumption, also known as potable water. Wastewater treatment is the treatment of contaminated or polluted water that comes from man-made sources like sewage or manufacturing process byproducts and the return of that water back into the environment with minimal ecological damage. Both systems typically include a centralized SCADA system with remote sites for collection and distribution.

Water resource management is the control of water resources to minimize damage to life and property and the allocation of water to maximize beneficial use, according to the U.S. Department of Agriculture. The control of water resources refers to managing water-related risks such as flooding, drought and contamination. Water resources must also be balanced between the demands of drinking water and sanitation services, food production and energy generation while sustaining water-dependent ecosystems and natural bodies of water.

Geographic information systems are leveraged by utilities to gather, manage and analyze water-related locational data to make informed operational decisions. Geographic information systems can track water usage patterns, contamination and identify problems like water main breaks and other maintenance requirements.

Who needs an SMP, and why?

SMPs, in many cases, are driven by limitations and deficiencies in legacy SCADA infrastructure or the necessity to prepare for the eventual life cycle end of the existing SCADA system.

In other cases, SMPs fall under the umbrella of capital investment improvement projects triggered by:

• Plant expansions that significantly affect the existing infrastructure.
• Necessity for improved reliability due to obsolescence, support issues or changes in technology.
• Compliance projects or regulations that require significant investments. In 2007, the Puerto Rico Aqueduct and Sewer Authority developed a master automation program that was driven by a compliance agreement with the S. Environmental Protection Agency.
• Legislation enforcement or agreements. The city of Tampa’s 1998 amended Interlocal Agreement required the water agency to update the long-term master water plan (with a horizon of 20 years) every five years.
• Increasing cybersecurity risks.
• Necessity to reduce operational expense through hardware and software standardization.
• Many projects over several years using disparate hardware and/or software.

Human factors, standardization and life cycle

SCADA operators, supervisors and operational technology maintenance support personnel are the primary stakeholders of a SCADA system, and gauging their level of satisfaction can be an effective method to measure the success of SCADA implementation.

In “Self-efficacy: Toward a unifying theory of behavioral change,” Albert Bandura wrote, “a person’s perceived ability to use a product successfully affects their evaluative and behavioral response to the product.”

Several studies have modeled this concept, as described by P C Lai in the paper entitled “The Literature Review of Technology Adoption Models and Theories for the Novelty Technology.” An evaluation of current technology readiness, technology acceptance, their necessities, expectations and work conditions of the SCADA staff and a clear understanding of the maintenance and knowledge management tools for the OT support units are all vital requirements for any SCADA planning.

The ideal SMP should provide a game plan that allows for the replication of the proposed solution by standardization of the SCADA system common elements, like control panels, hardware, software, human-machine interface software and programmable logic controllers and their applications. The standardization goal may vary; however, the most common rationale is to save cost, improve safety and maintenance or shorten the learning curve.

The process of standardization is well known in many industries but is poorly understood in practice. The standardization model effort, according to Carl F. Cargill’s model, is typically executed in five phases:

1. Pre-conceptualization: The team must evaluate what components, industry standards, best practices or vendor solutions can be subject to standardization evaluation. It is important to consider the pros and cons and obtain the consensus of the stakeholders to decide to keep or abandon each idea.
2. Conceptualization: The ratification if the idea is capable of being standardized, defining how it will be implemented and evaluating cost and maintenance.
3. Discussion: The standardization model must be evaluated in terms of risk and impact on the process, operation and business, including the level of flexibility for future changes.
4. Writing: The final stage of the standardization process consists of developing the specifications to be used for the construction phase of the SMP, including test methods and metrics to evaluate the product to be delivered with the specified standard.
5. Implementation: It is vital to ensure the validation of the final system against the specified acceptance criteria. Standards can be successful or can fail because of many unforeseen factors. There are no exact parameters of what makes a standard successful; however, some of the common causes of standardization failure can be traced back to a lack of consensus during the discussion phases or the inadequate definitions or specifications.

One of the goals of an SMP is to reach a solution that maximizes the investment life cycle. The main components of the SCADA system (PLC, network equipment/hardware, servers, operating system and SCADA software) are categorized by the following phases:

1. Introductory stage or development: Manufacturers offer new products that may or may not be backward-compatible or may include updates to the hardware, firmware or enhancements that reflect the market trends. New products could therefore require investment in supporting tools or training. The intro stage product needs to be evaluated in the context of the added value and vendor support capabilities against the associated risks of early adoptions.
2. Active sales or growth stage: Technology is accepted and well-known. It is possible to obtain useful feedback on product performance from current users or publications. The manufacturer supports investment in the product, delivering consistent updates and patches. The SMP team must pay attention to the age of each SCADA component in relationship to the average life cycle. A PLC life cycle is typically 10 to 15 years; however, server hardware and operating systems could have five years with extended support and up to five more years for the operating system.
3. After-sales support: This is the phase following the purchase of hardware. Sometimes vendors decide to stop product development in favor of an updated version that may require a complete change in hardware or software. Critical updates or spare parts to address serious issues such as security patches may still be provided. The SMP must include a clear strategy for replacing SCADA elements found to be at this stage of their life cycle.
4. Obsolescence or decline stage: SCADA components are no longer supported or maintained by the manufacturer. In this case, the client needs to use the secondary market without a warranty or rely on their maintenance stock and staff knowledge management.

SCADA master plan approaches

Although an SMP is characteristically done by many engineering firms using a multifaceted approach, the emphasis of the project will depend on the client’s particular needs and wants as well as the engineering firm’s core abilities and experience. Some of the most common approaches are:

Risk-based and reliability-centered approach: The risk-based approach is a concept borrowed from other sectors like the pharmaceutical and oil and gas industries, in which the goal is to avoid unacceptable risks by categorizing and identifying the potential risks that can affect monitoring and control capabilities. A reliability-centered analysis is done to evaluate if the proposed or existing design solutions are adequate to meet the process requirements, typically estimating the mean time between failure to each mode of failure to determine if the controls are acceptable, the reliability centered approach in SCADA design focus in redundancy and the avoidance of commons point of failures as design criteria.

For the water/wastewater industry, risk-based and reliability-centered approaches are critical for a system that serves large populations (too big to fail) or processes that have potential hazard risks for consumers, operators and communities. This is an example of risk factor calculation for SCADA components, as developed by 2013 Mitch Owens:

Risk factor = [1+X(HW) + Y(SW) + Z(OS)] *Cond * Fail

Where:

Cond = Equipment conditions (1= New, 5 = Failed)

Fail = Failure consequence (1= None, 5 = Catastrophic)

HW = Hardware is supported (0= Yes, 1= No)

SW = Software is supported (0= Yes, 1= No)

OS = Servers hardware will support OS or HMI version upgrade (0 = Yes, 1 = No)

X, Y and Z are factors that client personnel can assign based on staff abilities, spare parts and concerns.

Gap analysis approach: This approach compares the SCADA system’s current performance with the specified or desired level of performance. A typical gap assessment could include evaluating the current system to the original design’s level of automation, alarm management, rate of failures, obsolescence and situational awareness.

One of the most recent assessments incorporated into an SMP is a cybersecurity gap analysis, used to evaluate the differences between the current and ideal state of information security of the overall automation system. A cybersecurity assessment as part of the SMP should include, at a minimum, the following:

• Inventory of hardware, software, licenses and user’s access.
• Evaluation of policies and procedures.
• Identification of thresholds, risks and vulnerability.
• A vulnerability prioritization report.

Technology upgrade or replacement and standardization approach: Some of the most common drivers for an SMP are hardware and software obsolescence, lack of vendor support and the desire to unify technology. The SMP will focus on proposing solutions supported by market studies, alternative analysis, evaluation of OT’s support capabilities and capital expense and operational expense analysis to deliver a series of recommendations that outline the roadmap to the SCADA system upgrade or replacement. This engineering effort is usually finished with a set of instrumentation and controls documentation that outlines the SMP design criteria (see Figure 2).

Workshop-based master plan approach: The workshop-based approach is a multifaceted methodology that focuses on SCADA system assessment and client perspectives through a series of well-planned workshops that identify baseline status conditions and issues with the SCADA system. This methodology is focused on identifying known and unknown issues in-depth to develop the best fit solution that prioritizes and finds consensus among the shareholders’ shared vision. The best fit solution is an expandable plan with a sustainable framework aligning people, processes and technology.

SCADA master plan framework

A good SMP must have an organized set of tasks with a clear definition of the goals executed in a sequential and logical order where each step supplies input to the next task (see Figure 3).

Team conformation: The SCADA master plan’s success relies significantly on stakeholder involvement because the operator, supervisor and maintenance personnel are typically aware of the process, business and operational matters that add value to the overall SMP. The assembly of a team with experience and knowledge of processes, business and technology is the first task that needs to be fulfilled as a milestone for the project’s success.

After the team is finalized, a clear definition of tasks and responsibility is organized in a batch of workshop and group meetings to allow the volume of information and design process to cascade throughout the SMP (see Figure 4).

System assessment and needs development: This step includes collecting the relevant information to understand the current automation system, SCADA life cycle status and relevant issues and problems. It requires a complete revision of the system documentation as well as staff interviews and workshops which can be accomplished by three main tasks:

• Defining the team roles and interfaces with the client to collect the known issues from each stakeholder’s perspective.
• Documenting the existing architecture, flow diagrams, hardware and software, typical panel drawings, cybersecurity policies, etc. The overall scope is to highlight the elements that will support the design criteria for replacement or upgrade.
• Understanding the organization’s capabilities to support the current system.

At the end of this phase, the team must be able to list the organizational needs and capture a set of solutions that can be evaluated across many areas such as operation, maintenance, engineering, support and cybersecurity.

Alternate development, risk analysis and assessment: At this stage, the team’s focus is to elaborate on a compressive revision of the possible automation technologies and SCADA architecture that would be able to serve the process and business requirements. The team must work on systematic evaluation, alternative analysis and risk evaluation of each solution. Vendor presentations and technology workshops are an integral part of this phase; the idea is to expose the decision-makers to a range of feasible technology solutions that can maximize the system life cycle (see Figure 5).

Scenario development and solution evaluation: This stage aims to develop a list of alternatives that incorporate hardware, software and constructability methods through a well-documented evaluation. The team must present a reduced range of vendor options or solutions to resolve the system needs, including capital expense, operational expense and organization gap and challenges. The scenario assessment must develop an evaluation matrix describing the proposed architecture, vendor solutions, cybersecurity approach, disaster recovery, data flow (from alarm and event management to business integration), internal or vendor support requirements and level of skill needed for maintenance and development.

Goal definition, detailed plan, cost estimate, checkpoint and final report: The final phase of the SMP consists of the documentation development for the recommended alternatives and includes a detailed definition of the hardware, software and application features needed to fulfill the client’s requirements and needs. The documentation must emphasize reliability, the expected life cycle, knowledge management, upgrades, integration and expansion capabilities. The detailed plan must include design and construction processes and schedules, with a complete cost estimate differentiating hardware and software cost from development cost. The SMP must include all SCADA components and development specifications, including the documentation for standardized panels, PLC hardware and HMI applications.

Although there is no consensus about the process required for successfully developing an SMP, many consulting firms follow similar approaches that begin with a clear understanding of the client’s needs followed by a thorough assessment of the current system guided by intense interaction with the stakeholders. Because of this, it is extremely important to use experienced engineers who truly understand the current hardware and software as well as the latest technologies in hardware and software.

As a result, a final SCADA master plan “best fit” solution will be delivered that focuses on system and organizational needs; develops a complete understanding of trends, market maturity and potential improvement; maximizes the investment life cycle; and defines sustainable methods for maintenance and expansion.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.