Know whether it’s time to replace a control system

Control system insights

• As owners cope with operation and maintenance of aging facility and utility infrastructure, maintenance and updates to their control systems are typically overlooked.
• As a result, many owners are facing costly upgrades or replacements due to active or impending obsolescence of their control system assets that introduce vulnerabilities to their operations in the form of reliability, resiliency and security risks.
• The first step in determining whether their control system is an asset or a liability is for owners to understand their current system by completing a detailed assessment.

Control systems have an expiration date, a shelf-life, a best-by date. This can be difficult to predict and is driven by advances in hardware and software technology, changes in manufacturing processes and in some cases forced obsolescence by vendors.

The life span of system components can vary by type with some requiring more frequent renewal than others. Once they reach the end of their useful life, they can represent operational vulnerabilities that must be addressed to mitigate reliability concerns due to degraded performance and lack of vendor support., creating a situation where there may be no available spare parts or direct upgrade path available to address equipment failure.

As with most technology, this phase in the life cycle of the hardware and software that makes up the owner’s control system cannot be avoided. While the life of some system components can be extended through proper maintenance, renewal is imminent from the date of installation.

A strategy for maintenance and renewal is key to managing system assets and avoiding a costly obsolescence cliff that monopolizes owner investments and resources to keep their system operational.

But what should owners be prepared to do if they do not have a robust maintenance and renewal strategy in place?

The control system challenge

Since the industrial revolution, control systems have evolved to form technological nerve centers providing supervisory control and data acquisition to accomplish tasks that help improve human conditions or produce automated outcomes for many applications (see Figure 1). From managing occupant comfort in high-rises, to automating manufacturing processes in factories, to advanced control of treatment processes that produce clean water, the industrial control system has become a crucial element in society.

Figure 1: Industrial control system applications. Courtesy: HDR

A reliable, well-maintained control system can help realize many benefits for owners, some of which are outlined in Figure 2.

However, in many cases, control systems are still not treated as assets requiring maintenance, updates and renewal. While a focus on cybersecurity has helped drive and improve the emphasis on maintenance of some of the system components, many owners still do not know the extent of obsolescence, maintenance issues or security vulnerabilities that are present in their systems. Unfortunately, once the problem is discovered, systemwide obsolescence requiring complete system replacement is often the result.

Without replacement and system updates, owners risk unplanned downtime due to component failures that are not easily repaired, or exploitable security vulnerabilities that threaten continuity of operations as well as personnel and public safety. Owners may suspect that they have vulnerabilities that need to be addressed, but do not know the full extent of work and investment required.

As control systems grow to meet operational demands over time, they can represent a liability requiring resources to react to issues rather than serving as an asset that realizes benefits for the owner.

Figure 2: Benefits of the modern control system. Courtesy: HDR

Common characteristics of a system that may need improvements and possible replacement include:

• No existing system documentation, or documentation does not reflect the current installed state of the system.

• Significant system growth over time without governance and standards driving consistency in design and implementation.

• Unreliable or untrusted instrument or sensor measurements.

• Instrumentation that is in poor condition or not routinely calibrated/maintained.

• Obsolete controllers and associated hardware that are no longer supported by manufacturers or there is limited or no available replacement stock.

• Obsolete network appliances that are no longer available or supported by manufacturer.

• Controller, radio or network appliance firmware has never been updated or status of updates unknown.

• Outdated field network or backbone communication protocols.

• Computer workstations and servers that have not been replaced for more than five years.

• Outdated and unsupported workstation and server operating systems or status of patches/updates unknown.

• Outdated and unsupported data acquisition, alarm management or historical data software or status of patches/updates unknown.

• Degradation of system performance and communication errors.

If a system exhibits these characteristics, there may be widespread issues indicative of a system that may require significant investment with areas of large-scale replacement. The top drivers for system replacement are shown in Figure 3.

Figure 3: Technology use for consistent data collection. Courtesy: HDR

Owners often have insight into these characteristics due to staff knowledge and reporting but may not know the extent of issues present that may threaten successful operations. Some owners only suspect potential issues because the system has received no maintenance or renewal since the original installation.

Depending upon the size and complexity of the installed system, the prospect of taking on the effort to identify and address vulnerabilities can be overwhelming. Only parts of the system may need to be updated or replaced, or a comprehensive replacement may be required. The system will likely have dependencies meaning that a change in one area of the system may have unintended impacts to other parts of the system so continuity with a holistic approach will be required.

The first step is to complete an assessment to baseline the system, assess risks and identify potential vulnerabilities and determine what actions may be to mitigate liabilities introduced to the owner’s operations.

Complete an assessment of the control system

The foundational step for successful deployment of control system improvement or replacement is to understand and document the status of the existing system. This will serve as an opportunity to review potential vulnerabilities and serve as a springboard for recommended improvements. Control systems can cover large areas and include many assets, so this step may require significant time and resources to complete. Depending upon system size, complexity and owner’s knowledge of installed assets, a pre-assessment may be prudent to develop a preliminary opinion on the need for a current state assessment and to inform the level of effort required.

A typical current state assessment has two phases: data collection and data evaluation. During data collection, information is collected that best represents the installed or current state of the system.

This is a hands-on effort that requires a detailed investigation of the system including:

• Available documentation such as asset location drawings, network diagrams, piping and instrumentation diagrams, function diagrams, panel drawings, input/output (I/O) lists, loop drawings, instrument lists, control descriptions and operation and maintenance manuals.

• Sensors, analyzers and instrumentation.

• Calibration records.

• Controllers and control panels.

• Network racks and equipment such as switches, routers and radios.

• Workstations and servers.

• Installed control system software including human machine interface, historians, antivirus and operating systems and how they use and share data.

• Installed enterprise software that accepts or shares control system data including maintenance management systems, facility management systems, asset management systems, enterprise resource planning, etc.

• Power distribution and raceway paths.

All subsequent steps will build upon the information collected during this step. Spending time to collect and organize information for reference is critical. Data collection should not be rushed. An organized collection process must be established and where possible, technology and data collection tools should be used to develop relational databases and reports to share information for ease of future reference (see Figure 4).

Collecting, evaluating control system information

Field verification of available documentation is critical to determine the level of effort for replacement and system improvements design. Depending upon system size, this step may require multiple teams to collect data. Adopting a method to consistently collect, verify and annotate variances across multiple teams will streamline the process and reduce the overall effort in the field to support this step.

If possible, the date of installation and condition of equipment should be determined during the investigation. This information is used to establish the position of the installed hardware and software components in the product lifecycle and to determine obsolescence. The condition is used to determine the extent of future improvements.

For example, if a controller requires replacement due to obsolescence or planned obsolescence and the panel in which it is installed is in poor condition, future improvements to replace the controller should also consider replacement of the associated panel.

Figure 4: Top drivers for control system replacement. Courtesy: HDR

Discussing how members of the owner’s team interact with the control system to perform their work is also important during this step. This is typically done in an interview setting with members of operations, maintenance, engineering, information technology and management. During the interviews, information is gathered on how data is used by staff, what features users feel are broken, where potential problem areas exist in the system or what can be improved to help staff perform their duties most efficiently.

Evaluating the data collected is primarily a desktop exercise. Objectives of the data evaluation can include:

• Capture variances in as-built condition versus available record documents.

• Document installed hardware and rate its overall condition.

• Document installed software including version and patching.

• Determine hardware and software position in the product life cycle so that need for replacement can be planned.

• Determine how data flows in the control system and if best practices for securing the control system are in place.

• Document the baseline system architecture.

• Document communication protocols deployed.

• Identify vulnerabilities in the overall system architecture that present security or reliability risks.

• Determine how the system operates and identify opportunities for optimization.

The data evaluation is used to develop a list of recommended improvements for the following areas of work which will serve as a benchmark for subsequent steps:

• Policies and procedures.

• Field devices.

• Controllers and I/O.

• Network architecture

Ongoing control system assessments

Control systems comprise technology assets that require routine maintenance and renewal to mitigate vulnerabilities that threaten the reliability, resiliency and security of owner operations. If a system is exhibiting characteristics indicative of a system that may need improvements and possible replacement, the best place to start is to complete an assessment of the system to determine if there are vulnerabilities that should be addressed. This assessment will serve as the foundation for identifying, planning, designing and implementing system replacement and associated improvements.

Once the current state assessment is complete and the benchmark is established for identified improvements, the next step is to complete pre-project planning.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our WTWH Media editorial team and getting the recognition you and your company deserve. Click here to start this process.