Converged networks for controls and operational technology
By better understanding building controls and converged networks, engineers can make smarter IT and OT choices
Learning objectives:
-
- Define converged a network and understand the difference between information technology (IT) and operational technology (OT).
- Understand how to plan for and document an OT network during design.
- Know the benefits and risks of a converged network.
Building control insights
- Engineers must design converged networks that integrate IP components across building systems, reducing cabling and enhancing interoperability.
- This requires careful coordination with IT to ensure security and efficiency, understanding different topologies and planning for future-proof infrastructure that can adapt to evolving technologies.
Building controls can have life cycles that exceed 30 years, so it takes a long time for advancements in controls to gain widespread adoption. Traditionally, the networks connecting controllers to the operator’s station have been in silos with segregated networks for each system. Because it has been easier for the vendors to coordinate than the engineer, they were only addressed with performance specs and maybe a generic wiring diagram.
Many communication protocols have been used to run these networks over the years, but internet protocol (IP) has rapidly increased in usage. As technology has evolved, we are now at the point where most new control systems have at least some IP components, with many going native IP for all core infrastructure.
At the same time, there is a push to integrate the different building systems — like the building automation system, lighting control system or energy management system — onto a common (or converged) network. Using a converged network can reduce the amount of cabling and headend needed while setting the foundation for interoperability between the different systems.
While these converged networks have greater functionality, they are also more complex and require more coordination to get the most out of them and to keep them secure. In the past the information technology (IT) group may have provided a couple of data drops to facilitate remote access, but new management strategies and tight coordination between IT and facilities are needed to implement converged operational technology (OT) networks.
Since all of this starts in the engineering design process with the engineers, it is important to understand the difference between IT and OT, what a converged network is, how to design one and the benefits and risks to the building owner and operator so that owner project requirements can be met in efficient and effective ways.
How controls are related to OT
The definition of OT from the National Institute of Standards and Technology (NIST) is “programmable systems or devices that interact with the physical environment (or manage devices that interact with the physical environment)” and can include building controls, internet of things (IoT) devices, access control and anything else that is used to make the building function
This differs from IT, devices such as personal computers (PCs) and printers that allow work inside the building to happen. While they both use networks to allow devices to communicate, there are major differences between how they operate.
IT networks became much more complex as the internet became integral to daily work. It is now standard to employ large teams to manage these networks and all the devices that live on them. Technology designers and consultants are regularly brought onto projects in addition to mechanical, electrical and plumbing engineers to design this infrastructure and coordinate with the owner’s IT group.
OT networks exist in many buildings as a collection of siloed systems that each serve only one building system and connect to the owner’s IT networks for remote access, although these connections are often not accounted for until construction. With proper planning in design and coordination with IT and facilities, we can get OT networks that use appropriate hardware while still being cost effective and secure, and eliminate system silos. This results in a unified architecture that is scalable, centrally managed and secured and less dependent on the vendors that installed it.
Converged OT networks eliminate the duplicated components that come with silos like separate backbone cabling and switches or panels for each system by sharing common network infrastructure like copper and fiber cabling, switches and routers. There are two approaches for creating this converged network: expanding the IT network to incorporate all OT systems or creating a standalone OT network.
The best approach will depend on the complexity of the OT systems and the sophistication of the facilities and IT teams, but most of this article’s content is applicable for both approaches. Opinions may vary, but in general, simple commercial, residential and K-12 buildings can make sense to integrate OT with the IT network. Health care, higher education and other complex buildings or campuses can benefit from a dedicated OT network. Federal and critical infrastructure projects have increasingly strict standards around OT networks for cybersecurity so extra caution must be taken.
Bringing the control network together
Whichever method is chosen, the overall architecture and components are similar. Figure 1 shows an example riser diagram of a converged OT network. Starting at the edge and moving in, we can see the physical equipment (air handlers, lights, meters, etc.), field controllers and gateways, network switches, servers and computers, core switch, a router and firewall.
Field controllers connect to the equipment, delivering commands and collecting data. There are many options for manufacturers, input/output types and quantities and compute power. They communicate with open protocols like BACnet, LonWorks and Dali or proprietary protocols specific to their vendor, although an increasing number are adding IP as well.
Gateways can come from the same manufacturer as the controllers or third parties and are used to convert other protocols to IP, which is necessary to connect to the network switch if IP capability is not built in, allowing communication between all connected devices. The characteristics to consider with these switches include the layer type, number of ports, power over Ethernet (PoE) capacity and speed; specific requirements will depend on the controls being used and owner’s requirements.
The servers and computers run the software programs controlling everything; they are defined by their electronic components and operating system and may be cloud-based instead of in the building. Dedicated workstations that both run the software and serve as the user interface are not common any more as most applications are web-based so that any PC on the network with the right credentials can access any application. Virtual machines can also be used to remotely access programs on servers that are not web-based.
The switches connect via fiber to a core switch that helps route all traffic on the network. The router, which controls communication between networks, is used to connect to either an internet service provider or to the IT network for remote access.
Firewalls can be placed at any point between components where there is a desire to control the data that can pass between for cybersecurity. They are often used near routers to control information going between networks and their requirements are defined by the owner.
Other miscellaneous pieces of equipment can include uninterruptible power supplies, a network controller that gives more management options and other gateways or firewalls as needed.
Converged networks
The main benefits of a well-implemented converged network are flexibility, control and data accessibility. By using a common infrastructure for all building systems, it becomes simpler to add new systems or change components not performing to expectations. When controllers are on the network, the owner has increased visibility of them from their computer, making it easier to manage firmware updates and reprogramming remotely.
With the increased bandwidth of Ethernet cabling, devices at the edge like field controllers can do more computing of trends and alarms locally while still transmitting everything back to the server and eliminating the need for physical supervisory controllers, which act to pull information from other controllers before sending back to the headend. By owning and managing the physical infrastructure, the owner is set up well to make the most out of the data coming from building systems because it can all be accessed through a single network.
As we move into the artificial intelligence era, there are countless potential uses for robust building data but the main ones today are building optimization and tracking and reporting things like energy or carbon use and indoor environmental quality (IEQ). The next step of integration is consolidating data into an integrated data layer (IDL) or single pane of glass (SPoG) user interfaces.
The IDL acts a single point of integration for all building systems to share information, a cleaner alternative to many individual direct connects between systems. The SPoG provides a single platform for facility operators to quickly access information on all building systems. While some owners may not be willing to make that investment on Day One, having a converged network layer makes it much easier to implement the IDL or SPoG in the future because all the communication needed can happen on this single network instead of having to cross over multiple or be limited to the cloud. These benefits are all in addition to the elimination of redundant infrastructure and devices inherent to the traditional building system silos.
Wiring topologies for control systems
A wiring topology is one of the first things you can add to your specifications to design better OT networks. The topology is the typical way in which devices are physically connected; more than one can be used on the same project.
The list below describes the common topologies along with the benefits and limitations of each (see Figure 2). Reference ANSI/TIA-568-D for more information in general on cabling standards and topologies.
-
Star topology: Every device is connected directly to a switch with one cable. This is the most robust approach, but also the costliest. The main limitation is the maximum length of the cable being used, 300 feet in the case of category 5/6 copper cabling. This is best used for critical equipment that requires high uptime and is constantly sending data.
-
Daisy chain: Every device connects to the one before and after it in the chain, with only one device connecting directly to the switch. Each Ethernet cable in the chain is still limited to 300 feet, but the controllers can repeat the signal allowing devices more than 300 feet to be connected to the network. This is a cost-effective option and does not require as much planning, but a device or cable failure in the line means every device downstream loses connection. This topology should generally be avoided when using IP controllers due to the cascading effect of device failures, but can be useful for reaching distant devices.
-
Ring topology: Similar to daisy chain, but with the devices on both ends of the chain having a connection to a switch. This simple modification to the daisy chain requires little additional infrastructure but provides increased resiliency as any single failure will not impact other devices. More planning is required as the devices used in a ring, including the switch, must be set up to use rapid spanning tree protocol to communicate and ring sizes should be kept in check to limit risk, but the ring topology will be the preferred option most of the time.
The second part of planning the cabling for building controls is assigning responsibilities. RS-485 wiring has been run directly by controls vendors, but there are several reasons that those vendors may not be the best option for pulling the Ethernet cabling between controllers and switches. For one, controls vendors may not have the BICSI certification, skill set or tools needed to terminate onto RJ45 connectors and test the cable.
Second, you will be much more likely to have neat and organized cable trays or J-hooks when only one trade is running the cabling. Cost savings can also be expected from having all cabling provided by one trade rather than splitting it up. The biggest potential pitfall of having the structured cabling contractor run all Ethernet cabling for controls is that they are used to only accounting for a few data drops to be coordinated with controls.
Therefore, providing enough information in the construction documents for them to properly estimate those costs and communicating with them early is the best way to avoid change orders stemming from using a converged network. Considering the OT network, simple projects can pull the scope under the structured cabling vendor by using ARCAT CSI Division 27 specifications for cabling requirements with the general contractor overseeing coordination between them and the other trades.
As projects get more complex, there is more potential value in assigning the scope to a master systems integrator (MSI). A relatively new role on construction teams, MSIs are responsible for connecting all the building systems on the converged network, implementing any interoperability use cases and coordinating between all the trades; they often respond to Division 25 specifications.
For example, their scope could include working with the mechanical, electrical and security contractors, facilitating their connection to the OT network by coordinating IP addresses and cabling, implementing software integrations between the systems and ensuring cybersecurity standards are met.
Wireless control considerations
In addition to the wired infrastructure, there can be many wireless devices in the OT category. If multiple vendors are brought in to provide different IoT solutions such as IEQ monitoring and occupancy sensing, it is likely that each system will want to use its own proprietary gateway(s) to connect their wireless devices.
There are different reasons they claim to operate in this way, but locking owners into their ecosystems is almost always an ulterior motive. Each additional gateway is another device on the network that must be managed, kept secure and potentially comes with a yearly license. They could also be using a different wireless protocol, each with its own strengths and weaknesses that impact how well they fit with a particular project.
Below are some of the more common protocols with short descriptions of how they work. Covering all the differences and technical aspects could be an entire article on its own and the landscape of devices compatible with each is always changing, so this is not an exhaustive list.
-
Wi-Fi: This system exists in almost every building and most people interact with it daily. The physical infrastructure includes access points scattered throughout the building connected back to switches with Ethernet and are typically PoE. Devices connect directly to the network that is being broadcast and receive IP addresses. It can be tempting to use, but it is not the most reliable connection and traffic from OT devices can reduce speeds for IT devices trying to use the Wi-Fi. This protocol is also more energy intensive and so battery-powered devices will die quicker. In addition, the frequencies Wi-Fi operates at (2.4 and 5 GHz) is crowded so transmissions are more likely to suffer from interference.
-
LoRaWAN: This is an open standard that has gained a large market share for connecting environmental sensors outside the U.S. It has very low data speeds (<50 kbps), which limits the amount of information that can be sent, but single access points can cover large areas and penetrate walls well, reducing the amount of physical infrastructure needed. Transmission also has lower energy consumption than Wi-Fi. In most cases, a LoRaWAN access point will be compatible with any LoRaWAN devices, even if they come from different manufacturers.
-
Zigbee: This protocol involves connected devices creating a mesh network in which they both receive and send data. Because every device extends the reach of the mesh network, you can often use just one gateway to physically connect to the rest of the OT network. Speeds are low, but still higher than LoRaWAN, coming in around 250 kbps. Reliability is also strong in a mesh network because transmissions can take multiple routes back to the gateway. However, systems using Zigbee made by different manufacturers are unlikely to be able to communicate with one another.
-
Cellular: This is primarily used at the head end for vendors to connect to the internet without going through the IT or OT network, but many solution providers are developing devices that can connect directly with it using SIM cards. Cellular is a highly reliable wireless system with huge bandwidth and very high speeds available through 5G, so potential use cases include everything from field controllers to security cameras. However, there is a high cost to implementing a private cellular network using citizens band radio service, so very few owners are investing in this currently and we are unlikely to see a large movement toward it outside of niche cases in the next 5 or more years.
With a little bit of planning and coordination, you can often select a single protocol to be the standard for the building and specify an infrastructure solution for all wireless systems to communicate on, putting the control back in the owner’s hands. To do this, you will need to consider what the use cases for wireless technology within the building are (IEQ sensing, leak detection, people counting, etc.) and see how the applicable solutions overlap in connectivity options. Always start with the use cases and end with the technology, so limiting to a single solution may not always work out, but considering all wireless systems together will result in a more efficient design.
Cybersecurity in control systems
With all the benefits of a converged network, the cybersecurity risks cannot be overlooked. When all building systems exist on the same network, vulnerabilities in one system can lead to all of them being impacted. When connected with the IT systems as well, an entire company can be hacked through one compromised device without proper protections in place.
Cybersecurity for OT devices is also different from IT devices in many ways, so owner’s IT departments are often ill-equipped to manage the OT network. It is important to note that engineers are not cybersecurity experts, nor are they expected to be. At the same time, cybersecurity cannot be entirely ignored in specifications.
The role of the specifying engineer in this process is to communicate with the owner and understand what standards, if any, they have for cybersecurity to which the building controls will need to adhere. Ideally, the owner will provide a standards document that can be incorporated into or referenced by the specifications. If they need guidance, there are OT cybersecurity specialists available to create standards for them and/or manage their OT network directly. Public guidelines like NIST CSF, UL 2900, ISA/IEC 62443 and the Real Estate Cyber Consortium are also available.
While the design and setup of the systems and networks can mitigate risk and set a secure foundation, operational standards and practices will play the biggest role in maintaining cybersecurity. Because of this, the expectation must be set that the owner understands and drives the standards, even if they must bring in a specialty consultant to accomplish this. No matter your level of knowledge around cybersecurity, the conversation must be had on every project.
Timing it all correctly
A second risk that directly impacts the construction process is timing. IT networks are typically not fully activated until late in the construction process, but vendors need access to a network for their systems to program them. If the OT network is not put on an accelerated schedule to be available for each systems’ vendor, temporary networks will be required which comes with a cost.
There is also additional effort needed to move from the temporary to permanent network. If not properly coordinated, the OT network schedule can cause delays and cost increase to a project. “Pull planning” exercises in which all parties coordinate the time and resources they need to deliver their systems are essential for establishing when a network must be available. Network availability is dependent on more than just the equipment and cabling, as things like network switches can be damaged by dust in the air, power surges, water leaks and are often targets for theft so preparing spaces to house them requires coordination between multiple disciplines. It is certainly possible to create a clean environment in IT rooms or other designated spaces before the surrounding areas reach that point, but only if the need is identified and clearly communicated from the start.
As the controls industry moves in the direction of IP, converged networks are the logical next step for best-practice design. With the long life span of building controls, every project needs to consider ways to future-proof its buildings and prepare for the new technologies that will drive efficiency, resilience and exceptional occupant experiences. Engineers looking to take on the role of trusted adviser to their clients should be leading the charge in this area of innovation.
Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our WTWH Media editorial team and getting the recognition you and your company deserve. Click here to start this process.