Commissioning control systems for MCFs

Controls success starts in design with clear requirements, and ends with thorough testing and complete turnover. Here are eight steps to follow.

By James McEnteggart, PE, Primary Integration Solutions LLC, Charlotte, N.C. December 31, 2012

The phrase “integrated MEP control systems” generates images of holistic design where all components of the mechanical, electrical, plumbing (MEP), and fire protection systems work in harmony. Unfortunately, industry professionals see far too many mission critical facilities (MCFs) in which MEP systems are joined by a menagerie of control systems and protocols with varying degrees of success. Successful integration of MEP control systems depends as much on effective independent commissioning as it does on the right design and proper implementation.

Experienced commissioning agents (CxAs) do not claim to be design experts; however, they have seen many integrated MEP control systems that did not perform as designed during testing and commissioning. When commissioning begins at the planning phase of a project, the questions and insights gained from experience in commissioning MCFs contribute the greatest value to meeting project objectives and saving the owner and project team rework, change orders, and associated time and money. Here are eight key issues and insights at each phase of an MEP control integration project.

1. Develop a clear, written statement of operational and performance goals.

From commissioning experience, there are two critical success factors in the planning phase of an MEP control integration project: a clear, concise written definition of project goals in operational and performance terms; and the right decisions about multi- versus single-system automation and network architecture.

It is surprising how often project teams assume that the goals of the project are clear to all participants without clearly defining them in a written document. The CxA can help the team define the project goals in as much detail as possible, and put them in a concise, written owner’s project requirements (OPR) document.

The document should identify and define the owner’s operational and performance objectives in detail. For example, operational details might include the owner’s requirement for real-time communication of system operating data to staff of a remote, 24/7 network operations center. The owner’s performance goals might, for example, include key parameters such as refresh time for data reporting. Vetting the requirements through the client’s IT department for security requirements at this stage will save headaches in the future.

The document should be periodically reviewed over the course of the project, and key decisions that inevitably impact design and implementation of the project should also be documented.

A CxA can also help to get everyone literally on the same page by summarizing the typical 100-plus-page program document into a concise 10-page document that outlines the specific project requirements, including setpoints, required redundancy, and so on. The agent also uses this summary document to review design to ensure it remains true to the project goals.

2. Multisystem automation provides the most flexibility.

At the core, there are two ways to integrate multiple systems: select a single controls and automation vendor and make it responsible for all aspects of the integration, or use a third-party integration system to connect the various systems, process the data, and present it in a uniform manner. From experience, a CxA may urge use of a third-party integrator. Although the single-source solution has the advantage of ease of initial implementation and a single point of responsibility to resolve problems, the main drawback is that once a single vendor is in place, purchasing power is severely compromised. Switching controls vendors in an operating facility is costly and disruptive.

The third-party integrator introduces another vendor to the project, but provides the most flexibility for the owner’s future system modifications and upgrades. It also has the benefit of fully leveraging the controls and automation that are supplied by the manufacturers of most building equipment.

With the continuing development in automation protocols, the differences between the two approaches has diminished, but not disappeared. A CxA may also advise bringing to the planning table a third-party integration consultant. To ensure objectivity, specify that the consultant is ineligible to bid the implementation contract.

3. Network architecture: Two networks are better than one.

Historically, MEP control and automation networks were set up on a dedicated network infrastructure physically separated from the production IT network. As protocol translation technology has improved, the industry has increasingly relied on a shared Ethernet infrastructure for IT and MEP control systems. This saves money; however, the design of the active network components in this shared network must accommodate the protocols and data types being carried by all connected systems. Often, the standard Ethernet network hardware is inadequate for MEP control data, and this becomes obvious at testing and commissioning.

Over the past year, one CxA saw weeks of lost time in a number of projects due to the need to resolve issues related to single-system automation and single-network architecture. Time and money would have been saved had the potential drawbacks been anticipated by all members of the team in the planning phase.

4. Design/specification: Leave no room for multiple interpretations.

The proper design and specification is the most critical component in delivering a fully integrated MEP control system in a cost-effective manner. Having a “hands-on” experienced system integration consultant on the design team is usually money well spent. The participation of this profession will ensure that design and specifications are practical, economical, and, most importantly, achievable.

Equally important is development of a specification that leaves no room for interpretation. The CxA can be invaluable in helping to ensure that the specs are clear, concise, and detailed to avoid the misinterpretations that can lead to change orders at the testing and commissioning phase—or worse, disagreements that lead to litigation.

As the system design takes shape, the underlying concepts should be simple and easy to understand. If necessary, the concepts should be broken into smaller components until they can be easily explained to any programmer, system integrator, installer, or operator. Simplification of the concepts to their core elements ensures that the development of control sequences will be clear and easy to understand.

Similarly, the mark of high-quality MEP control system construction documents is a singular meaning. Leave no room for multiple interpretations. Ideally, invite both experienced industry professionals and stakeholders, such as the MEP operators, to review the documents for completeness and singular meaning.

5. Avoid software issues related to control code, graphics, and protocol mapping.

MEP control system vendors’ general approach to the development of control code is to have a single or small group of programmers prepare the code based upon the project documents and the programmers’ own approach. From experience as a CxA, this approach creates multiple opportunities for problems during implementation and throughout the entire lifecycle of the system. Ideally, criteria for control system vendor selection should include the creation and use of small, standard reusable blocks of code, which will reduce total programming time and troubleshooting requirements in the field.

No project can be delivered without some level of customization, but the standard blocks should be the starting point, with all modifications and additions tracked in a version control system that supports restoration of the stored code to various points in the development so the program can revert to a state when it was known to function. This functionality greatly reduces the level of effort to manage future updates and code enhancements.

Project specifications should require the controls vendor to submit a complete control graphics screen map and mockups for the facility’s staff to review and accept before code development begins in earnest. This enables the initial development of the code to more closely meet final requirements on the initial pass, saving rework and time later.

Similarly, specify that the controls contractor require from each manufacturer a detailed index of all their control points and how they are accessed via standard protocols. This information should be cataloged into a master points list that is used for network planning, as well as development of the control sequences.

The CxA will validate that the control graphics screen map and protocol map meet the client’s requirements.

6. Implementation: Factory testing saves time.

Some CxAs strongly recommend factory testing, which has gained some traction in the industry but is still underused. Requiring the controls and automation contractor to assemble its panels in the shop and complete programming and code development prior to deployment of equipment in the field can shave a great deal of time off the project delivery schedule when it is most valuable: at the end of the project just prior to turnover.

Typically, the engineer will specify that the test must be performed, and the manufacturer will write the test procedure. The engineer and CxA should review the procedure prior to testing. Schedule the factory acceptance test to allow the engineer, CxA, and facility’s operations staff to assess the graphics and operator interface and review the modes of operation (including presentation of alarms). The test should be conducted to the greatest extent possible by using the actual hardware that will be deployed to the project. The CxA witnesses the testing process and validates the results. Identifying issues in the shop and getting user feedback before delivery to the site reduces field modifications and the potential errors that they introduce.

7. On-site testing and commissioning starts at the component level.

The process for testing and commissioning any system should start at the component level and expand in scope until the entire system, and ultimately the whole facility, is tested. By their nature the full implementation of integrated controls will be one of the last items on the schedule; however, this does not relieve the installer from completing proper testing of all installed components via point-to-point verification and loop checks.

A key concept that is not well understood in the industry is the term "done." All too often a CxA receives a status report that something is "done,” only to find that several key components, such as power connections, are unfinished. This wastes time, and mission critical projects rarely have the luxury of wasted time. Before testing the integration of systems, the constituent systems themselves must be completed: that is, all open items are closed, and a complete, successful start-up has been documented. One way to streamline project delivery and avoid increasing project costs over time is for engineers to contractually require that contractors pay for the CxA’s additional time resulting from misreporting completion status.

Here is the recommended commissioning process from the component to system level:

• Original equipment manufacturer (OEM) controls: Verification of all OEM controls at each device is a prerequisite of any integration testing.

• Protocol transfer tests: Whichever communication protocol is being used, standard tests should be used to confirm proper configuration of the network infrastructure to support each of the protocols being used. Special care must be taken to test the robustness of the infrastructure to handle both normal and abnormal traffic levels.

• Silo system testing: Similarly, systems that have control systems installed by the controls contractor should have comprehensive testing done on all points to ensure proper connections and configuration within the control system. Some CxAs try to combine OEM and silo system testing; these should be conducted for each component and then for the overall integration.

• System-to-system tests: Once all subsystems and components have been successfully tested, the commissioning program can move into the integrated system testing phase. At this point each component system has been fully tested in a stand-alone fashion, so integrated testing does not need to cover the same ground. Rather the focus should be on the interfaces between systems; for instance, the interaction between the fire detection systems and the HVAC systems. Each scenario that requires that two or more systems operate in a unified manner should be simulated and appropriate response verified. Inviting facility staff to participate in these tests is an excellent opportunity for the CxA to provide training.

8. Ongoing operation/change control: Specify a complete, updated set of documents.

Upon successful completion of the construction and acceptance testing, the facility is ready to be turned over. As part of the turnover, the contractor should provide a complete set of updated documents and control code. A lessons-learned session that includes the construction, commissioning, and operating staff yields significant knowledge transfer to support ongoing operations.  

In addition, having the controls contractor periodically return to the site (monthly or quarterly) to work with the facility staff on small system adjustments is a good way to ensure that facility operations remain optimized. Changes in setpoints or programming should be updated in the documents. As time progresses and the various suppliers issue updates to the on-board controllers or system software, great care must be taken to determine the impact the upgrade will have on the integrated control and automation system. Advise owners to make all service technicians aware that a formal review must be undertaken before any firmware or software upgrades can be installed at the site. Manufacturers that supply heavily to the mission critical industry are familiar with this requirement and often issue full briefings on the upgrades they issue. However, owners should be made aware that equipment suppliers that deal across a broader scope of industries may not be familiar with this requirement and will require some direct interaction. The details of the upgrade should be reviewed by someone with in-depth knowledge of the control system to identify possible impacts.

Finally, periodic retesting and recommissioning of the controls and integration during scheduled maintenance windows will provide a level of assurance that the system’s performance has not degraded over time. CxAs can provide this relatively low-cost, high-value service over the lifecycle of the system.

The integration of MEP control systems in a mission critical facility is an exacting and demanding task. Careful and deliberate planning, coupled with professional installation, testing, and commissioning, is the only way to achieve a successful outcome. Assembling a team with controls integration and commissioning experience in the planning phase of a project will yield substantial benefits in avoided costs and improved schedule.

James McEnteggart is vice president of Primary Integration Solutions, a mission critical commissioning firm. He has more than 20 years of experience in MEP design and commissioning for mission critical and healthcare facilities.