At the start of the project, automation typically starts with a definition of what functions the system are to perform. This is defined in a PCN which is the Process Control Narrative or an FRS which is the Functional Requirement Specification. These documents define what the automation is to do within the plant. The programming team will take these documents to write the code, the PLC code that will execute the commands of the system. These documents are part of the technical specification and are really what you will use during commissioning to verify the systems, and this needs to be defined at the beginning of the project. So, everybody knows what they’re designing, and what will be tested later on in the project. The code is often written in parallel with the hardware design, so you’ll need to know the cubicle wiring, the interfaces to complete the code, and the software in advance that is developed by simulation on a hardware test bench.

The problem is though the code can’t be finished without the complete hardware design, you’ll be able to progress to some point in designing the code, but without the full complete hardware design, you won’t be able to finish it. It’s certainly a good idea to advance developing the code as much as you can, but if this is not properly managed this can cause delays and must be paid close attention. You need to plan your project sequence in advance to accommodate hardware and software design, because too often I find that the code is an afterthought and not developed in advance, to know more about how to set your project up for success, learn from here Project Mindset Lifecycle. Often, people will get towards the end of the project and say, okay we will need some code to run on these PLC’s, can we whip something together? Well, it takes a little bit more thought than that. 

Terms in Automation Systems

A proper project sequence in advance of the factory acceptance test is required, otherwise, you will not get the value needed from FAT, so a few terms to mention before we proceed.

The central control room is an important aspect of the project related to automation systems. This is where the on-site operators are located to control the plant process from a single location. The control room has all the HMI screens to allow the operator to see the status of any of the plant processes throughout the plant from a single location.

A PLC is the programmable logic controller and many of these are used within several cubicles distributed throughout the plant. the PLC connects to all the remote IO devices to control, and monitor the devices throughout the plant. Some of the PLC’s that I’ve worked with are Alan Bradley systems and Schneider PLC systems.

The DCS is the distributed control system or automation system that is made up of many devices throughout the plant including PLC’s electrical cubicles remote, IO devices and is the electrical automation system that controls and monitors the plant process.

HMI is the human-machine interface and is the graphical display of the plant process on several screens to allow the operator to monitor and control the plant process. The alarm screen is one of the important screens on the HMI where all the alarms from the plant are generated. The operator can view the alarm screen to monitor the state of the plant alarms are typically categorized as a priority, one priority, two priority, three, or something similar, so that the operator is alerted to the priority of the alarm being generated and indicates the response that they should take.

BMS, this is the building management system it’s really a subset or a separate system from the HMI, but it’s a specific central control function that can monitor the building management system; such as HVAC or lighting systems.

Now one thing to note is, that when it comes to automation systems and alarm screens there’s an infinite amount of information that’s available to the operator from all the various devices throughout the plant, and an HMI system that can gather all this information and present it in a logical manner to the operator is critical. If not designed properly you can end up with information overload.

You can have thirty thousand points that can be monitored throughout the plant. If you’re displaying all thirty thousand points, and the operator doesn’t know what to do with each of the status points there, then there’s not really much point in displaying that information. So, it’s critically important that the automation system can gather that information, decipher it correctly, and show the operator exactly what they need to know. Really, there shouldn’t be anything that’s displayed on the HMI that doesn’t require the operator to be aware of or require action if it’s displaying information just for the sense of displaying information, then that can often lead to information overload, additionally, that does need to be a priority when designing some of these systems.

Factory Acceptance Testing at the Site

When it comes to automation, a factory acceptance test should be performed as an integrated FAT, for further information about FAT, check this article out The Commissioning Process: A Step-by-Step Guide.  You’ll need both the hardware and the software to do this. The project must be planned in advance to accommodate this. If software verification is deferred to site, because the hardware design or manufacture is delayed or the software is not able to be written, because the hardware is not complete, there will be many problems at the site.

The factory acceptance test should include verification of the hardware with regards to electrical inspection of the equipment cable, pull tests conformance to technical requirements, verification of robust design, and installation so that the hardware is visually inspected and confirmed to meet contract requirements in the factory. But then, as well as the hardware, it should also include the software during the FAT, and that’s why it’s called an integrated FAT, because it tests both concurrently. The test configuration for the IFAT is that all cubicles are energized. All in the same room cubicle communication interfaces are established between all the cubicles if possible one of each of the remote IO devices is connected to the system, but if that’s not possible then IO devices may need to be simulated. The HMI displays operational hardware devices programmed with the preliminary code developed on the test bench, and preliminary set points and ranges applied, so prior to the IFAT you’ll want to define all the operating scenarios, and fault scenarios to be tested and determine the test inputs required for each scenario. When the hardware and software setup is available you will know what scenarios to run. You can run each scenario, and evaluate the results. You can repeat until the system generates the desired outcome, and that’s really the whole point of factory acceptance testing is by repeating the scenarios, then you can confirm the system is responding appropriately and making any adjustments when it’s not so that you know that the system is operating correctly through this post process.

This will also test the functionality of the HMI if at all possible. This is the time to have the operating personnel review the HMI screens and not just a screenshot of the screens, but the actual functionality. The earlier you can get this input, the better. This will avoid many wish list items later if you can get early operation buy-in of the systems because now is the time to make any changes to the HMI or software code. It’s much easier to incorporate now than on-site during critical path commissioning activities. If the operators are only seeing the HMI screens in the HMI system for the first time during commissioning, they’re for sure going to have comments or things that they would like to change, and it’s very difficult to do that while you’re trying to commission the system time. To get that input is much earlier in the project, and FAT is a good time to do that. 

Once hardware and software are both verified, the test setup can be disassembled and shipped to the site. This is a critical milestone because there will be a lot of pressure to ship the panels to the site earlier so that they can get installed. But without a full and complete FAT, then you won’t know that the hardware and software are designed correctly and are operating correctly. There does need to be a balance to confirm a proper completion of FAT in advance of disassembly and shipping those panels to the site. FAT is definitely an important part of the commissioning process and cannot be skipped during the project.

Often the DCS system is a large system that can control multiple areas in the plant, and it can be brought online in stages for different areas. The commissioning team needs to define this for the construction team in advance, so they know which equipment to install, and in what order as well. All remote IO devices are required to be installed on-site. The cubicles will only be able to be verified so far. If remote IO devices are missing, and when you get to the plant process even if you’re missing one device, that can potentially halt testing. If that’s a critical input to the control system that can’t advance the logic without that input, then you need to have that remote device installed in order for the system to respond, accordingly. All cables need to be pulled landed to the cubicles and terminated within cubicles.

The construction team performs the mega checks on all the power cables, as well as sometimes they will also test the control and protection cables for megger checks. The commissioning team will want to review this or maybe even oversee some of this activity to make sure it takes place. The other thing the construction team will do in all cases is point-to-point checks, and this is important for automation, because if there’s a rolled cable that’s wired backward or a cable that’s missing, then the automation isn’t going to get the inputs and outputs that it needs to respond accordingly. The commissioning team will definitely want to review this to make sure that point-to-point checks have taken place. The point-to-point and megger checks will identify any problems with the cables prior to further testing of the systems, and since the cubicles were tested extensively during FAT, wiring can be the largest source of error. It is important to complete proper point-to-point and megger checks for this reason. The commissioning team would like to focus on testing the functionality of the automation system, not troubleshooting wire wiring errors, so proper execution of this in advance is critical.

An integrated construction and commissioning schedule is critical. To align the mechanical completions with the commissioning sequence. And the development of this sequence will help the construction teams plan and manage the work in order to meet each milestone of the project and feed into the overall project commissioning sequence. This is one of the main reasons it is critically important to have the commissioning team involved early in the project to define this sequence. Development of the commissioning process should be done early in the project, in order that all project participants are aware of the deadlines they are working towards to achieve the project and service date.

The Role of the Commissioning Team

Each mechanical completion is handed over to the commissioning team as defined in each mechanical completion package. The construction team and commissioning team will both walk the systems confirming all temporary wiring has been removed and panels are in their final configuration, if you’d like to learn more about the role of the commissioning team, this is one of the lessons in our Mechanical Commissioning Training Course and Electrical Commissioning Training Course. It’s important that equipment is actually in its final configuration because there’s no point in testing the system if cabling still has to be replaced. Part of the process of testing the final configuration of the system is to know that all terminations are complete and everything is going to be working. If you’re going to be taking out cabling and replacing it, then essentially you’ll have to end up testing the system a second time to confirm that the final cabling is correct. An important deliverable for automation systems is the red line drawings, these are a deliverable from the construction team that is defined in each mechanical completion. If there were any changes made in the field during installation, the commissioning team needs to be aware of these particularly wiring changes. An accurate set of redline drawings is required in order to write the checklists and procedures for your automation testing.

The commissioning team will also use the set of drawings to make any additional changes if there is a change during commissioning mechanical completion for one system can take place, so the commissioning team can proceed with testing. Other areas of the plant may still be under construction while testing proceeds in another area, and it’s important to define your boundary isolation, have fully developed, and established LOTO procedures lockout, and tag out as cubicles often interconnect between each other even when one is de-energized. So, particularly for automation cubicles control voltages power levels, there may be multiple interconnect between multiple cubicles, and it’s critically important to define those boundary isolations. If one system is under test, while another is being constructed mechanical completion is critically important, as well from a safety perspective as each mechanical completion defines which group has care custody control of each cubicle, while some systems are under test while others are being installed. So, you need to pay close attention to your boundary isolation to keep everyone safe since a thorough factory acceptance test was conducted.

Pre-Commissioning

Pre-commissioning largely consists of verifying there are no installation errors, so in the case of automation pre-commissioning consists of applying first power to the PLC cubicles to confirm all voltage supplies are functioning correctly, communication checks to confirm cubicles can communicate amongst each other, as well as back to the control room. Loop checks verify the electrical circuit from the automation panel to each end device and confirm that the device can successfully communicate with the PLC cubicle and that alarms function correctly, you can find more information on loop checks here Pre-commissioning of Electrical Systems. This is done by stimulating the end device, confirming alarm set points are triggered correctly on the HMI that ranges and set points have been applied correctly by viewing the outputs on the HMI, and that all cabling is installed with no wiring errors, no backward signals. Interlock testing is also completed to verify hardware and software interlock functions correctly.

Pre-commissioning confirms that each electrical and automation component functions correctly prior to system or process automation testing. Following pre-commissioning, commissioning of the automation systems can take place. This involves operating all the equipment under the automation system control to confirm the plant systems operate as a system. Initial checks confirm that the automation can properly control the plant process before actually starting the plant process. This commissioning can take place under dry conditions or using clean water without chemicals applied to the system automation testing prior to startup operates the system under dry conditions or with clean water and no chemicals in the system and regardless of what the automation system is controlling. If it is a mechanical system such as; a wastewater treatment plant, a coal plant, or an electrical system such as; a hydroelectric generating station or HVDC system, the process to commission an automation system is relatively the same. A test methodology needs to be implemented to ensure plant operators can use the distributed control system to control any type of plant process from the central control room. 

PCN and FRS during Commissioning

The PCN and the FRS are used during commissioning. To do this, there are a couple of ways to verify the PCN and the FRS functions during commissioning. To further your knowledge of commissioning, check this article What is Commissioning?

One way is, in advance of commissioning, procedures are prepared, and identifying each test configuration is tested. This includes all normal operating scenarios and all failure scenarios with procedures written in advance. The team can better prepare for each scenario, and how the system needs to be configured in advance of testing. Procedures are written to cover all test scenarios that need to be verified since FAT to confirm the proper execution of the permanent installation.

The PCN and FRS will often include a text description of the plant process, as well as logic diagrams or state diagrams, so this is another way to commission the system. If you’re not able to prepare a procedure in advance, these diagrams can be used as a guide for commissioning scenarios set up in the field to execute each branch of logic. And as each branch of logic is executed correctly, the diagram is highlighted and testing proceeds to the next branch of logic. This method does not always provide 100 percent coverage of the logic, but this may not be required since the logic was tested extensively during FAT. This method does require the expertise of an experienced commissioning engineer specializing in the commissioning of DCS systems to confirm correct operation. However, this method is not recommended without doing a proper and complete IFAT as things can get missed.

All commissioning is performed from the central control room with support from individuals in the field, and because people are all over the place in various parts of the plan, a reliable communication method that everyone can hear is required. Since personnel is located in multiple locations for example; handheld radios are excellent to use for this since everyone can hear. Everyone can have one and hear the same conversations. Phones don’t work as well since the person receiving the call will hear, but others will not be aware of what is going on. It is important that everyone knows what the next steps are, and is part of the same conversation. It is also critical that all communication goes through the commissioning engineer. The commissioning engineer will be working closely with the control room operator to execute each test, and there can only be one test of the DCS taking place at a time. The commissioning engineer will coordinate all of these activities.

DCS commissioning will follow the same methodology as just discussed, but will always be specific to the plant process that is being tested. Commissioning experience is necessary from an automation perspective, and from a plant process perspective. As tests are being executed in order to understand the system and how it responds in real-time, the commissioning engineer will analyze the results in real-time, and determine the next steps for the commissioning process. Automation commissioning isn’t a short activity, it can take days, weeks, or months depending on the complexity of the system. For example, the HVDC project I worked on had roughly six months of automation testing that we went through to confirm all the controls of the HVDC system. The stability controls and the proper reaction of all the normal, and fault scenarios of the system needs to be expected and planned for in your overall commissioning experience sequence. Once the automation system has been confirmed to meet the requirements of the PCN and FRS, it is now time to start the plant process for the first time.

Startup

The startup will occur in stages for each system of the project the automation and system interfaces with all equipment on site. As each component is brought online for automated control in stages, the automation system is used at this point for a remote startup of equipment, since this is how the operators will operate the plant. The startup consists of process fluids being introduced, chemicals introduced to the system, or high voltage current being transferred into the system for the first time. As each component of the system is started, the process is fully verified using the automated control system. This confirms that the plant process is functioning correctly, as well as verifies that the automation system can successfully control and operate the plant processes. The automation system contains the logic for automatic control of the plant process, so all operation scenarios will need to be tested to confirm the correct response of the system. Fault scenarios are tested to confirm the automation system reacts appropriately.

The startup phase will always be specific to your project depends on the plant process. The automation system is controlling once all plant processes are operational after several weeks or months of testing. It will take place to confirm the plant processes are operating correctly as well as to define, as well as fine-tune the system. Fine-tuning consists of adjusting settings within the automation system to operate and optimize the operation of the plant. Once all the pro plant processes are started, the system undergoes performance verification. The system is monitored for a period of time with the uninterrupted operation, so the performance verification period could be a series of days or maybe even a month of uninterrupted operation. And should the operation be interrupted during that time such as a priority one alarm or whatever is defined in the contract as indicating an upset, the performance verification period restarts until the duration of the uninterrupted operation is achieved. Once the performance verification period is achieved, the system can be handed over to the owner.

Operational Readiness

Operational readiness is an important aspect of your project, particularly for automation systems, for detailed information, feel free to check this article Operational Readiness . Operators may be used to older systems that maybe didn’t have automated screens or were more manual control. So there needs to be more of a focus on operational readiness to make sure that they’re comfortable with operating systems electronically and remotely through a computer HMI system.

A DCS simulator is often a good idea to include in the project to use as a training device to help operators become familiar with the new systems. The simulator is essentially an exact replica of what the HMI screens will look like and a replication of how the systems will react when various portions of the HMI are operated. It gives the operators a good opportunity in advance to become familiar with the systems, and learn how to remotely control the systems through the commissioning process. Operators will also become familiar with HMI screens, and how to operate the plant processes and ongoing maintenance.

Any changes to the automated system can be challenging as this requires a specialized skill set or may require vendor support to update the automated systems. One thing to note is that changes to the automated automation system are likely not required during the warranty period without having the vendor involved. You can’t modify the system without informing the manufacturer or the vendor, since that will likely void the warranty. The owner may choose to keep a record of changes to be made to the system at the end of the warranty period. Documentation and specialized tooling or programming equipment are required for the operation and maintenance of the DCS. This will be part of the operational readiness activities that the commissioning team can support.

Cyber security again is a large requirement of the system, and this needs to be prepared in advance of hand over to the owner, for more information about cyber security, please learn from here Cyber Security During Commissioning. Any software patches that are issued by third-party vendors or firmware upgrades issued by manufacturers need to be applied to the system in order to address future security issues and keep the system compliant. Cyber security is a huge topic on its own. 

So, that’s an overview of the automation commissioning process from start to end. 

If you’d like to learn more about the commissioning and startup process, please join our free three-day mini-course. The course is free and flexible to take any time online. It gives you a good start so that you can understand or have an early understanding of the commissioning and startup process. 

Question and Answer Session

Should FAT tests be witnessed by Cx Authority or only certificates be submitted to Cx for documentation? 

My strong recommendation is that the commissioning team participates and witnesses FAT tests. There are two main reasons for this:

The first main reason is that this allows the commissioning team the opportunity to become familiar with the equipment since they’re going to be working with it on-site when it arrives once it’s shipped to the site.

And the second reason is if there are any issues that are encountered during FAT, sometimes they’re corrected during FAT, sometimes they’re noted on a deficiency list and will be addressed at site. But that’s probably the main reason, it’s important from a continuity perspective that the commissioning team has that knowledge of what went well, and what didn’t work well during FAT, so that when the equipment is at site those issues can be addressed and maybe other things to watch for. If there was a problem with a particular function of the FAT, then the commissioning team would pay closer attention to that when it’s tested on site.

That’s my strong recommendation, the commissioning authority should participate and witness a lot of the FAT tests, particularly for automation.

 

For loop check, which one is better? Closed loop or open loop?

It may depend on the complexity of your system. You may choose to start with an open loop test, and then go to a closed loop test. It may also depend on the design of your system, and if there’s feedback that applies to the plant process that you need that closed loop function for. There’s different aspects to both, and the commissioning team will likely choose one or the other depending on how the system is to be tested.

So, of the two methods that I described, it’s important to know which will be done in advance, so that those lists of tests that are going to be completed are known in advance. Yes, we need to do these open loop tests, and we need to do those closed loop tests, to plan that out in advance, you can find more information on loop checks here Pre-commissioning of Electrical Systems . If that discussion is taking place on the morning of testing where everybody’s showing up and saying okay, how do we want to test? Things will not go well.

The more advanced thought and preparation you can put in advance is going to help define what is the best test methodology to apply to those systems. If it should be closed loop or open loop testing, and it will just depend on your system.

 

As a commissioning electrical engineer, how do you deal with situations where FAT has been skipped because of schedule constraints to ship the equipment to the site? 

That’s a very good question. It’s not all that uncommon where there’s pressure to do a partial FAT or skip it completely and saying, we need to get this hardware to site so that we can get it installed. I’ve seen it happen too many times, and I’ve never seen it work well because at that point then, you’re just deferring the automation and PLC code design until later in the project. Likely what happens is, it’s being designed as it’s being tested, and that is never a good situation.

I’ve seen lots of catastrophic failures occur because of that. A proper factory acceptance test of the software wasn’t completed where the PLC code didn’t function correctly at all on a large rotating machine. Oil wasn’t applied to the bearing of systems, and there was major damage that caused months and months of delays and huge cost to repair.

When there’s pressures like that, the best thing that the commissioning team can do is voice those concerns and express the risk to the project by deferring those test activities to site, to strongly voice that automation needs to be tested during the FAT.

But assuming that was unsuccessful, the executives or others that are making those decisions has determined that the panels do need to ship to site, then it’s still strongly encouraged to test the software at least on a test bench prior to programming it in the field. If it’s being programmed in the field for the first time, and only being tested that first time, then it’s only going to add significant delays to the project to troubleshoot the logic in real time as it’s being designed in the field. So, best case would be at least test the software on a test bench system or a simulated environment to prove that it works before it’s programmed in the field, while the cubicles are being installed.

 

Who is responsible for preparing PCN and FRS documents? 

In my experience, that’s typically from the design group. If they’re the group that’s designing the hardware cubicles themselves, and the devices in each cubicle and the PLC’s, and how they’re interconnected, that same group or a branch of the design group would also be designing or preparing the PCN and the FRS indicating how the cubicles are to operate together, and how they’re to control the plant process. That hardware design and that software definition in the PCN and the FRS really have got to come from the same group indicating what their intentions are, for how these systems are going to work together. It wouldn’t fall with the construction team, and I haven’t seen it fall with the commissioning team. It needs to be an upfront design function early in the project, so that whoever’s writing the code knows what the logic needs to execute, and how to implement. It’s paired with the hardware, so that it can operate as a system.

 

Are PCN and FRS included in the project contract? 

That has been my experience. Yes, the PCN and the FRS are included as an appendix to the contract, so that the contractor that’s bidding on the job can see the functions to be included and get a pretty good idea of the scope required for programming to implement the functions and the processes defined in the PCN and FRS. It’s certainly a good idea and a good risk mitigation activity to include those documents in the contract. If they’re not included in the contract, then that opens up the discussion for unknowns. If they come out after the fact, then the contractor didn’t know or wasn’t aware of certain aspects of the project. That discussion can get quite challenging. Yes, my experiences the PCN and the FRS are included in the contract. I do think that’s a good idea. Read this article to understand more about contracts Commissioning Contracts 101: Everything You Need to Know

 

A sub-system cannot be commissioned because another related sub-system is not ready yet. How to deal with this condition? How to prevent potential trouble during plant start up later on ? 

This is definitely a scenario that often occurs, that one sub-system isn’t ready when another is needed. That can be mitigated in advance by defining the mechanical completions, and when each is required well in advance to make the construction team aware of priorities for which systems are required, in what order. This compiled article about mechanical completion will help, please feel free to check Mechanical Completion, Substantial Completion, Final Completion

However, despite everybody’s best efforts, there still may be delays of a particular mechanical completion, or one of the subsystems is not ready. That’s always going to be a judgment call to see how can the system still proceed with tests. In the absence of a particular portion of the project, maybe there’s ways to mitigate certain inputs or outputs from that portion of the project that can be simulated. You’re going to have to see what portion is missing and what is not. If the one subsystem that’s related is not ready, and it is critical to the plant process then maybe you’re stuck. For example, if the auxiliary power distribution system is not ready, then it’s going to be very tough to commission a lot of the cubicles without power to be able to energize them. You’re just going to have to see what are the fallback scenarios, or what portions could be commissioned in the absence of the one system that is delayed. Certainly, when you get to the actual startup of the plant, you’re going to want to have all those systems available. If there is a critical portion of the project that isn’t available yet, then that is going to impact startup and potentially cause significant delays. I wouldn’t recommend proceeding with starting up the plant without that critical piece available to be able to proceed, unless there’s appropriate operational or mitigation scenarios that can be put in place.

Who usually makes the commissioning procedure? 

That would be a function that’s performed by the commissioning team. Through the process of systematization of the project, all the equipment in the project is going to be grouped into subsystems and systems that’s done by the commissioning team. People on the commissioning team are going to write a commissioning plan which is the higher level document indicating how all these systems are going to be coming together. And then the more detailed commissioning procedures associated with each subsystem or system of the project. The commissioning team will write the commissioning procedures looking at the as-built drawings, redline drawings, the PLC code, and determining what is the best method, or what are the detailed steps required to prove that the automation system is meeting the functions defined in the PCN and the FRS. That’s another reason to have the commissioning team involved early in the project to develop commissioning plans and commissioning procedures based on the design documents and the contract documents. It will take some time to go through the PCN and the FRS, and define what is the best test methodology, or what are the operating scenarios, what are the failure scenarios that do need to be tested, and defined in your commissioning procedure. So, that’s who writes the commissioning procedure is the commissioning team, likely the electrical commissioning lead or mechanical commissioning lead depending on the system under test.

 

Does the Commissioning Team have the final say on which Inspection and Test Report to use in the project?

The way I like to define this is that, nobody can force the commissioning team to test something if the commissioning team doesn’t feel that it’s safe to do so, or that it’s actually ready to be tested. In that regard, yes, if the commissioning team feels that a system hasn’t been verified or hasn’t gone through the proper ITP process, then that would definitely be an assertion that the commissioning team should put forward. To say that, this system isn’t ready to be tested, isn’t safe to be tested, or isn’t complete.

There is hopefully lots of ITPs specific to each portion of the project. If you’ve got one portion of the project, that’s a large mechanical rotating piece of equipment, there would be a series of ITP’s executed by the construction team specific to that piece of equipment. There would be ITP’s related to the electrical systems and automation systems and all of those components. So, the right ITP would have to be used for the right systems, and if the commissioning team felt that wasn’t taking place, then I would definitely encourage the commissioning team to raise that concern and push back on the construction team saying that, this isn’t being tested correctly, and commissioning team isn’t receiving properly inspected equipment.

If that’s not the case, then the commissioning team is only going to be receiving equipment that may be still deficient or hasn’t been inspected correctly, or maybe isn’t even fully installed yet. So, I would hand those back to the construction team, and say no these are still deficient. The deficiency process can be used in that case identifying deficiencies as either a Type-A, Type-B, and Type-C deficiency, and any deficiencies that are Type-A would be show stoppers, saying that testing can’t proceed until this particular deficiency is resolved. Any of those Type-A deficiencies need to be resolved before being passed to the commissioning team.

 

If we have done FAT of equipment at the factory/ex-work, then what is to be done for Site Acceptance Test (SAT)? 

If a proper FAT has been done, then that confirms the hardware design, and the software design, and the actual implementation of both. That is a huge risk mitigation activity, leaving mainly installation to be verified of the cubicles on site and that’s what the site acceptance test is. Once the cubicles are shipped to site, they’re placed on the concrete pad, power is wired and all the control and protection cable is wired to the cubicle. That’s when the site acceptance test takes place from two aspects, one from an installation perspective. The cubicle will be walked through and inspected by electrical inspectors to confirm that it’s installed correctly, that all cabling to remote IO devices are properly pulled, glanded, and terminated into the cubicles, that everything looks proper and is a neat installation. That would be one aspect of site acceptance test. Then the other aspect would be verification of proper function of the software. Since this was done in FAT, this could be done in a more limited manner knowing that the software works, but you want to do a site acceptance test on the software to confirm that there are no cabling errors. It’s wired in the same manner in the field as it was in the factory, just more distributed amongst the plant confirming that everything is interconnected, everything is communicating correctly, and the plant processes can properly function as part of that site acceptance test. That would be the main function of the site acceptance test, just is to confirm the installation, the cable pulling, and basic software function before getting into more advanced commissioning testing.

Who is the right person to do the FAT when the commissioning team is still not available during procurement of such automation equipment?

So if that may be the case that the commissioning manager or a few key members of the commissioning team roles aren’t filled when equipment is being procured, then obviously they won’t be able to participate in that FAT. I would strongly encourage at least some of the key members of the commissioning team are hired and added to the project early on so that they can participate in the FAT. But I recognize that’s not always the case due to budget constraints or other issues. When the commissioning team is not available to participate in the FAT, then the best group to participate would be the design group – the people that defined the PCN and the FRS. The people that designed the hardware, they’re the ones that are going to want to be at the factory to inspect the cubicles and confirm that cubicles are assembled to meet the requirements per their design. Likely, even if the commissioning team is available, it’s probably a joint witness of the FAT from the design team that’s leading the procurement as well as the commissioning team, since both groups can add value to ensure that the equipment is built correctly, between what’s tested in the factory, and what’s tested in the field.

 

If the contractor forced me to do TAB for the HVAC system without the automation system completed because of a deadline like motorized dampers are opened mechanically, is it okay?

Sometimes, the HVAC systems are somewhat standalone where the HVAC controls can operate the air handling units, the dampers, the smoke management systems, and can operate those systems without the remote capability to the control room. So, certain aspects of the HVAC could certainly be tested and balanced, and it is likely okay to proceed. When you’re doing that testing, then of course the status and control from the HVAC control systems back to the control room would not be available. If it proceeded in that manner, and the HVAC system was brought online without the PLC or the automation system, then HVAC systems could be operating independently. But, you would definitely want to go back at a later point once the PLC or the automation system is available to confirm those points are available to the HMI system, and that it can be properly controlled and monitored from the control room. It will depend on the construction sequence and priorities within the project. I know that HVAC is one of the critical systems to get on earlier – same with lighting and control. Those two building auxiliary functions are often required earlier to get some heat or cooling in the building and to get some light, so people can continue to work. It may be okay to advance some of those systems knowing that at a later point when the automation systems are available, you would want to go back and verify those interconnections to the automation system.

 

Is it possible that the commissioning procedure should be given to the owner or consultant before proceeding to start the commissioning activities? 

Yes, I do think this is a good idea. The commissioning procedures definitely need to be written in advance. You won’t be able to write those kinds of documents in real time as you’re proceeding with commissioning – that will cause delays. When these commissioning procedures are written in advance, it’s a good idea to share them with anybody that might want to see them, not necessarily from a detailed review perspective, but the owner may want to see the procedure and know that proper documents are in place before proceeding with commissioning. Maybe even more so for the consultant, since the consultant will likely get involved in the commissioning process. Maybe from a subject matter expert perspective, they’re going to want to review the commissioning procedures to make sure that everything is captured, and that everything is being tested correctly. Certainly, I would share those documents with both the owner, and the consultant in advance to allow them the opportunity to review and provide any inputs that they feel are important to the commissioning process. By doing that, it’s certainly going to help achieve buy-in from all stakeholders in the project to make sure that they’re happy with the commissioning process, and are seeing the tests completed that they expected.

Who is responsible for preparing LOTO procedures? 

The LOTO process can be accomplished by a few different people, but you definitely want to choose one authority that is the lock out tag out authority for the project. If the owner has operations personnel available and on site already, then that’s the best group to be responsible for the overall lock out tag out process and be the LOTO authority. Then, other groups would come to the operations group and make requests for any LOTO that’s required. So that could be someone from the construction team approaching the LOTO authority to say that they need a particular piece of equipment locked out so they can continue construction in a certain area. That may be the commissioning team that’s going to the operations team saying that they need to switch in or switch out or start up a new piece of equipment.

So, with regards to preparing the LOTO procedure, that would have to be the group that’s preparing to do the work. If the construction team knows that they need a certain aspect locked out, then they would identify this one, two, three whatever breakers need to be locked out, and approach the LOTO authority to say, this is the lockout that they’re require in order to complete their work. The LOTO authority would then review that to confirm that everything has been identified properly, that nothing has been missed, that all the proper lock out tag out is in place, and respond back to the construction group saying yes, this is correct, you’ve captured everything, this is safe, and issue the permit to work in order for the construction team to proceed with that work. Same with the commissioning team – the commissioning team is starting up a new piece of equipment, then they’re going to work with the owner’s operator who is the lockout tag out authority to say that, okay we want this large transformer energized. We want it unlocked so that we can apply high voltage power to the system. The lock out tag out authority is going to review that and respond back yes, okay everything is in place, we can proceed with this lock out tag out, and issue the permit to work to do that.

 

Is it okay to override or bypass a new subsystem during pre commissioning of another subsystem for sample gas turbine compressor subsystem? Because the new system has tie in with existing plant? 

It’s always not a good idea to override or bypass certain aspects of the system, but there may be certain aspects that are reviewed during your lock out tag out, where you do need certain things locked out or tagged out in order to be able to proceed with certain energizations of the system. With proper planning or review of drawings, this can be reviewed in advance to see if this makes sense. You certainly don’t want to do this on-the-fly to say, okay we’re going to bypass this to proceed. There does need to be some proper planning to know the state of the system, and exactly what needs to be locked out in order to properly integrate new systems into existing systems. That could be part of the lock out tag out process where certain things are disabled or locked out in order to proceed with aspects of the work, but that does need to be reviewed pretty thoroughly. 

 

Who will prepare test procedures and checklists prior to commissioning?

This is something that the commissioning team will perform in advance of on-site commissioning. All the checklists and all the commissioning procedures will be prepared by the commissioning team with input from others, or maybe prepared with significant input from the consultant. The commissioning team or maybe the consultant as a member of the commissioning team will prepare the checklists in advance, the commissioning procedures in advance, and this is the best thing that can be done to know that all the documentation is in place, and know what exactly will be tested before getting to on-site commissioning. With those documents prepared in advance by the commissioning team, when you get to on-site commissioning, execution of those procedures during on-site commissioning will proceed much smoother.

How to solve the problem if there is a vibration issue in a cooling water pump, the suction line and support pipe was not changed, but vibration is out of tolerance, even though the design was already approved?

That to me would sounds like potentially an installation error. If the pump pad has been formed/poured and the pump installed, but there’s vibration issues that are coming out of that, it sounds like it may have not been installed correctly. However, I guess it could potentially be a design issue. If the piping is out of tolerance significantly, then that could be putting a lot of stress on the incoming and outlet piping that’s causing vibration issues on the pump. Hopefully that’s not the case, if it was, then that should have been caught during installation. If it is an installation issue, and there are huge vibration elements coming out of on site testing, then that would have to go back to the construction team and indicate this doesn’t necessarily meet the design and would need to be reevaluated. Typically, the vendor would be involved if their pump is vibrating significantly, because they’ll want to assure from their equipment perspective that it has been installed correctly, and it’s not an actual equipment issue. But to me, that does sound like it could be an installation issue, and I would put that back to the construction team or the inspectors to confirm that it has been installed per the drawings, since it shouldn’t be vibrating that much.

 

If just in case the automation equipment either in a sub system or system, test procedures have been missed during FEED, could the commissioning team carry out the job without prior approved test procedures, and the commissioning engineer will develop a test procedure to carry out the test in order to complete the test safely. 

This could be an option, it’s not the preferred option. The much preferred option is to have the procedures prepared in advance, but that may not be the case for whatever reason. If that’s the case, like I discussed earlier you can take the PCN and the FRS documents, and the logic diagrams that are included, and an experienced commissioning engineer can go through those logic diagrams and test each branch of the logic. Looking at the logic diagrams, you can see how the system should be reacting based on certain inputs, and an experienced commissioning engineer can set the system up with those inputs, and see how the system reacts to confirm that branch of the logic. If the system reacts accordingly, then you can move on to the next portion of the logic branch. After highlighting that piece of the logic, now that can work on a piece by piece basis, but what can get missed through that is the overall system function. If one portion of the logic is dependent on another for real complex designs, then that can only get you so far. So, there really should be some upfront analysis to determine the test procedures or test scenarios to be conducted rather than going through that in real time in the field. It could be a fallback scenario to get through a portion of the testing if required. That is a real reason for the commissioning team to be involved early to push for development of those procedures in advance, so that they’re not missed to not be placed in this scenario, but could be mitigated in some aspects for simpler systems.

 

What is a Close Loop / Open Loop? 

Closed loop would be feedback from the system that’s coming back into the automation system, you can find more information on close loop/open loop here Pre-commissioning of Electrical Systems. If there’s an output that’s going to a certain aspect of the plant, and that remote IO device is then providing status back to the automation system that would be closed loop where the automation is looking for a response in order to determine function of the next portion of the logic. Open loop would be just looking for say, that you can control the pump on and off, not necessarily looking for the status of that pump to see that it’s actually turned on and off with that control and feedback process. You close the loop to know that pump command on has been issued, and then looking for the status, confirmation that the pump status is actually running. Open loop would just be confirming one of those communication paths.

 

Does Cx specs or test methodology supersede standards’ ones? For example, if NEBB standard gives a tolerance of 10% for AHU’s outlets airflow measurement but Cx insists on 5% tolerance?

The standards are always the baseline to be achieved during commissioning. You may have a contract or your technical specification may require a tighter tolerance than what’s required in the standards. So, your example of a tighter 5% tolerance, that may be a requirement of the technical specification. If it’s going in that direction where it’s looking for a greater performance or a higher tolerance, then yes the contract could demand stricter requirements. If in the other case, the technical specification is looking for a wider tolerance, say a 20% tolerance, then you would at least have to meet the standard to achieve 10% tolerance so that you’re compliant to the particular code or standard that applies to that part of the equipment. That really should be caught during design. If the technical specification is asking for a wider tolerance than what’s required by the codes and standards, then that’s probably an error in the technical specification. It should be at least meeting the required of the applicable local codes and standards and not relaxing them in that case. But, in the other case if it’s asking for a tighter tolerance, and that might be a requirement of the plant process where it needs a finer control or finer tolerance of the equipment in order to achieve the plant process, and that could be the case. Yes, that technical specifications would be requiring a tighter tolerance.

 

Follow up, on PCN or FRS shall be included in the contract, what other standard items needed to keep an eye on in the contract before commissioning the project? 

Definitely, your technical specification. I would consider the PCN and the FRS one subset of the overall technical specification. The complete technical specification may be made up of a few different appendices.  The contract is the other critical document that the commissioning team will want to pay attention to, because that will define what we just discussed in the last question. Any tolerances that have to be met or any additional technical requirements that the system needs to meet, always the commissioning team will want to be familiar with the overall contract, but particularly the technical specification. The overall contract is certainly a good thing to be aware of from a commissioning perspective, because the commissioning team is really that specialized quality assurance group to ensure the owner that the project that’s being delivered to them is meeting all requirements of the contract with a strong focus on the technical specification. All the drawings or any codes or standards or anything that’s in the technical specification as well as the PCN, the FRS are important documents to keep an eye on in the contract.

 

Are the activities between SAT and pre-commissioning the same? If there’s a difference, what are they? 

Let’s consider the SAT as maybe a vendor function. So what occurred in the factory was one set of verification of the equipment. But, the vendor may choose to travel to site or maybe a requirement by the contractor to come to site and inspect the equipment after it’s been installed. This would be one aspect of site acceptance test where the vendor confirms the equipment is installed correctly, initial startup of the equipment and no vibration issues. Yes, the equipment now meets the technical requirements. The vendor’s happy with it, and then it can be used for further-on commissioning activities. Some of those further-on activities may be further pre-commissioning now that the motor is actually installed, it’s coupled to the pump, and some of the pre-commissioning activities are completed there. Further to some of the bump testing, or vibrational testing now that it’s coupled to other equipment. Other pre-commissioning that maybe wasn’t part of the vendor’s responsibility would be communication to the remote IO systems or automation systems for remote control, and operation that would be outside of the vendor’s scope of work and maybe outside of their SAT inspection activities to confirm proper installation. Those would be some of the pre-commissioning activities that could take place to make sure that the automation system can communicate with that piece of equipment, and that the proper settings, and ranges are applied specific to that piece of equipment as part of pre-commissioning. That could be some of the differences.

 

What is the procedure to do offline download, if you want to download and modify logic in a system, but your system links to many systems? 

I’ve recently gone through this issue, and it may depend on how your system is designed. If it’s a redundant system, and it is hot swappable you may have the option to upgrade one of the PLC controllers on the primary system while the secondary system is operating, and swap those out without interruption to the system. If your system’s not designed in that manner either that they’re not hot swappable, or they aren’t redundant, then it will require potentially an outage to download the logic to get a copy of it. But, when the new modified code is applied to the system, that likely requires a temporary shutdown of the system in order to apply that logic, because some of the equipment it won’t be connected to the PLC. In that case, we will need to operate in manual mode. The actual PLC system won’t be able to control and monitor the status of that equipment, so you will have to plan for that to have manual operation, while the system is updated with the new modified logic.

If the system integrates with many different PLC racks or many different cubicles, maybe there’s a larger portion of the system. Maybe multiple cubicles that need to be taken offline for a period of time while they’re updated. This is quite common during commissioning as issues are encountered or updates need to be made to the system. These need to be planned for. Once the system has been started up, it is actually controlling the plant processes, but before hand over to the owner there may be some of these final changes that need to be applied to the system to address some of the outstanding deficiencies. There may need to be a period of time where this is planned for where several issues are consolidated into a single upload so that this doesn’t have to be done multiple times, and the system taken offline for a period of time in order to apply the new modified logic, and close out the remaining deficiencies during the running commissioning test for the motor.

During the running/commissioning test for the motor, who will be conducting the test? Electrical commissioning group or mechanical commissioning group?

Very good, question! There are aspects of both. The electrical commissioning group is definitely responsible for getting the auxiliary power to the motor. The electrical feed from the electrical distribution system, switchgear, MCC’s, getting the power to the pump or the motor from that point on. Maybe, there’s some additional assistance from the electrical team to get the motor up and running, but from my experience for testing pumps and motors, that’s a mechanical commissioning group’s responsibility to perform.

The bump test of the motor, that would be a mechanical group function in my mind. Assuming that the power polarity is connected properly to the motor, and the bump test proves the motor is operating in the correct direction, the mechanical commissioning team would continue to exercise the pump, measure flow rates, pressure rates from the pump, confirm that three points on the pump curve are measured, and the pump is operating correctly, no vibration issues. A lot of those are the mechanical commissioning groups responsibilities. Now, that certainly doesn’t mean that the electrical group can’t be involved. Hopefully, we’ve got a tightly knit construction and commissioning team, and everybody’s working well together. If there’s issues from one group to the other, and they need to collaborate or get one group involved, then the mechanical group may choose to get the electrical group involved and work together to figure out the issues, assuming that everything’s working as it should.

 To understand more about the roles and responsibilities of the people who are involved in commissioning, we offer our in-depth training, check these links out Mechanical Commissioning Training Course and Electrical Commissioning Training Course

What’s the difference between cause and effect and dynamic testing procedure? 

Okay, let’s try and take a stab at this one. I hope I understand your question correctly. Cause and effect in my mind is a troubleshooting method if something’s not working, then you may have to go through a formal process like a cause and effect analysis, root cause analysis, fish bone diagram or something like that to show the documentation to the owner of how a particular issue is being troubleshooted.

Maybe the biggest difference, I would use the terms dynamic testing procedure versus functional testing procedure. Functional testing would be more of a logic state diagram where you apply an input, and you see a discrete output from a system where it’s very much binary one or zero, that would be kind of functional testing, or cause and effect testing. In the case of dynamic testing though, it’s not as discrete. Dynamic testing would be something like testing the stability controller of an HVDC system where it’s not discrete, either on or off, depending on how you’re stimulating the system. The stability controls will react in a dynamic manner, and you’ll see that more of a waveform where you’ll see maybe a disturbance to the HVDC voltage, and then it’s damped out over time to obtain the desired frequency response of the system.

That would maybe be the largest difference is more of a discrete binary approach during cause and effect or functional testing, where dynamic testing is more of the squiggly waveform or review of response of the system in a dynamic manner to see how certain functions of the automation system, or stability controls react to damp out signals, or react in a manner to dynamically control the system as opposed to discrete logic. That’s probably a discussion that we could get into quite a bit of depth on with regards to dynamic testing of automation systems, and maybe that’s a discussion for a future webinar.

How is safety integrates into factory testing? 

Safety is an important aspect through all stages of commissioning, learn more about safety here Safety During Commissioning. There will definitely be certain aspects of safety that you’ll want to inspect in the factory as part of that factory acceptance test plan if you’re testing the automation. If you’re testing electrical mechanical properties, you’re also going to be confirming and testing that mechanical safeguards are in place. For example, that the vendor has in fact integrated all of the appropriate safety aspects that have been specified in the technical specification in the factory before it even leaves to go to site. Once the equipment is at site, you may want to ensure that those same safety aspects are still in place. But, there may be additional ones that are required now that the system is in the plant and installed. You want to ensure that these additional safety aspects are in place. It could be that in the factory acceptance test, the equipment was tested more on a standalone nature. And maybe once it’s installed at site, there’s particular interlocks between the systems to confirm, and that the interlocking between equipment is properly implemented at site. Now, that’s something you’re definitely going to want to test once the equipment arrives at site as well.

 

Everyone is welcome to join our next webinar. We’re happy to answer questions like these, there have been fantastic questions, and I love this process to answer your questions. So, keep them coming, and I’ll see how I can help you out.

Please, watch the full video of our live webinar. The presentation and Q&A portion provide knowledgeable information about automation commissioning.