In North America, cyber security compliance is governed by NERC. NERC stands for North American Electric Reliability Corporation, and the NERC CIP standards are what governs existing facilities and new projects to provide guidelines on physical and cyber security of in-service assets. All in-scope NERC CIP assets must be NERC CIP compliant before being connected to the bulk electric system.

NERC Assets

Let’s start with what is a NERC CIP asset? CIP stands for Critical Infrastructure Protection. This is any device with an addressable port, so this would typically be your CAT5, or CAT6 cables or your LAN connections, but it could be others. Anything that has an addressable communication port – it could be a device that is connected to a network or not connected to a network, and it could also be an internal network. So, if your internal network is strictly within the perimeter of your fence line, and has no physical connection to the outside world, it is still in scope for NERC CIP. So, what this isn’t would be a serial connection – like an rRS232 connection, that’s not an addressable port so that’s not in scope for NERC CIP, but anything else that does have an addressable port is in scope for NERC CIP. Some examples would be if you’ve got a PLC cubicle out in the field, and it’s got a communication card in it, a network card, that would be an in-scope NERC CIP asset. If you have a transformer control panel, and it’s got an internet interface card in it, the entire panel would be NERC CIP in scope. For NERC CIP, all your control and protection devices, all your protection relays, typically communicate back to the HMI system. Those are all in scope for NERC CIP. Pretty much every electrical device that has a communication port that is addressable is in scope for NERC CIP.

NERC CIP Reliability Standards

Let’s review some of the NERC CIP reliability standards. I won’t go through them all because there’s quite a few, but I’ll highlight some of the more important ones here. First one is CIP-005  – this is your electronic security perimeter. This would cover anything for any firewalls that you have set up in the system. All your router settings, any of the electronic devices that define the perimeter of your network would be classified as an electronic security perimeter. So, even if there’s no externally routable connection, your internal network and the perimeter of devices around that internal network form your ESP. All your ESPs need to be documented and verified to be compliant and typically you would have multiple ESPs and each needs to be identified.

CIP-008 Incident Reporting and Response Time. When a NERC CIP incident occurs, how is it reported? What kind of incidents are reported? And what is the immediate response? When an incident occurs, not necessarily all incidents are reported immediately. NERC CIP defines an incident as a potential violation or PV. Different classifications of PVs may need to be reported immediately, but typically a lot of them wouldn’t. There’s monthly reporting and annual reporting when you self-declare or self-report your PVs for review by regulators.

CIP-010 is Configuration Change Management and Vulnerability Assessment. How are NERC CIP assets disposed of if you’ve got an old asset or a failed piece of equipment or maybe something that needs to come out of the system? You can’t just throw it in the dumpster behind the building. It does need to be disposed of properly, because it does contain CIP confidential data, so it does need to be disposed of in a safe manner to make sure that nobody can access that data. It could be shredding the piece of equipment. It could be as simple as hitting with a hammer so that the data is no longer accessible on the device. But you do need to document and dispose of old assets and CIP assets properly. Then, when the new device is installed in the system, how is the NERC CIP asset replaced? The new asset needs to be configured, documented and confirmed compliant before being added to in-service NERC-CIP assets. The configuration would be downloaded from the previous device, or from your historical records, so that it’s configured to be NERC CIP compliant and all the tasks and evidence have been gathered to demonstrate that device is NERC CIP compliant. A vulnerability assessment is required before you place the equipment in service and must be updated accordingly if there’s any changes to the system.

CIP-014 Physical Security, this would be your card lock systems or your controlled access to the facilities, your perimeter fencing or physical barriers. How is access requested and approved to get within the NERC CIP physical perimeter.

The last one that we’ll highlight here is CIP-011 Cyber Security, this is really the meat and potatoes of what NERC CIP is all about. This covers your port scans. your screenshots of security software installations, documenting firmware, and software versions as well as the business justification for open ports. We’ll go through each one of these in more detail.

BES Cyber System Categorization

The first thing to be done is to categorize your NERC CIP assets – this is your BES cyber system categorization, your bulk electric system cyber system categorization, and it’s classified in three different impact ratings – high impact rating, medium impact rating, and low impact rating. There are some ratings in between. You can get a Medium+ impact rating, but we won’t focus on those for right now, we’ll focus on high, medium and low.

A high impact rating facility would be your control center or your backup control center – anything that if that central control center was compromised, and you no longer were able to access those facilities or use them to operate the system, that would have a high impact on your bulk electric system.

A medium impact rating would be any of your in-service generation, transmission facilities operated at 500 kilovolts or higher, or any of your transmission facilities that are identified as essential to nuclear facilities. This is probably where you’ll find the bulk of your NERC CIP efforts is in your medium impact rating. Typically, a lot of the assets you’ll be dealing with will fall into one of those categories.

Low impact rating would be your typical transmission stations or your substations that aren’t necessarily essential services. If you lost that particular substation, you’ve got options to route distribution power through other stations or other lines. The impact of losing one of those facilities would be low. So when it comes to NERC CIP, there are two portions of activities that need to take place. There’s NERC CIP tasks, and there’s NERC CIP evidence to be gathered. Let’s go through each one of those right now.

NERC CIP Tasks

The largest bulk of effort as part of NERC CIP compliance is the port scans. Port scans are performed on all NERC CIP assets, and there’s a few methods to do this. Command line execution of NETSTAT will scan all the IP addresses and all the open ports on your system. There’s other software packages you can use to do this such as a NESUS vulnerability report that can be generated by a separate software package. Each of these software packages scans all your IP addresses and confirms which ports are open, and which ports are closed. Your contract may require to use one or the other methods for port scanning. For example, the last project I work on required NETSTAT to be used for port scans. But either way it’s performing the same function to confirm which ports are open, and which ports are closed. These scans can take a long time. They can take many hours sometimes overnight, and it’s not uncommon for these scans to only get eighty percent complete, and then they fail and need to be restarted. So typically, you would set your port scan up at the end of the day once you’re done commissioning and leave it run overnight. You come in the morning and find out that it failed at 80 percent well, that’s really frustrating. You can’t necessarily do the scans during the day while commissioning is taking place. So, you try again the next night, let it run overnight, and hope that it works. I’ve seen lots of frustration with completing port scans. They can take a lot of time and do need some dedicated focus to get completed.

Once your port scans are complete, what you need to do is perform a business justification for any open ports. If no business justification can be determined, ports must be closed. There must be a valid reason why a port is open, such as a service in your system that’s actually using that open port. But if there’s no business justification for that, it must be closed. However, this isn’t all that straightforward because often you’re dealing with vendor equipment, and you don’t necessarily know the internal workings of their equipment. You have to contact the vendor, and find out if there is a business justification for that particular open port. The vendor doesn’t necessarily know, or maybe their equipment doesn’t have that capability to close that port. It can be a bit of a back and forth to figure out a business justification. Either way, if it’s determined that it’s a hardware limitation of why the port cannot be closed, then that must be documented as part of your business justification for open ports. Even if you have an unconnected port, you may have a CAT6 CAT5 connection, and there’s no LAN cable connected, it’s just an open port, that port still needs to be filled with a physical plug, and a picture taken to show that the ports are physically closed with some sort of sticker that would indicate if there was any tampering with the system. This would prevent anyone from locally accessing the system.

Some more tasks are documenting your firewall rules. Each firewall is configured as part of your ESP – electronic security parameters, and the configuration needs to be documented and confirmed to be correct across the entire ESP.

Another large task is updating drivers, software and firmware. All security patches need to be applied to ensure the drivers and software are most current, and there must be a process to track any future released updates. When the equipment arrives at site, it will arrive with a certain version of software and drivers and firmware. And that’s all fine, you’ll start commissioning the equipment with those version. But during your commissioning period, it’s very likely that some of those software systems have patches that are issued, and you need to be able to track all these patches so that the system is currnet at in-service date – you need to know that everything’s updated and current and there’s no vulnerabilities in the system. This needs to happen at a certain frequency after in-service date.

During commissioning, there’s a grace period, and you may choose to proceed through commissioning with a stable version of software and only update it at the end. This can potentially introduce issues if you update all your patches right before the in-service date. You need to prove that the updates don’t upset the system. Some re-testing needs to be done, it can get a bit complicated, and a bit busy at the end of the project to confirm that everything’s updated, and that everything still functions per the contract. Either way, the system must be confirmed to be current at the in-service date to ensure you are NERC CIP compliant.

NERC CIP Evidence

The other large task is gathering all the NERC CIP evidence. This would consist of taking screenshots of your systems to confirm or demonstrate that a particular software version or firmware version is installed on a PC system. You can take screenshots of your security patches that you’ve installed or your virus detection software that’s installed. On non-PC based system, you may have an LED screen on a pump control panel that you take a picture of, and it would show the particular firmware version that’s being run on that pump control panel.

All of your open port justifications form a part of your NERC CIP evidence, as well as the results of your port scans. This can generate mountains of documentation. Some of it’s required to be submitted to regulators at the in-service date while other information needs to be held on to and be available for when a future audit is done by regulators at a future date. If you need to be gathering evidence because at a later date, you won’t be able to because the systems are in-service, you can’t shut them off, or you can’t access the systems due to electrical clearances, you will need to have that evidence in hand so that you can produce it for regulators at a later date.

Approach to Evidence Gathering

There’s a couple different approaches to gathering evidence. Regardless of how you choose to gather the evidence, all NERC CIP tasks need to be completed without exception. You need to do all your port scans, you need to have all your business justifications in place, and all the NERC CIP tasks need to be completed. When it comes to evidence, sometimes there’s a huge number of devices on your project. You may have a thousand devices or more on a project. For example, my last project had over 1200 in-scope NERC CIP assets all requiring compliance. It was a huge task, but what we did was some of those devices are identical. You may have 20 interface cards, the same 20 pumps, and by similarity you can show that the evidence that applies to one applies to all of those 20 devices. So we identified one of those devices as a typical device and we gathered all the evidence for that device, and then by similarity showed that the evidence for this device is the same for all 20 of those devices. That was a huge time saver. Otherwise, I don’t know if we would have made it on time to be NERC CIP compliant. So it can be a large effort to gather all this evidence, but this method to gather evidence for a typical device did help us to achieve our milestone.

The other aspect of NERC CIP is password management. When your equipment arrives at site, it may have all the factory default passwords installed. Username is admin, password is admin, and that’s fine for commissioning because it’s actually easier for everyone to log into devices for testing.

There’s going to be lots of people accessing the equipment, and default passwords are probably just fine for commissioning. However, passwords need to be changed just prior to in-service date. There would have been a whole bunch of people accessing all the systems during commissioning, and that needs to be closed before going into a service. And then from the in-service date onwards, only individuals that are authorized to obtain the password and access the device will be issued the password. Often devices only support a single user interface. unlike a PC. You could have multiple users that are logging into a PC, but for a pump control panel, it would only have a certain version of software, and maybe only one user access interface for that device. You definitely need to change passwords so that only authorized people can access that particular piece of equipment.

Another aspect to manage during commissioning and after commissioning is who has access to the CIP data. NERC CIP data during and after the project needs to be protected. All the NERC CIP evidence that is gathered needs to have controlled access. This is not so bad for evidence that is being gathered, because as you’re gathering the evidence, you can be placing it in a controlled location. But where it gets a bit more complicated is that some of the project documents will need to have access control. Any drawing or document that contains an IP address is deemed a NERC CIP asset. So what do you – do you go through all your drawing packages and determine that one in ten or one in twenty drawings has an IP address? Those drawings definitely need to be controlled, but do you try and filter out just that subset of drawings and say they’re related to NERC CIP and need access management? That can get quite tricky too. Likely what you do is you say all of the project data needs to be NERC CIP controlled so that you don’t have to try and filter out which drawings are NERC CIP and which ones are not, and then you need to establish a data repository and control who has access to all of the project data. This can start quite early in the project to ensure that you are NERC CIP compliant at in-service date.

Access to the project data will require a PRA, a personal risk assessment, to determine if access can be granted. And only once the PRA is returned as all clear can access to the project documentation be granted. With online data systems that are accessed from all over the world, this can be quite challenging to implement and maintain. For example, on our last project we had individuals from Canada, United States, Germany, Spain and all over the world, and each required a PRA before being granted access to the data systems to access the project documents. It was a huge undertaking but had to be done to be NERC CIP compliant.

NERC CIP Asset Passwords

Of course, all of your NERC CIP passwords need to be managed to control who has access to devices and only passwords given out to authorized individuals. You need a process to be able to manage who can request access and is responsible for granting access, and administering the passwords.

So as you can see, there’s a lot of effort required for NERC CIP compliance. How do you pull this off while you’re trying to commission a system during commissioning? For more details about commissioning, please learn from here What is Commissioning?  There’s a lot going on and then throwing NERC CIP on top of that, it can get quite complex. And how do you accomplish this during on-site commissioning? There are two milestones to consider to determine when in scope assets need to be NERC CIP compliant. The first milestone would be when the devices are first connected to the bulk electric system or first connected to other already in scope NERC CIP assets – that’s the first milestone, and this would be at the start of commissioning. The second milestone to consider is when the assets are placed into commercial service at the in-service date. This first milestone, and the second milestone determine the start and the end of commissioning in order to complete NERC CIP tasks and gather evidence. Often, devices need to be powered and possibly connected to in-service NERC CIP assets. So you get in a bit of a chicken and egg situation where you need to gather the evidence, but you can’t necessarily gather the evidence until it’s connected to the NERC CIP assets. But you need to have the evidence to confirm that it’s compliant before connecting it to the NERC CIP assets. Thankfully there is a bit of a grace period during commissioning where you’ll be completing some of the tasks and gathering evidence to confirm in-service compliance at the in-service date.

However, you still want to perform as many of the tasks and gather as much evidence as possible in advance of when you’re first making these connections to in-scope NERC CIP assets. The final tasks will require network connections to be established. For example, if you’re doing port scans, they may require the network to be connected in order to scan the entire network ring for open ports, and thankfully during that grace period, you can complete some of the last remaining tasks for NERC CIP compliance. Without exception though, all NERC CIP tasks need to be completed and all evidence gathered before the assets are placed into commercial service. The system will not be deemed NERC CIP compliant if all tasks are not complete and evidence is not gathered at the in-service date. If you did do this, this would be your first PV, your first potential violation, to self-report to regulators. If this critical cyber security milestone is not achieved, essentially the situation you’ll be in is your in-service date will need to be delayed until all NERC CIP tasks are complete and all evidence has been gathered. It will not go over very well with the owner of the project if you have to tell them that their in-service date is delayed due to NERC CIP requirements. It’s a huge push at the end of the project to complete all the tasks and all the evidence. But it must be completed in order to complete the project and hand over to the owner.

NERC CIP Ongoing Compliance

During the grace period during commissioning, physical security parameters need to be established and access control is in place to ensure that only authorized personnel are on site who could potentially access NERC CIP assets. This can definitely be a huge undertaking, and does need a dedicated team to focus on completing the tasks and gathering the required evidence. Once you meet this huge milestone of compliance, then ongoing NERC CIP compliance transitions from the project team completing all of this to the operation team to continue with ongoing NERC CIP compliance. But the effort doesn’t stop, there’s still a huge task to maintain your NERC compliance that you’ve achieved. The largest task to maintain NERC CIP compliance is application of software patches. At the in-service date, everything’s current, but as time goes on, firmware and software updates are regularly issued by the manufacturer of the equipment. Each of these sources of updates need to be monitored to determine if new software patches are issued.

When new software patches are issued, the NERC CIP compliance team needs to assess each software patch. If it’s determined to not cause issues within the system, it must be applied within 30 days of being issued. Any software patches that are deferred to annual maintenance or not applied need a business justification of why it is not being applied within the 30 days.

This assessment of software patches can be quite difficult. You know how often Microsoft issues a patch for windows? Every time that one of those software patches is issued, it needs to be assessed and determined if it doesn’t cause issues within the system, if it doesn’t open up new vulnerabilities within the system, and can safely be applied to the NERC CIP assets.

But how is anybody going to know what exactly a Microsoft software patch is doing unless you work at Microsoft? You have no way of knowing and assessing exactly how that patch is going to behave within your system, and assessment of these patches can get quite complicated.

Say there was a new version of firmware that was updated for a particular pump controller, you would have no way of knowing or assessing whether that firmware upgrade is going to cause issues with your system, or open up new vulnerabilities, or if it is in fact NERC CIP compliant. Regardless, this assessment needs to take place in 30 days.

So what do you do? Sometimes, a replica of the control systems need to be in place in order to apply the updates to the replica, perform all your port scans, gather the evidence and confirm that it doesn’t cause any issues with the system and is in fact NERC compliant. Once you’ve done that on the replica, then you can apply the software patch on the live system.

This all needs to be done within 30 days of the patch being issued in order for it to be applied, and in order to remain NERC CIP compliant. So this does require a dedicated team, and is a huge effort to keep up with NERC CIP compliance. Also, hardware may fail and need to be replaced during the life of the system. The NERC CIP team will need to ensure that old assets are destroyed properly, that new assets are configured correctly, the tasks are completed, and evidence is gathered to maintain your NERC CIP compliance. As people come and go from working with the NERC CIP systems, PRAs will need to be completed access requests reviewed and approved and card lock systems synced to ensure the correct people have access to the appropriate systems. So I would like to stress that NERC CIP compliance can be a costly endeavour to obtain and maintain, but may be required depending on the impact rating of the equipment being developed.

A few tips that I can give is you do need to ensure that your NERC CIP requirements are clearly defined at the beginning of your project, and the appropriate budgets are established to cover the cost. To achieve compliance, it is very difficult approaching the end of the project, trying to scramble to obtain compliance with no budget in the project estimate. As well, a helpful tip for operators at the in-service date and beyond, operating budgets should be reviewed to establish funds for personnel to maintain NERC CIP compliance after project completion.

So, that’s a little bit of an overview of how cyber compliance works in North America. I’m sure it’s different elsewhere in the world. NERC is a North American, standard but there may be other standards elsewhere in the world, and it’s likely very similar to ensure cyber compliance. But as our systems become more and more complicated in the world, cyber security definitely becomes more and more of an issue, and more and more of a concern. You do hear stories in the news of systems being jeopardized and power being disturbed, and it is a topic of more and more concern as things go forward.

What really drove the NERC CIP requirements in North America was the the large power outrage on the Eastern seaboard of the US. There was something like 50 million people that were out of power for a few days as there was a cascading power event that took out all those power systems, and that’s when regulators really started to ensure reliability of the systems – NERC CIP became even more of a priority since then.

If you’d like to learn more about the commissioning and startup process, please join our free three-day mini course. The course is free and flexible to take any time online. It gives you a good start so that you can understand or have an early understanding of the commissioning and startup process. 

Question and Answer Session

What is NERC CIP? 

NERC CIP is the cyber security list of standards that govern electrical systems in North America, and define the tasks, and evidence required to in order to maintain NERC compliance. So, really it’s about securing your electrical system so that it’s not vulnerable to other people logging in and hacking into the system, manipulating switch yards, or controlling some of the devices within your facility to disrupt power systems to your customers, and to other utility connections as well. It’s really the electronic cyber security rules that you need to meet in order to be NERC CIP compliant, and sometimes in the case where you’re interconnected with other utilities. It’s a requirement that if you’re interconnected with them you must be NERC CIP compliant, because they don’t want your facilities to cascade and start taking out their facilities. So in order to have some inter-transfer agreements between utilities, you do have to demonstrate NERC CIP compliance. If you’re not NERC CIP compliant, then they won’t necessarily accept power from you or allow sales to other neighbouring jurisdictions, and you do have to maintain your CIP compliance for that reason.

 

In a normal operation, how if we remote connect to modify a program from a personal computer, is it allowed? As we know, usually engineering workstation is a common tool for modifying program. 

This is always a good question during commissioning. It’s often that the vendor might have the ability to remote into the system, and that definitely can be very helpful during commissioning, if not all the resources are on site. If someone that’s off-site is able to log into the system, make any changes, and that can certainly be helpful. But at the in-service date, all those connections need to be shut off and more likely physically disconnected, so that if there’s a device that allows that remote login connection, that device would physically be removed. so that there is no ability to remote log into the system. That is a vulnerability that can exist, and it must be closed at the in-service date. Now there is very complex methods to put in place to allow external connections to exist through VPN connections, through other more complex connections, so that outside people with the proper authorizations can access the system, but typically that’s not the approach. Typically, those types of connections would be shut down and disabled. When it comes to something like your central control facility that’s monitoring the systems, an RTU is installed, a remote terminal unit. The RTU is the break in connection between the external network and the internal network, and any connection between those two is strictly either rs-232 serial or discrete logic so that there is no routable connection from the outside world to the inside world. You can only communicate through discrete logic or through RS- 232 for that very reason so that nobody external can get into your system, get any further, and start messing with any of the devices in the system. Worst cases, they could toggle your discrete logic high and low which would still cause issues, but they can’t necessarily get into the system, and start changing passwords, and locking others out of the system. So those external connections are a big concern, and definitely need to be shut off at in-service date or with more complex methods put in place to manage external connections.

 

Why are we (Cx team) always struggling with the QC department? 

The quality management system that’s set up during construction is really the first piece that’s going to set up commissioning for success on-site, learn how to manage your team to set your projects up to be a success, this article will help 5 Dysfunctions of a Project/Commissioning Team. If the construction team has a very strong quality management system, then commissioning is going to go a lot better. If the construction team is just winging it, and making things up as they go, and kind of inspecting things, or not, which is maybe more of my experience, then yes you’re going to have issues with the quality of the devices that you’re receiving from the construction group and the QC department. The best thing I can suggest is to very clearly define the mechanical completion handover between those two group. The earlier you can do this, and the more detailed you can make it, the easier it’s going to be for everybody to understand what are the expectations to meet for that critical handover stage. If mechanical completion is clearly defined, the scope of what’s being handed over, the scope of the quality inspections, and exactly what needs to be completed, then when that mechanical completion is signed, everybody understands that’s the transfer of care custody and control from one group to the other. And both groups know exactly what’s required of them, and exactly what’s being handed over from one group to the other. It does seem to be a challenge. The handover stage is probably the most difficult thing that exists in commissioning, to establish a smooth handover from construction to commissioning, and I always see it being a struggle as well. And even trying to define the mechanical completion handover well in advance, everybody still wants to change things and do something different. Then schedules get pushed, and everything’s tight. Everybody wants to try and cut corners. It’s definitely a challenge, but that’s the best thing that I found is to try and manage expectations in advance with clearly defined mechanical completions.

​

In your area, who is responsible for maintaining NERC CIP? 

In our last project, that was largely the commissioning team’s responsibility to obtain NERC CIP compliance. We had a dedicated team of four or five people that were part of our commissioning team, and were completing all these tasks, and gathering all the evidence to meet that critical milestone. To say okay, now we are next compliant at in-service date. Ideally, that is turned over to the operating team and they’ve got resources to maintain the new assets as well as all the existing assets in the system to maintain NERC CIP compliance. Although, NERC CIP is such a huge task that it’s not all that fair on day one of operation to say, here you go operations it’s yours now, run with it. So, what we did was we supported them with a soft hand over having our dedicated NERC CIP team of five people continue on after in-service date, and slowly transition out of that role as the operating team became more familiar with the systems, and could integrate the systems into their existing NERC CIP compliance tasks. And that worked much better than trying to just dump this on operations, and say here you go good luck. That wouldn’t have worked very well. But in our case it did work really well. Over about a six-month period of time from in-service date afterwards for six months, we helped them along, and got them up and running on the tasks that were completed, and so that they could maintain compliance going forward. On the project I last worked on, they had a NERC CIP compliance team that was focusing on these assets as well as every other asset in the system to maintain compliance. It just took some time for them to incorporate these new assets into their existing processes. But that’s a very good question, because it does require some dedicated resources after in-service date to maintain NERC CIP compliance.

 

What are the latest trends in commissioning cyber security?

Let’s maybe discuss that from the aspect of how are these activities overlapped during the commissioning period. There’s a lot going on with commissioning, and this of itself is a big enough task. Throwing NERC CIP compliance on top of that is even more effort. It’s a huge effort to do these tasks in parallel, and I would say that cyber security is a fairly new topic. A lot of companies are still learning exactly what is required for NERC CIP compliance. Particularly, for international companies that work all over the world, NERC compliance would apply in North America, but there’d be different standards elsewhere in the world. And it’s definitely a learning experience for everyone to figure out what this all means, how to obtain compliance, and how to maintain compliance. So from a trend standpoint, it’s more of a learning approach. You may have to work very closely with the contractor to help everyone understand what are these NERC CIP requirements, what are we trying to achieve , and how are we going to obtain NERC CIP compliance.

The biggest hesitation would be underestimating the level of effort required to do this. The contractors may feel that the commissioning guys can do this, they can just do it in their spare time, but I’ve never seen any commissioning guys with any spare time. You definitely need to have those discussions and help everyone realize that this is a big task. And it is going to require some dedicated resources, not just telling your commissioning guys – go figure this out. It’s definitely a bit of a process to coordinate those two activities for commissioning and NERC CIP compliance. When you’re working on the systems commissioning, then you’re making changes as you should be during commissioning to get the system fine-tuned. But those changes impact the evidence you’ve gathered for NERC CIP. You do have to schedule it pretty closely, that if you are going to overlap these activities, you need to determine that a particular subsystem is complete, and the NERC CIP team can then move in to complete their tasks and gather their evidence. If the commissioning team needs to go back and make some changes to the system, then that’s going to require the NERC CIP team to go back and redo some of their efforts. Alternatively, you push all the NERC CIP tasks right to the very end of the project when commissioning is complete, and it’s very rushed to gather all this evidence, or you allow the time in the project schedule to push out the service date to gather all this evidence. It’s definitely a bit of a balancing act. I would say, those are kind of the local trends on NERC CIP compliance. Just the sheer magnitude of trying to obtain compliance amongst all the other commissioning activities that are taking place at a very busy period of the project when we’re trying to get a system in-service.

How do we migrate from NERC CIP non- compliance to compliance? Can you give us some ideas?

This is quite common. We went through this at the utility I was working at several years ago, because none of the systems that were installed 20 to 30 years ago were never NERC CIP compliant – hat just wasn’t a consideration then. But as NERC CIP compliance becomes more common, and there is a requirement to interconnect with neighbouring utilities, then at some point you do need to deem that the systems are NERC compliant. That is a big task in its own. Think of a large utility network and none of it is NERC compliant. It’s a very similar process to putting a new project in-service. You take each portion of the system, and just start going through and completing the tasks and gathering the evidence. Some of the older systems are actually easier to obtain NERC CIP compliance if you’ve got non-digital systems, more mechanical relay systems, and those are super simple, because they don’t have any externally routable connections or addressable connections. But invariably, a lot of your PLC control systems that weren’t compliant, you do need to go through all of those, and perform all the NERC CIP tasks, gather all the evidence, and demonstrate that it is NERC CIP compliant. So that when the system was first installed, maybe 20 years ago, you wouldn’t have done all those port scans, because CIP compliance wasn’t a concern. You’re going to perform all those port scans, and do all your open port justifications to confirm which ports are open and which ports are closed.

Now you will have to be somewhat careful in doing this, because if you’ve determined that you want to close some ports on a system that’s been functioning without issue for 20 years, you want to make sure you’re not damaging anything or going to break any of the connections by closing that open port. Likely, when you’re doing those business justifications of open ports you’re probably going to have a really good reason to close that port as equally as having a very good reason why that port is open. But it’s very similar to putting a project in-service, you just have to go through the effort, and perform all the tasks, all the port scans, gather all the evidence, and confirm that your system is NERC CIP compliant or not. Any gaps in compliance, maybe the systems can be modified with additional settings or closing some of the ports if the piece of equipment just cannot be deemed to be NERC CIP compliant, then maybe there are a few odd systems that need to be replaced or upgraded in order to meet NERC CIP comliance.

From our experience, the equipment all functioned properly, and there were very few pieces of equipment that had to be replaced that couldn’t be proved to be NERC CIP compliant through port scans, and the open port justifications. So when you use the term migrate, I wouldn’t necessarily say that you’re replacing a bunch of the equipment, because that wouldn’t be very desirable. But it’s more the effort to go through, and perform all the tasks, and gather all the evidence to show that your already existing systems are in fact compliant with maybe a few hardware pieces that you may have to change. It’s more of a focus on making sure that all your software patches and your firmware in your systems are all up to date. And that could definitely be a challenge too, because maybe the vendor from 20 years ago is no longer around, so how do you have a contact to confirm the internals of that system? How do you confirm that you do have the most current version of firmware installed? It can be challenging from that aspect, and maybe you just can’t for that particular piece of equipment – the vendor is no longer around. That piece of equipment maybe does have to be replaced, because there is no other way to determine if it is in fact NERC CIP compliant.

It’s definitely a big task. If you look at one project that is maybe one substation, that’s a huge enough task to get that system compliant. If you’ve got a utility that’s covers a huge area, and it’s got transmission facilities, generation, and substations all over the place, then this can be a huge effort for the NERC CIP team. Maybe to focus on one substation, go through all the tasks and evidence there, then move on to the next one. This can take years to go through and demonstrate NERC CIP compliance for a large utility, but definitely needs to be done. In our case, it did take that long – it was a two or three-year effort to get through and demonstrate NERC CIP compliance, as well as then maintaining that compliance as years go forward to show that the entire system is NERC CIP compliant. Maybe one thing that would help is that when you go through and classify all your assets as low, medium and high impact, it would depend on how many of your assets are classified as medium plus or high impact. You maybe only have one or two high impact facilities, most of them are going to be medium, and medium plus. A lot of them are also going to be low impact, and for low impact, then there’s minimal or less requirements for tasks and evidence to complete for NERC CIP compliance. It would just depend maybe on the configuration of your system, and how detailed you need to get into the various aspects based on their assets, based on their impact reading.