Top 3 Reasons for a Manual Process Operation Plan

by Tim Ondercin

Recently the National Security Agency and the Department of Homeland Security issued a joint alert warning that foreign hackers are targeting critical infrastructure, including water and wastewater systems. The agencies recommended that operators take immediate action to secure their systems. We spoke to Tim Ondercin, an experienced field technician from Energenecs about implementing a manual process operation plan for security and other issues.

According to Ondercin, who has over thirty-three years of experience in the field, security is an important issue, though his top three reasons for having a manual process operation plan are power outages, loss of PLC’s and HMI (Human-Machine Interface) computer or touch screen, and loss of communication links.

1. Power Outages

For some, power outages do not happen very often, it might be every five or ten years, but if you are in a rural community and you have an undependable power grid, it might be as often as once a year or more.

One of our customers lost power twice in the last four years, for 18-20 hours each time. They provide water for a large city and do not have a generator, so they had to bring in a portable generator at three in the morning, then find an electrician to hook it up.

The power company promised them they would never lose power because they had two feeds. Unfortunately, both of the feeds were running side by side and when the heavy winds came through, it knocked a tree into both the feeds.

We recommend installing a generator at your plant if you can afford it, plus a portable generator for all remote equipment, at least. A water tower can be run by a small big box store generator, as they don’t use much power. In a perfect world all lift stations should have a permanent generator. Some utilities keep a portable generator and run it around from lift station to lift station, but the problem is if they have five or ten lift stations out of power and only one generator, it can make for a long day!

At least once per year, take the portable generators to every station you use them on and test the system. This testing burns the older fuel and exercises the breakers. Test with all your pumps running, if you are using the generator during a power failure, chances are that multiple pumps will be needed.

2. Failure of Computer that Provides your HMI

Computer components fail, mostly hard drives and power supplies, or lucky you, if it is one of the other thousands of components. For this reason, We recommend having a second computer on standby. If your computer fails, the SCADA already installed on a standby computer can be substituted for the defective one, in most cases, without breaking the licensing agreement. In the event of a computer failure, you can just move the cables to the standby computer and turn it on. This is less expensive and much faster than buying a new machine and having to install the OS and custom software like Wonderware.

Backups of your SCADA computer are critical, we recommend purchasing an external USB drive and using Windows Backup, which is free on every machine from Windows 7 and up. Backup your computer and keep a disk image, plus if you keep DNR files, back them up incrementally so you don’t lose all that information. Windows backup can do the daily or weekly backups also. Creating a backup only gets you partially there – make sure you check those back-ups on a regular basis and can both see and access the data!

Another thing I see is people never cleaning their computers! Dust will build up and cause overheating, due to the plugging of vents.

3. Loss of Communication Links

You could lose your communications to your plant, or remote PLCs (Programmable Logic Controllers), when your fiber, copper, a switch, radio system or the internet goes down. Then you could lose that portion of your Human-Machine Interface (HMI) like Wonderware or a touch screen, that the communication failure effects. Loss of communications can also be from PLC failure, and other malfunctions of the PLC control panel.

Tips for Creating a Written Plan

Create a checklist for continued manual process operations if the control system or power becomes unavailable. You will have to manually provide all the critical tasks the programmable logic controllers (PLCs) were performing. Make sure you know how each critical process works in hand:

We recommend implementing an annual practice of the plan, or after a change of staff.

Water systems

Run your water system, well pumps, and chemical feed etc like you don’t have communications or auto control.
Run your well pump on the Generator or engine if so equipped.
Practice backwashing a filter and putting back in service with hand controls.
Run Low and High Lift Pumps on Hand.
Check your gauge reading vs SCADA during your test. Have a book handy with gauge vs fill height of your storage units. This might be a good place to have your emergency contact list. A copy in each truck is a good idea, as phones tend to be busy during difficult times.
It is also a good idea to have one employee to be assigned as spokesman for your utility.

Wastewater and Lift Station Operations

For mechanical plants, run pumps in hand and check that any provided float backup system works. The backup system should work with no PLC control.
Run the best VFD speed for RAS, and WAS if not in timed mode.
Run all lift stations on generators (full load), test backup float system.
Have an emergency contact list that includes day and night phone numbers for vacuum truck operators, your power company, local electricians and your SCADA provider.

Contact Tim Ondercin for More Info