Uncomfortable questions to help you sleep better.
Scenario 1: It is 3 am and you are fast asleep at home in your bed. At your office across town, the SCADA system server that monitors and controls your critical water and wastewater infrastructure goes offline. Maybe, the hard drive failed or the power supply died. Perhaps it just lost connection to the network. Regardless, all polling, logging, monitoring and alarming have ceased. What happens next?
Is there a night shift to take care of it? Are you notified by text message or automated phone call? Will you find out the next day when you arrive at your desk, or worse, when you pass a spill on your way to work? Is there at least one other computer that can take over? If so, is that failover automatic or does someone have to drag a computer out of the closet, dust it off and plug it in hoping it has at least a recent version of the application?
When was the last time you backed up the historical, alarm, and event histories? What about the configuration log? Does the failed server contain your only copy of this important history? What happens to process data measured while you were starting up the backup computer? Is it gone forever? When the primary is restored, how does the history logged by the backup make it to the primary?
If any of these questions make you feel uneasy, then it might be time to take a long hard look at your system’s architecture. Luckily, there are now many options available to make this process more robust.
Scenario 2: Its 3 am. You are sleeping. The primary SCADA server meets its maker. Instantly, the secondary server in the next office begins polling your remote PLCs and RTUs and logs that data to its own copy of the Historian which was synchronized with the primary right up to the time of failure. The failure of the primary server is recorded in the encrypted events database and triggers a text message to your phone. You lean over, log into the running application, and see that everything is fine and go back to sleep. The next morning, you install VTScada on the brand new primary server and then import the application and all its history from the running server over the network in just a few minutes.
Scenario 3: During the night, the office that houses both your primary and backup servers is flooded and both machines are lost. Fortunately, a few miles away, the third server at the water treatment plant immediately takes over polling and logging with completely up-to-date historical and configuration databases. Again, you receive a text message. Again, your Thin Clients are still functional. Again, you can reestablish your applications and history on the replacement servers in minutes when the office dries out.
Our VTScada software includes many features that make scenarios 2 and 3 simple and stress free. The integrated design allows for any number of distributed synchronized historians and alarm databases. This effectively eliminates the need to run daily or weekly online or offline backups since each server contains an up-to-the-minute copy of the entire application. In fact, the VTScada Multiplexer (MUX) Tag even supports redundant communication networks or I/O devices.
That said, other SCADA platforms do have their own ways of tackling many aspects of server failover. In addition to knowing what your current system is capable of, it is up to you to learn about best practices and new approaches that can offer you and your customers peace of mind at any hour.
Setting up Redundancy and Synchronization in VTScada
https://www.trihedral.com/redundancy-and-automatic-failover
About the Author
Christopher Little, Sales and Marketing, Trihedral – For over ten years Chris has been producing articles, case studies, and videos covering topics related to industrial monitoring and control. His work has been featured in a variety of industry-specific publications.