How server architecture can automatically backup your whole system.
Modern SCADA software systems carry a heavy burden. They help to ensure uptime, contain a wealth of recorded history, and represent a significant investment of development time. Despite this, backing up applications and their data remains a comparatively low priority. Standard backup strategies have limitations. Offline backups leave your system blind while online backups can interfere with your process. Both leave gaps in your history when restoring backups and both require specialized training. Advances in networking provide a more elegant solution. If your application includes one or more networked servers, you can build automatic real time backups for history, configuration, alarms, and I/O tags right into your server architecture. If those servers are in separate locations then you also have an offsite disaster backup of your whole application with no extra effort. For those without a plan or those who fear the day they will need to put theirs into motion, this article is for you.
Traditional SCADA Back Up Strategies
Manual Offline (Cold) Backups – This process is common for smaller SCADA systems. First, someone manually shuts down the system. Then, they save the whole application (or just the historical and configuration data) to a secure location like a tape device, hard drive, or network folder. When this is complete, they restarts the application. Though relatively simple, this is a time consuming process that usually takes place outside of regular business hours. For this reason, backups can be irregular or dropped entirely.
Time Based Offline (Cold) Backups – At regular appointed times, usually in the middle of the night, the SCADA application shuts down automatically and its data is exported to an SQL-based database format. This database is then recorded and archived as outlined above. Backing up offline ensures that files will not be corrupted if they are read while being updated by the running system. However, depending on the size and age of the application, this process can take anywhere from a couple of minutes to a couple of hours. All the while, operators cannot see their process displays. Alarms cannot be viewed, disseminated or acknowledged. Thin Client remote access is unavailable. Worst of all, process readings collected during this period will be permanently lost. Should the system fail and need to be restored, all process data and configuration changes logged since the last backup will also be lost.
Time Based Online (Hot) Backups – Online backups are useful for mission critical systems where downtime is not an option. In this scenario, the monitoring and control process remains active while the application is being saved. This ensures that alarms are managed and operators are not left in the dark. However, this runs the risk of reading files while they are being written which can affect performance and corrupt the backup database. As with cold backups, restored databases will not include process or configuration data recorded since the last backup.
Change Based Online (Hot) Backups – Rather than updating the whole running application at once, the system backs up each change as it occurs. This eliminates the risk of losing data between backups but, like time-based hot backups, there is a risk of impeded performance and corruption as the backup process effectively “steps on the toes” of the running SCADA system.
Other Issues With Traditional Backups
Long-term Compatibility of Backup Utilities – Though some SCADA software platforms include built-in tools for backing up historical and configuration data, others require third-party utilities. As these components are individually upgraded over time, they can cease to function together resulting in dropped backups or lost data.
Specialized Technical Knowledge – Most backup methodologies require an SQL-based database format. SQL requires specialized knowledge to configure, backup, and restore. This means an investment of time and money for the system integrator or the internal IT department at each point of the process.
A Better Way: Real Time Full System Backup
Simplified SCADA Redundancy – Until recently, the cost of setting up and maintaining redundant servers with automatic failover was prohibitively expensive for most SCADA users. Modern monitoring and control software products not only make it easier to designate hot backups for fallen servers, they also can greatly simplify the process of synchronizing historical databases across networks in real time. SCADA manufacturers are even starting to incorporate configuration management into their products.
In addition to increasing system uptime, these innovations provide an opportunity to build real time full system backups right into your server architecture while simplifying the entire process. The key to this approach is using a software platform where the history, configuration log, tags, and alarms are integrated into the software itself.
Configuration – Set up two or more SCADA servers on a shared network. Then, configure a server list to designate the primary server as well as the order of failover to the other computers. At any given point, only one server should be responsible for polling remote monitoring and control devices. The ease of synchronizing servers will vary from product to product. Once completed, each server should contain an up-to-date copy of the historian and the change log as well as the tag, alarm, and event databases. In effect, each server is a real time backup of the whole application. No third-party utilities or databases required. Activities such as trending, reporting, and alarm analysis typically take place on the local server minimizing network traffic.
Disaster Backup – If one or more synced servers are located at geographically separate locations, then there is effectively a complete off-site backup in case the primary server or the building where it resides are destroyed. Restored or replaced servers should be able to pull a copy of the running application from any surviving server.
Failover Scenario – If the primary should go offline for any reason, the next designated server will take over polling and any other duties associated with the primary such as acting as the Thin Client Server. Configuration changes to displays, settings, and tags will be disseminated to all available computers on the server list. When the primary server resumes its duties, all missed historical and configuration data is bi-directionally backfilled over the network.
Example – Municipality of Colchester, Nova Scotia
The Municipality of Colchester in Nova Scotia, Canada, uses this approach at four of its sewage treatment plants that collectively service approximately 45,000 residents. Originally, each plant had its own autonomous monitoring and control system. They decided to transition to a single SCADA software platform that would allow them to monitor all four plants and make configuration changes from the main office in Colchester (Plant 1). In the process, they also managed to address long standing issues with remote data backup and automatic failover.
Colchester standardized on software that could communicate with all the various remote programmable logic controllers (PLCs) used at each plant. It also features an integrated historian, change list, alarm manager, and tag database.
The Colchester team took this idea a step further. Each plant has its own SCADA application with a local primary server and a synchronized backup at the main office. All four backup applications run concurrently on the same computer.
In addition to providing centralized configuration, monitoring, and alarm management, this approach also provides a complete real time backup of the entire application and its historical data. Should any local server fail, the office backup will take over logging from PLCs at the site. When restored or replaced, the primary is automatically backfilled over the network from the backup.
Never Forget Another Backup
SCADA backup strategies need to be scalable, comprehensive and automatic. By building your backup right into your server architecture, you can ensure that you are capturing every part of your system and leaving no gaps in your historical data.