A case study highlighting the importance of application version control for water and wastewater SCADA systems.
The falling cost of automation hardware is changing not only the scale and complexity of SCADA systems, but also the way in which users manage them. Smaller systems now support advanced monitoring and control features once reserved for large applications which, in turn, continue to grow along with the size of the teams who configure them. Systems of all sizes now support remote hardware capable of returning huge volumes of data. This results in a lot of configuration by a lot of people, which increases the likelihood of unexpected effects and unwanted consequences.
The Courage to Configure in Real-time
In recent years, Application Version Control (AVC) has become a common tool for managing this complexity. When implemented correctly, AVC should allow you to instantly roll your application back to a known good state, trace problems back to individual changes by particular users, and reduce the need to restart the application. The following is an example of how AVC has reduced downtime and increased workflow for mission-critical SCADA systems in the water and wastewater industry.
A Water and Wastewater Case Study
As cash-strapped water and wastewater utilities work to reduce operating costs while maintaining a high standard of safety and service, they are relying more and more on the recorded process data from their SCADA systems. This data drives the reports and trends that highlight inefficiencies and identify issues before they interrupt service. For example, discovering a partially clogged pump saves electricity, extends the life of the pump, and potentially avoids a costly spill. Protecting this uninterrupted application history means that it is no longer feasible to shut down to perform configuration. SCADA products, such as VTScada from Trihedral, allow authorized users to create tags, add remote sites and edit pages all while continuing to monitor and control the system. Users can make local changes on a single development server, test those changes and then disseminate them to all networked servers.
While this eliminates the need to interrupt data logging, the thought of editing ‘on-the-fly’ can be a cause of stress for some users who fear that they might make a change that has an undesirable result. That is where AVC comes in. With integrated version control tools, you can instantly roll back to the last known good version of the application before significant process data is lost.
That is exactly what happened recently at a major Florida water utility. A member of their SCADA team had made a breakthrough, or so he thought. The system had recently been upgraded to support a hierarchical tag structure. This parent/child architecture allowed them to group all the elements found in a typical site under a single parent tag such as sensors, pumps and polling drivers. A new site could be added in seconds simply by copying this parent.
This user (we will call him Todd) needed to add a new value to this parent template. Once he did this, he then figured out how to propagate this change to all the existing sites. Todd was pleased… but not for long. Almost instantly, alarms notified him that polling to all remote sites had ceased. No data was flowing into the historical database. It turned out that in the process of augmenting this parent tag, he had inadvertently removed the PLC device driver from every site. However, he didn’t know that at the time. All he knew was that he had made a very large mistake. Todd opened the Application Version Control panel, right-clicked on the version he had just saved, and rolled back those changes. Almost instantly, polling resumed. Todd’s breathing also resumed a short while later.
In this case, the problem was immediately obvious. However, in today’s complex SCADA systems, a negative effect of configuration may take time to show itself. Version control can also help you make changes in a highly controlled manner on a single live computer or subset of networked computers where structured testing can take place. In effect, you can create your own isolated testing environment. As each change within a specific version is successfully tested, it can be pushed out to all the computers in the system.
