Advanced SCADA redundancy helps a major oil and gas producer adopt a unified platform
Canadian Natural Resources Limited (CNRL) has grown to be one of the largest independent oil and gas producers in the world, in part through strategic purchasing of crude oil and natural gas assets. Typically, each new system acquired already has a Supervisory Control and Data Acquisition (SCADA) software application for real-time monitoring and control and long-term historical data collection. They currently own and operate over 200 such systems across western Canada. To simplify operations, licensing, support, and training, the company selected VTScada software by Trihedral as their standardized SCADA solutions across all their sites. In addition to being able to communicate with their wide range of field controllers, VTScada provides unique redundancy and data synchronization options that suit their widely distributed architecture. Sam Lau, a SCADA specialist with CNRL, describes below how these features help them to keep their applications online and their historical data safe.
Selecting SCADA
“We were looking for software to replace all our aging SCADA systems and evaluated quite a few vendors and types of software. When we first looked at VTScada, we saw features which we hadn’t seen in all the other software that we had looked at. We were initially amazed by how well the version control works.” This native component provides an encrypted list of configuration changes across whole applications. Authorized users can use this to trace who made changes and why as well as roll back any changes which are causing trouble.
VTScada could also help them to standardize and replicate how they represent their sites and equipment. “Being able to quickly build up reusable tag types and templates for assets and then go back to modify entire classes which propagate through one or more applications is really very valuable. We also liked that VTScada supports various application layers, so we could create our own OEM layer for integrators. These layers allow us to define the sandbox in which our integrators can use widgets, fonts, and the like. Because we control the OEM layer, our VTScada applications are consistent across the board.”
Road to Redundancy
“At CNRL, we have implemented a few different types of redundancy architecture using VTScada. We have the traditional SCADA host redundancy where we have the primary and backup servers right on site using the two physical machines.”
VTScada
software uses an integrated architecture that includes all core SCADA
components in a single license and install. This means that an application can
include any number of redundant servers configured to instantly fail over to
one another in a preconfigured order. This is the standard approach to server redundancy;
however, this integrated design also opens the door for other redundancy models
that were suited to the needs of CNRL.
Bi-directional Synchronization is Key
Timestamps will be matched for each data-point to the millisecond. Should the primary database server fail, associated workstations and Internet clients switch to the next designated database. When it is restored, historical data automatically synchronizes across a local or wide area network at up to 160,000 values per second. This speed is automatically throttled such that real-time communications between SCADA servers is not significantly deteriorated. Any data on any historian that is missing on another will be propagated automatically regardless of how long it has been since the databases have communicated.
Centralized Application Backup
“Another powerful feature is the ability to run multiple VTScada applications on the same server. That was something we’d never seen before. Traditionally, you would have one server which could host one instance of that SCADA application but VTScada allows multiple ones and that opened a whole new world of redundancy for us.”
“Previously, the only way to have redundancy was to have multiple physical servers. So, if you wanted a primary and backup server, you had to create a dual server pair for each installation. In the oil and gas market, we have a lot of servers out in the field, off in the middle of nowhere, so the hardware and maintenance costs can be quite high.”
“The ability to run multiple applications on one server allowed us to use a single backup server that supported multiple applications covering a whole region. That one machine would serve as backup for many posts. We now use this architecture for most of our systems.”
“For our large, critical systems we do have instances where we have a couple of physical servers running a primary and a backup, but in general, we have just one large central backup in our data center and the primary host is wherever it happens to be located out in the field. In our case, the central backup is just a virtual machine, selected as it is scalable and provides redundancy where we traditionally would not have had anything due to the cost of the additional servers.”
Local Monitoring with Local Data
“We also have another type of redundancy that is very innovative; one we hadn’t seen before VTScada. On some of our applications we have more than one important facility in a single app. With this architecture, it’s entirely possible to have issues such as partial network outage where facilities become disconnected. When that occurs, we might not be able to connect to the primary facility. Historically, that second or third facility would be blind while they could not connect to the primary host. Fortunately, with VTScada, the application historian can be distributed such that it doesn’t require a connection to the primary historian. We can configure the local facility historian to look solely at the I/O at that facility. Therefore, the operators there can still run their plant. I really like that idea.”
This approach also allows the company to save money by using smaller VTScada I/O licenses at some locations. “Our VTScada application might have, say, a hundred thousand I/O, multiple facilities, and field devices, but we have backups at some of the other important facilities that use just a five thousand I/O license on premise. Beyond the local facility data retrieval, new data at the facility will be logged to the local historian and then synced to the primary when it becomes available. The fact we can scale down a large application into smaller pieces, means it’s very affordable for the licensing and the PC hardware required to achieve this level of redundancy.”
Long-Term Historical Data Management
“Currently, we don’t spend all that much time thinking about how to manage redundancy of our historical data because it’s looked after by the software and it just works. Our primary server stores a copy of the historian and that gets copied to our backup server in Calgary. In the next phase of our system, as we grow the number of VTScada applications, we are evaluating implementing a historian farm in our data center.”
“Because VTScada allows us to specify a server list for several of its services (i.e., historian, drivers, thin client server, etc.) you can have your primary application run on one server and run your historian on another server. We can split up our historians into larger virtual machines with a lot of disk space. This benefits us because we do want to store our historical data forever; there’s no reason not to do it these days. This contrasts with our previous systems from a decade ago which used to keep historical data for one to three months and then that would be the end of the data.”
Putting Historical Data to Use
“One of our operator’s favorite features is VTScada’s trending. They get a lot out of seeing the oil well data trended over time. As of today, our engineers are just discovering the power of the trending that’s available in VTScada; partly because all the data is there for them and they aren’t expecting this.”
“Our old SCADA systems would only hold a few months worth of data and we would only get a snapshot of the data, for example, a daily average. Now, a system engineer can go back and look at what a piece of equipment was doing a couple of years ago when he first turned it on. They can compare that data to the same or other similar pieces of equipment which may be experiencing symptoms.”
The Future
“We see a big future in analyzing the data we now have. Analytics has exploded in popularity and we’re at the point now where companies like us have so much data that they don’t know what to do with it or how to process it. That’s where data analytics come in. We’ve been moving forward with some initiatives on using this data more efficiently. Instead of looking at all the data that comes in, we are developing reports that look at the exceptions, the stuff that’s not normal.”
Read the Whitepaper
Learn more about this innovative approach to long-term SCADA scalability by reading the whitepaper produced by Sam Lau, Kurtis Jackson, Leon Bakaas, and Jamie Walker. https://www.trihedral.com/cnrl/
