It’s arrived: Commoditization for industrial process control

For decades, deeply embedded, purpose-built devices based on proprietary hardware and software have dominated process control. Why? Because control systems must run, without failure or interruption, 24/7 and with minimal downtime. The impacts of a failure can range from a sticky floor or ruined machinery to financial loss or, in the worst case, human casualties. Additionally, many factories have difficulty scheduling the downtime needed for updates, and often there is far less time available for updates than what is required. For example, when it came time for the Port of Dubai to upload a small update to its control software in the mid 2000s, it was only able to open a 30-minute window in which to pause operations.

For these reasons, innovation in manufacturing and industrial systems has been slower than in industries like telco, datacenter, and cloud. Fortunately, advancements in modern software development, management, automation, and security have coalesced with industrial requirements to bring new benefits on commodity hardware.

Defining the future of distributed control systems

With the advent of industrial process-control commoditization has come technological advancements that have expanded the boundaries of modern manufacturing–right to the computing edge. Traditionally, administrators had to walk out to a control system–USB stick in hand–and apply an update manually. Today, thanks to the combined work of Intel Corporation, Schneider Electric, and Red Hat, manufacturers can enjoy an edge-ready, software-defined, industrial control system that relieves the burden of manual effort and runs on commodity hardware and a commodity operating system and uses commodity automation techniques.

The three companies have worked hand-in-hand to create a more modern, network-based experience–similar to modern automated, cloud-based deployments–and demonstrate a next-generation smart factory that not only showcases leading-edge advancements but illuminates the way to the next generation of distributed control systems.

Understanding the architecture

The compute portions of the demo are broken into two main concepts: (1) the advanced compute platform (ACP), which is ​​the supervisor of the control workload and DCNs, providing the automation and content control capabilities needed to securely run on an Intel Xeon class processor, and (2) the distributed control nodes (DCNs), which are banks of nodes dedicated to running controls.

The advanced compute platform (ACP)

The ACP is the supervisor of the control workload and DCNs, providing the automation and content control capabilities needed to securely and programmatically deploy workloads to available DCNs, while also providing virtualization and monitoring functionality. Red Hat Device Edge serves as the platform for Red Hat Ansible Automation Platform, providing the automation platform used to deploy, lifecycle, and redeploy the DCNs and their workloads as needed. In the next-generation smart factory demo, the ACP only had purview over one bank of DCNs with one control workload, but this approach would scale to thousands of DCNs without needing additional hardware.

The distributed control nodes (DCNs)

The DCNs are low-power, Intel-based industrial systems designed for workloads of mixed criticality, which include both real-time and non-real-time workloads. The DCNs are actually two parts: (1) a compute module that contains the processor, RAM, system board, and primary storage and (2) a cradle that provides multiple networking interfaces and power. The Intel processor provides additional functionality for real-time computing, such as cache allocation technology and time-sensitive networking, two features crucial to running control workloads that rely on deterministic performance, even over a network. This commodity hardware served as the basis for running the control workload.

Again, Red Hat Device Edge provides the operating system layer, but in a specific configuration to support these mission-critical workloads. First, the operating system is built into the edge deployment type leveraging rpm-ostree as the image and package system. Next, the realtime kernel was leveraged to provide more deterministic performance, as control workloads are particularly sensitive to jitter. On top of this foundation, Podman was leveraged to run the containerized control workload.

Finally, Schneider Electric provided the SoftdPAC-HA to serve as the control runtime. SoftdPAC interfaces with the I/O subsystem and is attached to the sensors and drives of the demo. The SoftdPAC control runtime has been engineered to facilitate bumpless failover leveraging the dedicated network interfaces on the cradles of the DCNs. It also housed the control loop within the construct of a boot project, allowing the runtime and control strategy to be packaged into one container image, then deployed into a highly available configuration simply by starting the container.

The automation framework

To automate the industrial automation, Ansible provided the capability to configure and control the DCNs, bringing them into a productive state as needed.

The automation was broken into a few individual function blocks, a workflow to:

• Prepare the DCNs to run the control workload
• Select a primary and secondary DCN according to existing workload and DCN condition
• Deploy the control workload according to the selected role of the DCN
• Deploy the monitoring stack to the ACP

These workflows were leveraged as individual blocks of work to be accomplished and allowed for the results and decisions to be passed along to other workflows. Thus, one parent workflow launched several child workflows to ensure the control landscape was healthy and in a highly available state. If the system was not healthy, or not in a highly available state, the proper remediation was run to reconcile.

Additional functionality

In addition, the ACP provided a host of additional functionality, such as:

• An image registry to house container images of the control workload
• Image Builder for composing images
• A monitoring stack
• Dashboards to display information from the monitoring stack
• Virtualization functionality for other non-Linux workloads

These supporting functions rounded out the demo and give valuable insight into the current state of the distributed control system, similar to an HMI or operator workstation today.

Realizing the next generation of distributed control systems

This new model using commodity hardware from Intel has enabled Schneider Electric to transform its business model from one that has been based on the sale of expensive and custom-designed hardware to a model that focuses on software, recurring revenue, and a better experience for its customers overall. The edge-ready infrastructure, automation, and orchestration of this next-generation smart factory are going to power the plants, oil refineries, and manufacturing lines that ultimately make the things we consume every day.