Building a soft real-time vPAC with Red Hat Enterprise Linux, KVM, and Podman

For decades, the power industry has relied on "black box" proprietary appliances. While reliable, these hard-wired fixed-function devices have created a landscape of vendor lock-in, where hardware refresh cycles (often lasting 20 years) dictate the pace of software innovation. The operational effort required just to apply a cyber-security patch can be immense, diverting resources that could be focused on introducing new functionality or AI capabilities.

As we face a global talent shortage and the urgent need to integrate renewables and AI data center demands into the grid, the status quo is no longer sustainable. We need a software-defined electrification strategy that untangles the grid of these black boxes. Similar to the 4G and 5G transformation in telecom networks, open source technology is a key enabler for this, and where Red Hat is the enabler for utilities to make open source components usable as enterprise-grade platforms for critical infrastructure. Our open platforms are tested for multi-vendor and multi-technology workloads of mixed-criticality running side by side.

This article details the Red Hat lab configuration for digital substations, providing a foundation for you to begin testing with open Red Hat platforms in your own environments.

Compute infrastructure for the digital grid

The IEC 61850 standard introduces digitization of substations on the electric power grid. This means many hard-wired signals are replaced by digital communications using Ethernet (so-called station and process bus).

The only hard-wired interfaces are those directly connected with the primary equipment such as circuit breakers or transformers. Merging unit (MU) devices are used to translate analog signals for voltage and current into digital format, and to make them available to all intelligent electronic devices (IED) for protection, automation, and control, as well as all the other systems performing monitoring, measurements, and visualization. Eventually, this will also make it easy for even AI systems to use.

Historically, utilities are used to buying single-purpose appliances from vendors. These appliances lock in proprietary management and maintenance procedures and tool ecosystems. Adding new functionality means adding (and wiring) more boxes. Cybersecurity updates happen rarely, due to huge efforts requiring personnel on-site to interact with equipment wired to significant voltage.

With communication now happening digitally, utilities can introduce an open platform for all their compute infrastructure in the substation. Process-interfacing units (PIU) such as breaker IEDs (brIED) or MUs cannot be affected now, but critical functions for protection, automation and control (PAC) will be run virtualized, as a virtual machine (VM) or container, side by side with non-critical workloads such as gateways, SCADA, human machine interfaces (HMI), or protection engineer workstations.

Figure 1: Red Hat platforms power compute infrastructure on the modern power grid.

At Red Hat, we believe that this compute infrastructure should be an open platform:

• Agnostic to any vendor: This includes the established dominant players in this space as well as new startups entering this space with innovative solutions.
• Ready for any technologies: VM, container, AI models, and more.
• Enabled for mixed-criticality workloads: Critical workloads requiring real-time performance such as PaC systems, as well as any other workloads even if it can only be connected occasionally to the substation (like a Windows-based engineering workstation to test or configure proprietary equipment).

We see this approach as the on-premises (edge) branch of the open hybrid cloud. The platform uses the same technology as you would find in a control room, labs, or even the public cloud. It allows for interesting new applications of digital twin and testing concepts, speeding up the go-live of new features on the grid.

Red Hat vPAC demonstrator

To show what is possible today, we have built a functional virtualized protection, automation and control (vPAC) demonstrator in the Red Hat labs. This setup proves that mission-critical protection workloads can run on an open, standardized compute platform without sacrificing the deterministic performance required by the digital grid.

The setup consists of an Intel Xeon-based industrial server that acts as the vPAC hypervisor hosting different functions, networking infrastructure to segregate process bus, management and station bus network traffic, as well as a precision time protocol (PTP) clock. We perform real-time tests mimicking a medium-voltage substation using a hardware-in-the-loop (HIL) simulator. 

Figure 2: Front view of the Red Hat vPAC Demonstrator rack.

Figure 2 shows the front of the small rack. At the bottom, you can see the Advantech server with Red Hat Enterprise Linux installed. Going upwards from there, you see the station bus switch, a power distribution unit for the rack, the Typhoon HIL simulator, the process bus switch, and the Grandmaster Clock from Meinberg at the top of the rack.

While we've used certain vendors and models in this setup, Red Hat is agnostic here and we support alternatives for your own setup. The following sections on the individual components highlight the requirements and known alternatives. Sections at the end of this article detail the bill of materials for both hardware and software components.

Our HIL simulates a medium voltage substation with ten bays on a single busbar. The HIL sends the simulated IEC 61850 sampled value (SV) streams of all ten bays to the process bus. We have completely separated the process bus from all other network traffic on the station bus by using two different switches.

These SV streams actually are picked up by virtual functions running on our vPAC host, an Advantech ECU 579. Like the switches, this system is 61850-3 certified for use in substations. We also certified other models and vendors that offer both fanless and fan-cooled servers for this use case.

The Grandmaster Clock with GPS keeps all components in sync through PTP.

Figure 3: Overview of components used in the demonstrator.

The vPAC host is running Red Hat Enterprise Linux with virtualization capabilities for both containers and VMs. We're using the real-time kernel with a vPAC-specific tuning profile, including the RT-version of the kernel-based virtual machine (KVM) hypervisor. Of course the system is hardened for trustworthiness and resiliency, and gets deployed and configured using Red Hat Ansible Automation Platform according to best practices and security profiles. The protection workload itself here is ABB SSC600SW and Euto Energy vSAS. There is also a vPAC Engineer Workstation with all the config tools to access the specific systems. This workstation is equivalent to a laptop a protection engineer today would manually plug into the network when they get to the substation. In our environment it is based on Windows as some of the configuration tools have a hard dependency here, generally this could also be a Red Hat Enterprise Linux system with full desktop experience.

ABB offers virtualized protection and control in all of their VM implementations and was one of the first players going to market with this approach. Running in Podman is Euto Energy, a startup from the Netherlands offering vPAC, virtualized HMI (vHMI), and virtualized remote terminal unit (vRTU) functionality in a container. This setup is a great showcase for us because:

• We can see multi-vendor vPAC in one box (ABB and Euto).
• We demonstrate readiness for multi-technology in one box (VM and container).
• We run mixed criticality in one box (critical real-time-specific workload for vPAC next to non-critical workstation VM). We are isolating the CPU cores for the real-time-specific workloads. 

Our end-to-end scenario includes injecting faults into the grid via the HIL; we see the protection function on the vPAC host reply with an IEC 61850 Goose message signaling a trip to open the circuit breaker on the simulator. Thanks to the PTP sync, we can actually measure that all of this is happening within the required 10 ms limits on the HIL and observe what is happening on the dashboards provided by both ABB and Euto.

While the simulation and networking equipment in your lab can be different, the deployment and configuration of the hypervisor is also reproducible in other contexts.

The core: Hardware and timing setup

The foundation of the demonstrator is industrial-grade, IEC 61850-3 certified hardware, in conjunction with a real time simulator for IEC 61850 9-2 traffic.

Compute node

We use an Advantech ECU 579 server as our stand-alone vPAC hypervisor node. The node is equipped with 16x Intel Xeon Gold 6226R CPU @ 2.90GHz, 128GB DDR4 RAM and a 1TB NVMe disk.

Red Hat’s minimum requirement for running Red Hat Enterprise Linux at the edge is one core (64-bit x86 or ARM), 2GB RAM and 10GB disk space. The additional available capacity gives us plenty of room to run multiple virtualized functions with dedicated CPU cores for deterministic performance as well as management functions. A minimum of 6 NICs is used to separate the different traffic in this single-node setup:

• 1 onboard IPMI (Intelligent Platform Management Interface, connected to Management network on Station Bus Switch)
• 1 management interface (connected to Management network on Station Bus Switch)
• 1 station bus NIC for host with bridge for workload (connected to station bus network on station bus switch)
• 1 PTP NIC (connected to process bus switch)
• 1 process bus NIC for host with bridge for workload (connected to process bus switch)
• 1 process bus NIC for pass-through per virtual workload (connected to process bus switch)

Networking

Two Advantech EKI-8528 switches manage the network traffic, ensuring high-speed data flow between the simulator and the compute node. All traffic of the station bus, as well as management connections, are on the station bus switch. All process-related traffic (PTP, SV streams, GOOSE) is separated on the process bus switch. Where possible, we're using fiber connections (especially for the process bus communication of the compute node and clock) as recommended by the vPAC Alliance. We have to use copper RJ45 interfaces for the onboard IPMI management interface of the server as well as the connections to the HIL simulator, resulting in a mixed operation of both wire types. All interfaces are Gigabit ports. For more advanced cluster scenarios with multiple nodes, we would require one or more additional 10Gb ports for intercluster communication for each server.

Grandmaster Clock

Timing is non-negotiable in a substation and especially for our real-time validation tests. A Meinberg Lantime M1000 provides the PTP grandmaster clock, ensuring sub-microsecond synchronization across the entire demonstrator. The clock features a global navigation satellite system (GNSS) antenna and provides PTPv2 data with the Utility IEC 61850-9-3 profile on the process bus. When operating the demonstrator in closed facilities, we enable simulation mode for GNSS and rely on the clock's internal oscillator.

HIL simulator

A Typhoon HIL 606 acts as the digital twin of the electrical grid, simulating primary equipment and generating sampled values over the process bus. We're using the 4 on-board ethernet ports to separate the different types of network traffic:

• 1 interface on the station bus for the connection to the HIL SCADA interface (running on the Windows VM)
• 1 interface dedicated to GOOSE messages (incoming from the protection function as well as outgoing)
• 1 interface for the generated SV traffic of our 10 bays and the bus bar
• 1 interface for PTP synchronization

The hypervisor layer: Red Hat Enterprise Linux at the edge

The "brain" of the demonstrator is Red Hat Enterprise Linux. The operating system (OS) decouples the protection logic from the hardware, allowing utilities to ship cybersecurity patches and protection updates in days instead of years. It's a hybrid environment that supports virtual machines for legacy-style software appliances and containers for modern protection functions. To meet the requirements of substation environments, we use several key open source components in our platform:

• Real-time (RT) kernel: The RT kernel achieves determinism through interrupt latency reduction and priority inheritance. Specifically tuned profiles matching the host and workload ensure deterministic execution for virtualized IEDs. This includes specific configurations for CPU-core pinning, hugepages, and Intel Cache Allocation Technology (CAT) for minimal jitter and determinism.
• KVM: This enterprise-grade, open source virtualization technology turns the Linux kernel into a type-1 hypervisor, allowing the ABB SSC600SW VM to run a high-performance VM with near-bare-metal efficiency, protected from the Windows Engineering Workstation VM running next to it.
• Podman: A daemonless container engine used to deploy and manage modern, microservices-based protection functions like the Euto Energy vSAS, ensuring these applications are isolated, trustworthy, and easy to update.
• Ansible Core: Infrastructure-as-code principles act as a "workforce multiplier," allowing a small team to automate the deployment, maintenance, and cybersecurity compliance of a heterogeneous fleet of devices at scale. Ansible's idempotency ensures that running a playbook multiple times results in the same system state without making unnecessary changes, even if tasks are executed repeatedly. This prevents configuration drift between different bays or substations over time. Here we run it locally on just the single demonstrator system, but we foresee integration into a central management system like Red Hat Ansible Automation Platform for broader management. 
• Cockpit (web console): A user-friendly, web-based graphical interface that allows substation technicians to monitor system health, manageVMs and containers, and inspect real-time performance metrics without needing deep command-line expertise. It’s an HMI for our vPAC server.
• PTP: PTP for Linux (Ptp4l) synchronizes the system's local clock with a master clock according to the IEEE 1588 standard.
• Security and compliance-profiles: Red Hat Enterprise Linux by default provides SELinux for mandatory access control, firewall daemon, system-wide crypto policies for UEFI Secure Boot. With the OpenSCAP scanner and the pre-installed SCAP Security Guide, we can compare a system against benchmarks from the Center for Internet Security (CIS) Benchmark Server - Level 2, and generate a comprehensive report. This even comes alongside remediation scripts to speed up fixes to any non-compliant settings and provides input for updates to the deployment template for the platform. This also allows for continuous compliance, which is a major pain point for utilities undergoing NERC CIP or NIS2 audits. 
• Virtual networking and passthrough: This layer uses advanced networking techniques to bridge the Station Bus for general management to VMs while using NIC passthrough for the process bus to ensure the lowest possible latency and jitter for critical IEC 61850 sampled value and GOOSE traffic.

Virtualized workloads

The Red Hat platform is open to mixed-criticality workloads in different form factors and agnostic to application vendors. On a single host, we run workloads from multiple vendors (in the demonstrator, that's ABB and Euto Energy) side-by-side:

• ABB SSC600 SW (VM): SSC600 SW is a virtualized protection and control solution that delivers the full functionality of a central protection relay as a software-only package that runs on standard industrial computers hosting a hypervisor such as KVM. 
• Euto Energy vSAS (container): The Euto Energy virtualized Substation Automation System (vSAS) provides containerized functions for vPAC, vRTU (such as protocol conversion), and vHMI for real-time monitoring and visualization. It is deployed as a container on Podman.
• Engineering Workstation (Windows VM): This node allows engineers to run familiar configuration tools directly on the substation host. In our demonstrator, this includes ABB PCM600 to configure the vIED. A browser allows access to the web user interfaces (UIs) of ABB SSC600SW, Euto Energy vSAS, RHEL web console, as well as managing all the switches and bare-metal server. We also run the Typhoon HIL Control Center for managing our HIL through the SCADA application. We can open a remote desktop connection to this machine, so that anybody with access can execute the demo. No need for setting up different tools on various machines with different SW versions and network configurations. 

In the context of software-defined electrification, containers provide significant operational advantages over VMs by sharing the host OS kernel, which reduces compute overhead and increases the capacity for consolidated workloads on industrial-grade hardware. This streamlined architecture supports a cloud-native approach to the grid edge, facilitating consistent testing environments and rapid deployment cycles through automated DevSecOps pipelines. While containers excel in scalability and agility, VMs are closer to traditional monolithic hardware appliances and can be easier to adopt. Both technologies contribute to the trustworthiness and deterministic performance required for the digital grid. By utilizing a hybrid strategy on a common platform, engineers can use containers for operational efficiency while continuing to rely on VMs as the market evolves.

Real-time simulation scenario

The demonstrator setup offers us a lot of flexibility to run different simulations against all kinds of virtual workloads for real-world test scenarios.

To validate that the mission-critical protection workloads shown above can thrive on a virtualized platform, we’ve built the following end-to-end scenario that allows us to perform high-fidelity, real-time tests that mimic a medium-voltage substation featuring 10 bays on a busbar. The single-line diagram below gives an overview of the simulated (and protected) environment.

Figure 4: Single-line diagram of the simulated substation.

Bay 1 is an incoming bay, bays 2 to 10 are feeder bays. We generate SV streams for all 10 as well as busbar 1 (20kV) with overcurrent protection on the feeders and a differential protection for the busbar.

1. Initializing the digital twin

The demo journey begins at the Engineering Workstation (remote desktop connection to the Windows VM), where we initialize the substation simulation via the Typhoon HIL SCADA interface. This simulator acts as the "digital twin" of our primary equipment, generating IEC 61850 SV streams for all 10 bays and injecting them directly onto the process bus. The screenshot below shows the SCADA panel with the simulated feeders with their current values as well as the circuit breakers and their status (open/close indicated by green and red lights). 

Figure 5: HIL SCADA interface showing simulated voltage and current.

2. Multi-vendor protection in action

Within the ABB SSC600SW web interface, we can see the captured SV streams being visualized in real-time. The screenshot below shows the phasor diagrams for currents on bays 1, 2, and 3.

Figure 6: Phasor diagrams on ABB SSC600SW.

The system performs constant self-supervision to ensure that processing latencies remain within the microsecond range, ensuring that the virtualized environment meets the rigorous deterministic requirements of the grid. This result is an important proof-point for the correct configuration of the RT kernel and CPU-core pinning. The screenshot from the self-supervision below shows processing latencies over a 12 hour period. The first 6 hours represent a more-or-less idle state (no simulation running), but even under load with the simulation latencies do not change much.

Figure 7: Screenshot of self-supervision of ABB SSC600SW.

Because the platform is vendor-agnostic, we can simultaneously spin up the Euto Energy vSAS container. Despite having a different network interface to the process bus, it processes the exact same SV data streams as the ABB VM, demonstrating how software-defined functions from different providers can coexist and cooperate on a single hardware node. Both running isolated from each other on different CPU cores, of course. 

Figure 8: HMI view in Euto Energy vSAS.

Figure 8 shows the details of one specific bay.

This capability allows protection use cases even on a single server. Both technologies could be run in parallel and operate in active mode to prove that utilities can achieve protection function redundancy on a single server using different vendors. In case of a failure, both solutions could detect it and send a GOOSE trip simultaneously (or we could measure which solution is faster). Or we split the responsibilities for the sweet spots of different protection algorithms and have a single vendor protect the busbar while the other looks after the feeders. 

3. Fault injection and "zero-time" response

The true test occurs when we inject a fault into the simulator, such as a three-phase-to-ground fault on bay 3, an overcurrent on bay 7 or a busbar fault. Our SCADA interface injects this type of fault from a drop-down selection menu and also highlights the status of our circuit breakers (open/closed) with green and red status lights. Figure 9 shows that the circuit breaker for bay 3 is already closed after the injected fault.

Figure 9: Simulating a fault via the HIL SCADA.

This is what happened exactly:

• Injected fault: The simulation was changed from normal operation to simulating a fault with all three phases A, B, and C to ground (N) on bay 3.
• Detection: The protection functions running on top of the Red Hat Enterprise Linux host instantly detect the faulty values, as seen on the phasor diagrams and alarm dashboards.
• Mitigation: The vPAC function issues an IEC 61850 GOOSE message signaling a "trip" to open the circuit breaker on the simulator.
• Validation: Thanks to PTP synchronization, we can measure the entire loop from fault injection to the circuit breaker opening, confirming it stays well within the desired 10ms limit.

When the virtual protection relay acts upon the overcurrent measurements, the ABB SSC600SW dashboard looks somewhat like figure 10.

Figure 10: Alarm dashboard in ABB SSC600SW

The delay time in our overcurrent protection is set to 300ms (see figure 11).

Figure 11: Protection function settings in ABB SSC600SW.

Because we know in our scenario here that a fault will happen, we set up time measurements also on the HIL SCADA side. We use the fault injection as a trigger for starting the measurement, so that afterwards, we can see how long it took the overall system from fault occurrence until receiving the GOOSE message to open the breaker. Looking at the captured values, we can see the wave forms of the three-phase meter on the upper panel and observe the failure state. The bottom panel shows the status change of Bay3.Digital Probe6, which is our circuit breaker in bay 3. The change from 1 to 0 indicates when the breaker status changed to open. 

Figure 12: Measurements in Typhoon HIL SCADA

Beyond individual "manual" faults, we use the HIL API to run automated stress tests over extended periods, such as 25-hour cycles with faults occurring every minute. This rigorous validation ensures that there are no spikes in response times that could lead to equipment damage or outages in a live grid environment. 

4. Observability on the host

So far we've displayed a lot of screenshots, but the Red Hat platform has done its duties quietly and reliably in the background. Let’s have a look at the Red Hat Enterprise Linux web console to see how the platform performs.

Web console provides a web-based graphical interface for real-time host observability, allowing technicians to manage the substation server without extensive command-line expertise, for example, to perform network configuration and software updates or to gather logs and data for troubleshooting.

The system overview page provides an immediate snapshot of hardware health and resource use, while the metrics dashboard allows for granular monitoring of CPU, memory, and network performance.

Figure 13: Web Console on RHEL showing system information and monitoring data.

For operational flexibility, the interface includes integrated tools for VM and container management, supporting the deployment and inspection of virtualized protection functions directly from the browser.

Figure 14: Management view of ABB virtual machine on the Red Hat web console.

Additionally, a built-in terminal provides protected access for advanced troubleshooting, ensuring that the underlying infrastructure remains transparent and accessible in mission-critical vPAC environments.

Figure 15: Management view of Euto Energy container on the Red Hat web console

This concludes the demo. By deconstructing the "black box" approach, this scenario proves that the vendor lock-in is gone. Utilities can now augment pure protection with advanced analytics or run redundant functions from multiple vendors side-by-side, simply by deploying a new container or VM. 

The art of the possible

Our demonstrator shows that utilities don't need to be locked into a single provider's lifecycle. You can run protection workloads from different vendors on a single node, update cybersecurity logic instantly, and scale your infrastructure for the age of AI and renewables.

A production architecture of this kind could demand for more resilience and high-availability clustering. While it’s possible to run 2 stand-alone systems like this next to each other, both hosting the same vPAC functions in an active/active or active/standby configuration on the application level, we recommend implementing a three-node cluster architecture to unlock all the benefits that virtualization has to offer. This approach also facilitates rolling maintenance and uninterrupted security updates, allowing utilities to modernize their infrastructure without the operational downtime associated with traditional fixed-function appliances. By decoupling the protection logic from the physical hardware, a clustered environment enables automated workload migration and high availability to ensure continuous operation during hardware failures or minimize the time for a single point of failure during planned maintenance activities. This brings the dream of zero-downtime maintenance a lot closer to reality.

This approach isn't just a vision by yet another vendor; it is an industrial-grade reference design based on the LF Energy SEAPATH project, comprising more than 1,200 mainstream open source software packages. This community-led model provides utilities with the digital sovereignty needed to maintain full control over their infrastructure lifecycles and security posture independently of proprietary vendor roadmaps. It establishes an agnostic foundation that ensures long-term investment protection by allowing the industry to collaborate on solving complex grid challenges through transparent and shared innovation.

Implementing the reference design in your lab 

The vPAC Demonstrator is your starting point for hands-on testing. Whether you are a utility, integrator, or vendor, we are ready to help you accelerate your lab setup, deploy Ansible playbooks, and test your specific workloads. This entire "substation in a box" stack is designed to be fully replicated directly into your own onsite testing facility. Let’s build the next generation of the digital grid together. Connect with us today to get started. 

Hardware bill of materials

Software bill of materials

Key software products:

Low-level software components: