Streamline lifecycle management of 5G Core CNFs on Red Hat OpenShift

Upgrades are necessary in any industry that relies upon IT infrastructure to do business. From adopting new features and addressing common vulnerabilities and exposures (CVE) to enhancing security and complying with support requirements, there are many business-critical reasons to upgrade your software infrastructure. This is especially true for service providers operating in rapidly changing market conditions. By offering the fastest network with advanced services and features, upgrades help service providers maintain customer satisfaction, grow revenue, and ensure long-term success. Even so, upgrades can be challenging—particularly when they don’t fit into maintenance windows. In fact, operations teams are often forced to postpone upgrades that require too much time.

Red Hat streamlines the upgrade, deployment, and configuration of Red Hat OpenShift clusters running 5G core cloud-native network functions (CNFs). A simplified upgrade process helps you keep your software infrastructure current with the latest features and security enhancements. By ensuring compatibility between CNFs, application platforms, and underlying operating systems, Red Hat helps you reduce operational complexity while maintaining optimal performance and reliability across your 5G network.

Prepare your Red Hat OpenShift upgrade

As with any upgrade, preparation is critical to success. Follow these steps to get ready to upgrade between Red Hat OpenShift Extended Update Support (EUS) releases. By updating between consecutive Red Hat OpenShift EUS releases, you can upgrade Red Hat OpenShift from version 4.Y to 4.Y+2 in a single process.

Step 1: Review the latest documentation

Red Hat OpenShift documentation provides the commands necessary to perform an upgrade. There are also comprehensive release notes detailing features, enhancements, and bug fixes in the latest version. Additionally, the documentation describes many configuration settings that can impact an upgrade. Incorrect configurations can cause upgrades to fail and lead to disruptions, so it's important to understand these settings.

Here are some key configuration settings to consider before starting your upgrade:

• The pod disruption budget (PDB) allows a specific number of pods to be down during voluntary disruptions. Ensure that at least 1 pod can be down at any time
• Anti-affinity spreads pods of a single deployment across multiple worker nodes within the cluster, reducing the time required to drain, reboot, and upgrade a node
• After receiving a SIGTERM signal, pods must finish in-flight tasks and shut down gracefully Optimizing terminationGracePeriodSeconds helps minimize disruptions, especially when pod shutdown takes longer than the default of 30 seconds
• If a pod starts—but is not fully ready—while another pod stops, significant problems can occur in your CNF deployments. Setting meaningful readiness probes helps bring pods back online correctly

Step 2: Determine the correct Red Hat OpenShift update path

It is important to upgrade to a release with the same patch level. Use the Red Hat OpenShift Container Platform update graph tool to determine the correct update path. Carefully read through any potential problems identified by the tool, as they may not apply to your cluster configuration.

Step 3: Plan your CNF upgrades

Maintaining functionality of your 5G core CNFs during the upgrade process is critical. Consult the CNF best practices 5G core solution documentation for more information specific to CNF and telecommunications cluster configurations.

Step 4: Ensure operator compatibility

Red Hat OpenShift does not automatically upgrade operators installed via the Operator Lifecycle Manager (OLM). Instead, you must approve each upgrade individually. Consult the Red Hat OpenShift Container Platform operator update information checker to determine which operators you need to update during the intermediate upgrade (see step 11).

Step 5: Load images into offline repositories

Industry best practices recommend verifying images before deploying them into production. Red Hat recommends updating offline repositories with all images before starting the upgrade.

Step 6: Backup etcd

The key-value store for Red Hat OpenShift is etcd, which persists the state of all resource objects. Use the etcd backup procedure to store etcd data in a secure location outside your Red Hat OpenShift environment.

Step 7: Pause worker node MachineConfigPools

Pausing MachineConfigPools (MCP) lets you prepare—but not push—all machine configuration changes for worker nodes. When you unpause MCPs, Red Hat OpenShift applies all changes to each worker node after the node reboots. This ensures that worker nodes only reboot once during an Red Hat OpenShift EUS upgrade.

Step 8: Prepare firmware updates

Changes in Red Hat OpenShift 5G core CNFs may require firmware updates for clusters running on bare-metal servers. Consult the release notes for further information.

Step 9: Verify cluster health

Ensuring a cluster's health before initiating the upgrade is essential. Use this command to verify the cluster's status:

$ oc get clusterversion; echo; \
oc get co; oc get no; echo; \
oc get po -A

Perform your Red Hat OpenShift upgrade

After completing the necessary preparations, it's time to perform the upgrade. Follow these steps to smoothly upgrade your Red Hat OpenShift installation between EUS releases.

Step 10: Update the Red Hat OpenShift control plane from 4.Y to 4.Y+1

Upgrade the Red Hat OpenShift control plane to the intermediate release (4.Y+1). The first step of the process is to update the cluster operators. While most operators reside on control plane nodes, some are—or have pods—on worker nodes. The process then reboots the control plane nodes. Although the upgrade does not affect CNF pods or worker nodes, Red Hat recommends that you perform the update during a maintenance window.

Step 11: Update OLM-based operators from 4.Y to 4.Y+1

Based on the results of step 4, upgrade any OLM-based operators that require an intermediate update to a 4.Y+1 compatible version. Even though most OLM-based operators reside on worker nodes, those nodes do not reboot during this process. Also ensure that all OLM-based operator images corresponding to the final release version are available in offline repositories.

Step 12: Update the Red Hat OpenShift control plane from 4.Y+1 to 4.Y+2

Upgrade the Red Hat OpenShift control plane to the final release (4.Y+2). As before, this process updates the cluster operators and reboots the control plane nodes without affecting CNF pods or worker nodes. This process updates the Red Hat OpenShift control plane operators and nodes to version 4.Y+2, while worker nodes in the cluster continue to run version 4.Y.

Step 13: Update OLM-based operators from 4.Y+1 to 4.Y+2

Upgrade the OLM-based operators to a 4.Y+2 compatible version. This may take longer than step 11 because a larger number of operators may need to be updated.

At this point, the Red Hat OpenShift control plane and all OLM-based operators are running version 4.Y+2. However, worker nodes are still running version 4.Y. This is a fully supported configuration, so you can safely stop the upgrade process at this point if necessary.

Step 14: Update worker nodes

Unpause 1 or more MCPs. This upgrades and reboots all worker nodes within each MCP. If there are multiple MCPs in a cluster, unpause subsequent MCPs as time allows.

As this step executes, the control plane and some of the worker nodes run version 4.Y+2 while other worker nodes run version 4.Y. Because this is also a fully supported configuration, you can again stop the upgrade process during this step. This lets you ensure that worker nodes are healthy before unpausing additional MCPs during the next maintenance window.

The upgrade-aware scheduler in Red Hat OpenShift helps reduce the number of times a pod is restarted. By adding a taint to all nodes in the cluster—and removing it when the upgrade completes—the upgrade-aware scheduler favors starting new pods on upgraded nodes.

Once all worker nodes are upgraded, the upgrade process is complete!

Observations from the field

IT professionals in the field have successfully followed this process to upgrade an in-service Red Hat OpenShift cluster running standard test traffic. The teams observed minimal impact on signaling and payload while the nodes were being updated, and no manual intervention was required to resume normal full throughput after each upgrade step completed.

That said, not every upgrade goes smoothly. There is always the potential for an issue, like a problematic process, hardware failure, or undocumented network change. Remember that there is a clear path to success:

• Preparation is key
• Follow the path
• Manage with stopping points
• Verify before proceeding

With these steps, your upgrade journey can be much easier. Learn more about the online documentation for upgrading a telco core CNF cluster.