Vodafone is transforming its 5G telco cloud operations by embracing OpenShift, validated patterns, and GitOps to streamline its lifecycle management of Kubernetes clusters. Tom Kivlin, Principal Cloud Architect at Vodafone, recently shared insights into their journey, highlighting the challenges overcome and the benefits realized through this strategic approach. He spoke at OpenShift Commons in London earlier this year.
The challenge: Accelerating lifecycle operations in telco
Traditional telecommunication service provider environments tend to operate at a slower pace compared to the rapid release cycles of Kubernetes, which sees three releases per year. Vodafone overcame hurdles in managing the lifecycle of their Kubernetes clusters, including:
- Faster pace of lifecycle operations: The adoption of containers and Kubernetes in its 5G network to necessitate quicker feature releases and software updates, demanding more frequent upgrades.
- Complex dependencies: Tackling inconsistencies resulting from intricate relationships between 5G software applications and the platform itself, often managed through spreadsheets. Complex dependencies are written in one single location, making them easier to discover.
- Manual management and human error: Improving manual lifecycle management of clusters which could lead to issues during pre-production testing and validation. This avoids using manual deployments and "quick & dirty" fixes.
- Time-consuming and high-risk deployments: Avoiding these time consuming and high-risk deployments by having everything well defined as code in a single, tracked and traced common repository made things smoother and faster. Using "validated patterns" reuses common components that have been shared and improved by the community
“If you are managing the lifecycle of these clusters manually,” said Kivlin, “that can lead to problems that need fixing in pre-production as you're doing lifecycle operation testing and validation. We wish to avoid this to ensure deployments and upgrades are quick yet effective.”
The solution: Validated patterns and GitOps
Vodafone addressed these challenges by adopting validated patterns and GitOps, fundamentally changing how it manages its platform.
Red Hat’s validated patterns simplify platform lifecycle management. They reference blueprints as code, combining infrastructure-as-code and configuration-as-code concepts to create a collection of operator subscriptions, Helm charts, secrets, and other artifacts. These patterns can be used as-is or adapted to specific needs. Vodafone has leveraged two main upstream patterns: the multicloud GitOps pattern and the DevSecOps pattern. Working with Red Hat Consulting, Vodafone adapted these patterns by adding its own Helm charts and artifacts to create a pattern for deploying its management (hub) cluster. This hub cluster is then used with Red Hat Advanced Cluster Management for Kubernetes and zero-touch provisioning (ZTP) artifacts to build and manage numerous workload clusters running cloud-native network functions (CNFs) that provide 5G functionality.
GitOps: Automation and consistency
Vodafone integrates GitOps by storing these adapted patterns in Git, utilizing Git version control, branch protection policies, and approval paths to automate configuration through GitOps with Argo CD. This approach provides consistency and reproducibility.
Vodafone's custom pattern: A closer look
Vodafone's pattern incorporates a variety of operators and custom Helm charts. Key components include:
- Red Hat Advanced Cluster Management for Kubernetes: For managing the lifecycle of multiple clusters and applying policies.
- Red Hat Advanced Cluster Security for Kubernetes: For security.
- Red Hat OpenShift Data Foundation: For storage within the hub cluster.
- Red Hat Quay: A container registry for storing images, particularly useful in disconnected environments where images need to be mirrored locally.
- Keycloak: Vodafone's authentication provider for single sign-on across workload cluster APIs, consoles, and applications.
- Red Hat Ansible Automation Platform: Used to manage the lifecycle of external Ceph storage clusters for workloads.
- Observability Suite (part of Red Hat Advanced Cluster Management): Centralizes metrics, alerts, and logs from workload clusters and platform components, providing a single view for operations.
- Custom Helm charts: For elements like custom certificates signed by their own root CA, and OpenShift APIs for Data Protection configurations for backup and restore needs.
Benefits and lessons learned
Vodafone has experienced significant benefits from this transformation:
- Increased reliability: Moving from manual configuration to version-controlled YAML files in Git, automated via Argo CD, allows for consistent and repeatable deployments.
- Reduced human error: The chance of human error is vastly reduced due to Git review techniques.
- Enhanced agility: The improved reliability and reduced errors contribute to increased agility.
- Simplified policy management: Red Hat's PolicyGenTemplate, a custom resource within ZTP artifacts, simplifies the creation of Red Hat Advanced Cluster Management policies for managing multiple clusters at scale.
- Robust version control: Utilizing standard Git mechanisms like branch protection rules and code owner files ensures proper review and approval of changes, triggering automated configuration.
The journey has seen Vodafone adopt new working practices and processes:
- Steep learning curve: Shifting from GUIs and spreadsheets to managing everything in YAML with Git and Argo CD pipelines has led engineering, operations, and architecture teams to adopt new methods.
- Tracking upstream helm charts: Vodafone aims to further refine the ability to link and track differences with upstream Helm charts for Red Hat components, especially since the company has customized some for its specific needs (e.g., Red Hat Advanced Cluster Management integration with its backup system). Vodafone plans to collaborate with Red Hat to incorporate the necessary features into upstream charts.
- Telco-specific pattern: Vodafone is exploring the need for a dedicated telco pattern, debating whether their specific requirements are unique enough to warrant a separate industry pattern or if existing patterns can be sufficiently adapted.
Outlook and key takeaways
Vodafone's adoption of OpenShift, validated patterns, and GitOps is a testament to the power of configuration as code in modernizing telco cloud operations. The benefits in reliability, reduced human error, and increased agility are driving factors in its continued success. Vodafone’s collaborative approach with Red Hat and commitment to refining their processes highlight a forward-thinking strategy in the evolving 5G landscape.
For those interested in learning more about validated patterns, Tom Kivlin recommends the OpenShift Commons Validated Pattern special interest group webpage, which offers a wealth of articles, videos, and blog posts. Here is the latest session on Youtube where Tom Kivlin, Principal Cloud Architect at Vodafone, explores how Vodafone Networks has successfully implemented the Validated Patterns Operator in conjunction with GitOps to automate & simplify the lifecycle management of bare metal OpenShift clusters.
关于作者
Red Hatter since 2018, technology historian and founder of The Museum of Art and Digital Entertainment. Two decades of journalism mixed with technology expertise, storytelling and oodles of computing experience from inception to ewaste recycling. I have taught or had my work used in classes at USF, SFSU, AAU, UC Law Hastings and Harvard Law.
I have worked with the EFF, Stanford, MIT, and Archive.org to brief the US Copyright Office and change US copyright law. We won multiple exemptions to the DMCA, accepted and implemented by the Librarian of Congress. My writings have appeared in Wired, Bloomberg, Make Magazine, SD Times, The Austin American Statesman, The Atlanta Journal Constitution and many other outlets.
I have been written about by the Wall Street Journal, The Washington Post, Wired and The Atlantic. I have been called "The Gertrude Stein of Video Games," an honor I accept, as I live less than a mile from her childhood home in Oakland, CA. I was project lead on the first successful institutional preservation and rebooting of the first massively multiplayer game, Habitat, for the C64, from 1986: https://neohabitat.org . I've consulted and collaborated with the NY MOMA, the Oakland Museum of California, Cisco, Semtech, Twilio, Game Developers Conference, NGNX, the Anti-Defamation League, the Library of Congress and the Oakland Public Library System on projects, contracts, and exhibitions.
