Worldpay's Platform as a Product: Revolutionizing development with Red Hat OpenShift

Worldpay has fundamentally transformed its development processes by leveraging Red Hat OpenShift as the foundation for its internal developer platform, dubbed the Global Kubernetes Orchestration Platform (GKOP). This strategic shift has enabled Worldpay to empower its development teams with self-service capabilities, fostering a culture of collaboration and efficiency.

GKOP: More than just OpenShift

Launched in 2020, GKOP is an enterprise platform built on Red Hat OpenShift. It is PCI DSS compliant and manages a fleet of 28 clusters across multiple regions. Worldpay views GKOP as a product, not just a platform. Things must run smoothly and reliably, so the GKOP team has built extensive automation around it.

“GKOP is our product, because it's so much more than just OpenShift. We built a lot of automation around it so the backbone of GKOP is the orchestration and configuration management. We asked ourselves ‘How do we do it? How do we run a fleet of clusters and manage them at scale?" asked Bernd Malmqvist, director of platform engineering at Worldpay. “Our cluster management basically is rendering the configuration [which is then] applied to downstream clusters… In disaster recovery exercises, we delete a region and rebuild it from scratch, from start to finish, within one and a half hours. We achieve this because the configuration is pre-rendered. When a cluster comes back online, configuration can be instantly applied.”

This cluster agility allows Worldpay to create sandbox and development clusters which are regularly reprovisioned daily and weekly, respectively. Higher priority environments, like staging and production, are longer-lived. Weekly platform releases bundle changes and promote them from lower to higher environments, making sure that all clusters are copies of each other, which helps catch regressions before they reach production.

The pillars of a compelling platform product

Worldpay's definition of a compelling platform product centers on user experience and efficiency. It must be easy to use, focused on consumer needs, and provide self-service with instant feedback. Worldpay had the following goals when building out GKOP:

• Feature-rich and flexible: The platform offers independent yet interactive features, providing flexibility and guidance to developers.
• Integrated CI/CD toolset: It comes with built-in CI/CD tools to guide developers through the application lifecycle, eliminating the need for them to spend hours integrating with various tools.
• Built-in enterprise service integrations: GKOP integrates seamlessly with Worldpay's enterprise services like NTP, authentication endpoints, secret management, certificate management, and logging. Notably, its integration with AWS STS provides short-lived tokens, enhancing security by eliminating the need for embedded credentials.
• Robust security: The platform includes container runtime security and vulnerability reporting to reduce developer toil in managing an expanding attack surface.
• Ease of use and reliability: Providing how-to guides, example services, and step-by-step instructions provides accessibility for teams with varying maturity levels. The platform is designed to be reliable, handling peak loads and inspiring developer trust.
• Extensive operators: GKOP leverages both third-party and in-house built operators to extend the Kubernetes API and provide self-service capabilities. Examples include operators for databases, caching, messaging, and a custom egress proxy operator for PCI compliance.

API-driven self-service and automated validation

A cornerstone of GKOP is its API-driven self-service capability, which unifies all platform services into a cohesive product, empowering developers to fulfill their own requests. Worldpay is committed to this paradigm, making sure that every new platform addition is automated and adheres to self-service principles.

To maintain stability and mitigate risks associated with this flexibility, Worldpay places strong importance on automated validation and review. Their custom validator service, configured via Kubernetes webhooks, checks API requests for adherence to best practices and configured rules. This process involves:

• Role-based access control (RBAC) check: Verifying user permissions.
• Mutating webhook: Adding sensible default values or improvements, such as pod topology spread constraints.
• Schema conformance: Kubernetes API automatically checks if the resource conforms to its schema.
• Validating webhook: Warning users about non-compliance with best practices or denying requests for impermissible values.
• Persistence and reconciliation: The resource is then persisted and reconciled.

This automated validation can provide recommendations and links to documentation. It also enforces critical rules, such as those for pod disruption budgets, to prevent issues during OpenShift upgrades. For platform resources, a GitHub pull request triggers the validator, which performs schema validation and suggests changes, and can even escalate reviews to a human engineer if a decision cannot be made automatically. Templated onboarding workflows with pre-configured and approved resources, coupled with automatic merging, further accelerate scaling and ease of use for developers.

Cultivating a great developer experience

Recognizing developers as internal customers, Worldpay focuses on providing a seamless experience. This includes easily accessible documentation through a developer portal launched 2 years ago. The portal follows an inner-source model, allowing any developer to contribute.

Worldpay also built a GKOP Free Tier version—a daily reprovisioned sandbox environment with limited features—allowing any Worldpay employee to log in, create a namespace, and deploy workloads without formal onboarding. This "easy way to get started" helps new users understand GKOP's comprehensive self-service capabilities.

Consumer engagement is fostered through forums, roadshows, and OpenShift chats, where developers can ask questions and provide feedback. Worldpay actively encourages developers to influence the platform's roadmap and contribute to initiatives, such as implementing OpenShift windows container support or improving continuous deployment models with GitOps.

The future of GKOP

Now 5 years into its operation, Worldpay continues to evolve GKOP. Future plans include further improving the developer journey by reducing blockers and manual processes, increasing adoption, and utilizing more OpenShift features. They also aim to add new features like OpenShift Service Mesh, database and Kafka messaging support, test coverage with continuous and synthetic testing, and an expanded platform status dashboard. Through it all, Worldpay's small but dedicated team remains passionate about their work with OpenShift and Kubernetes, fostering a collaborative and enjoyable environment.

Check out Worldpay’s talk from OpenShift Commons Gathering in London, earlier this year with Worldpay’s Matt Simons, Bernd Malmqvist, and Michal Umlawski.