Designing Golden Paths

Introduction

Golden Paths are a fundamental ingredient of well architected Internal Developer Platforms (IDP). Spotify uses the term “Golden Path”, but that concept has other monikers. For example Netflix calls the same idea “Paved Road.”

The general notion behind this concept is that the platform offers some form of pre-architected and supported approaches to building and deploying a particular piece of software. If a team can “stay” in the path (because the requirements of what they are building allow them to), then they get a supported road to production without having to learn all the details of the technology used to create that road.

This approach accelerates typical application development use cases (onboarding of a new team member, onboarding of a new application), and at the same time, injects the best practices that have been learned in the past. The more sophisticated Golden Paths are, the more they will be adopted, providing, as a result, more uniformity of configuration and behavior across the application portfolio.

Golden Paths

So, what is in a Golden Path? It depends on what is being deployed. In the case of an internally developed application, a Golden Path will have at a minimum, the following ingredients:

• A repository template to get started with. This could be a simple hello world application, but with the key resources and the configurations that allows any developer to get started quickly with the company IDEs and all the settings that embodies the best practices for code development.
• A pipeline that can take the aforementioned repository, build it, and push the resulting artifacts all the way to production. The pipeline will have all of the steps that are deemed necessary for the organization to trust that code being deployed to production.
• A set of manifests to allow for deploying the application. These could be Helm charts or kustomize configurations in the Kubernetes world, or other forms configuration descriptions, if not using Kubernetes.
• Observability capability baked in. Observability (logs, traces, metrics and alerts) is a foundational capability that should be provided by the platform. Exactly what to observe in a specific application has to be defined as part of the deployment manifest. A Golden Path should provide reasonable defaults for observability settings.

Golden paths should have the following characteristics:

1. They are optional. Golden Paths should not be the only way things are done in an IDP. There should be room for walking outside of the beaten paths. That is necessary to allow and foster innovation. IDP teams should take the responsibility of observing the cases where Golden Paths are not used, to identify situations where new Golden Paths might be necessary (for example, because multiple teams are starting to use and deploy a different technology or leverage different approaches).
2. Golden Paths’ abstractions should be transparent. Golden paths create an abstraction that allows for varying approaches without having to learn the underlying technology. A UI can be placed in front of the Golden Path to make the user experience for developers pleasant by reducing the “getting started” use case to only a few clicks (Backstage is a popular tool for this purpose). But, Golden Paths’ abstractions should be transparent and if a developer wanted to understand what happens behind the curtains, it should not only be possible for them to do so, it should also be easy. In a shift-left scenario where developer teams are responsible for the end-to-end processes behind the code they produce (you build it, you run it), this approach inspires trust in the platform.
3. Golden paths should be extensible. Of course, Golden Paths are configurable as they are essentially a set of templates with parameters. But, can they also be extended? There should be flexibility to allow for this type of situation. For example, let’s consider the pipeline. Any Golden Path will contain a pipeline that describes the process for pushing code to production. But, can a developer team add additional capabilities, such as another pipeline for handling some ancillary automation? Things like: backup/restore of databases, reload/refresh of test data, general cleanups, launching very specific tests that don’t have to run all the time?  A Golden Path should not preclude these use cases. Similarly for the deployment manifests, it should be possible to augment them with additional manifests. Extensibility, however, should be used with caution. The way a Golden Path is augmented should not conflict with the Golden Path itself, otherwise the team maintaining the Golden Path cannot provide support.

Golden Path Maturity Model

In my experience working with customers, I find that three levels of maturity for Golden Paths exist:

1. No Golden Path
2. Clone and forget
3. Golden paths as products.

No Golden Path

At this level of maturity, the platform is architected enough to be able to build and run applications, but no Golden Path exists. This is not a trivial level given that the actual ability to build and run applications is not a simple feat and it represents a prerequisite to then start creating Golden Paths. At this level, the platform should have the following capabilities:

1. A method for onboarding teams and applications. The more automated the better. We should strive to minimize the number of additional processes (tickets in many cases) that are necessary to be onboarded. In theory, there should be only one form to submit.
2. A way to create and run a CI pipeline. A pipeline tool has been installed and developers are empowered to write and run their own pipelines.
3. A way to deploy the application (CD). Ideally this would use a GitOps based approach. Again, developers are empowered to create their own manifests and the platform offers some ability to consume and reconcile them.
4. A way to provision credentials for all required integrations. Typical integration for a CI/CD pipeline include: Version Control System, image registry, static code analysis tool, vulnerability scanning tool, etc.
   Typical integration needs for applications include: databases, messaging, caches, invocations of remote services, etc.

At this level of maturity, every development team will have to figure out how to write the pipeline and the deployment manifests. Duplication of efforts and snowflaking are inevitable, but helps establish patterns that can be identified and used later in the process.

Clone and forget

At this level of maturity, templates exist for all of the artifacts of a Golden Path. Typical examples include a repository template, a pipeline template and templated manifests. At onboarding time, the developer clones and runs the templates with the specific parameters of the application they are onboarding, and obtains a ready to use set of artifacts. From then on, the developer team is the owner of those artifacts. This creates a bifurcation between the evolution of the Golden Path templates and the evolution of that particular instance. As the drift becomes wider, it gets increasingly harder to take updates from the Golden Path templates (typically teams have to do manual merge operations). Ironically, this situation hurts the early adopters of the platform more because they experience the largest drifts.

Golden Path as Product

At this maturity level, Golden paths are treated as products. Golden Paths will have releases with new features and bug fixes. Unavoidably backward-incompatible releases will happen (such as when a new pipeline step is introduced which requires specific files to be present within the code repository). Developer teams that are using a particular Golden Path will have an opportunity to take the new release when they can afford to do so. One or more Platform teams may be dedicated to maintaining the Golden Paths and treating them as products. This includes, as said, cutting releases, but also creating documentation, release notes, and helping developer teams adopt the Golden Paths. Golden Paths exist to help and serve developers, so it’s important to establish feedback loops with the developer community and make sure that they evolve based upon ongoing requirements.

An Example

Here is an example of how to build a simple Golden Path. To make this example concrete, we have to ground in some technology choice. In other words, we need to have some assumption on what the platform does for us (and it must be at the “No Golden Path” maturity level at least).

For the sake of this example, let’s assume we have the following set up:

We have three Kubernetes clusters dedicated to running the pipelines (build), QA validations (Preprod) and production (Prod). We adopt GitOps and each cluster is configured via Argo CD. Tekton for running pipelines is installed in the build cluster. A GitOps repository exists for each tenant. A folder convention exists such that when tenants create files in the GitOps repository, it’s understood in which cluster resources will be created (the exact convention is outside the scope of this article).

Each tenant gets some namespace in each of the clusters at onboarding and the Argo CD instances are configured to deploy manifests only in those namespaces (out of the tenant GitOps repository). Credentials and secrets are available in each tenant namespace, based on what the namespace is supposed to do.

Now, let’s assume that we onboard an application using a Golden Path. Here is what it might look like:

A new code repository for the new application is generated starting from a code repository template. A pipeline and application wrapper charts are deployed to the tenant GitOps repository. These charts are essentially empty, but they refer to the golden charts using the helm dependency management feature. Here is relevant section for configuring a dependency in a Chart.yaml file:

dependencies:
 - name: app-golden-chart
   version: "0.1.15"
   repository: https://myorg.github.io/golden-charts

When all is in place, the Tekton manifests will be deployed from the tenant GitOps repository and it will start building the application code. The application manifest will also be deployed and when the application image is ready, it will start running in the namespaces where it’s promoted to.

It’s important to note that the golden chart will evolve over time and that with this approach, the application team can take updates as they see fit, simply by incrementing the version of the imported golden chart in the wrapper chart.

Other technologies may have features to allow referencing a reusable, centrally managed artifacts, for example GitHub actions uses reusable workflows and GitLab ci/cd uses the include directive.

Once one has a development platform and Golden Paths in place, a UI can be set up in front of them, to improve the user experience for the developers. Tools like Backstage.

Conclusion

In this post, we discussed some of the main characteristics that a Golden Path should have. We introduced a simple maturity model for Golden Paths and we show an example of how to implement one. Designing good Golden Paths is a prerequisite to being successful with an IDP implementation. Although the IDP concept is still in its early days, the best practice seems to be to get a Golden Path right (i.e. good enough that developer teams will want to adopt it), before starting to build other Golden Paths. Getting started with your first Golden Path is the first step and with the approaches described in this article, it is the hope that they can be used to help increase the likelihood for success.