Ford’s community-first approach to CI/CD

Ford Motor Company highly values open technologies and open standards, so when the company set its digital transformation strategy last decade, it looked to open source solutions. An early adopter of Kubernetes, with production workloads going online in 2017, Ford has since made open source technologies a cornerstone for the company’s cloud-native development initiatives. More than just consuming open source, Ford has taken the next step in its open source journey by contributing back to the open source upstream.

But first, here’s where Ford is right now…and where they are heading from Satish Puranam, Cloud Leader at Ford Motor Company:

“Ford’s comprehensive, company-wide digital transformation is helping deliver its best product line-up ever – including becoming #2 in EV sales to Tesla – and make it even more enjoyable to own a Ford vehicle with software-led “always-on” technologies, services and experiences. For example, Ford learns customer preferences from connected vehicle data and then can add new features or make other improvements by shipping software over-the-air to those products. Just consider F-150 and Mustang Mach-E customers who recently received Ford Power-Up software updates with our BlueCruise hands-free, eyes-on-the-road highway driving technology. One day, customers were driving their vehicles normally but the next day they could go hands-free. To support all of this, Ford is making it easier for software experts to focus on the best parts of the job while removing many of the obstacles that before prevented them from being more creative and productive.”

As Ford became more proficient in Kubernetes, the company looked to make its processes and workflows even more efficient, consistent and scalable. To do so, the company’s IT team adopted Red Hat OpenShift Pipelines, based on the Tekton project, to standardize their CI/CD pipeline - helping them build, test and deploy applications across clouds. Ford’s IT team took to Tekton and saw success almost immediately. This encouraged them to to go beyond just consuming the technology to contributing to the project itself, especially when it came to better solving enterprise use-cases. 

Looking to Red Hat as its guide, Ford learned the ropes of open source communities, enabling them to make contributions that ultimately helped shape the next release of the technology.

Success with Tekton

When Ford first started experimenting with CI/CD, their goal was process and component repeatability, along with delivering a unified experience across IT teams, even those with differing skill sets and tooling opinions. Its IT organization adopted Tekton in part because of the project’s composable nature. With no specific language to learn, teams were able to work in the ways that made them the most comfortable and relied on their existing knowledge. Additionally, because Tekton could extend across clouds, the IT organization liked that they wouldn’t be locked in with one vendor. At the production level, Ford consumes Tekton via OpenShift Pipelines for a fully supported, hardened version, a better fit for enterprise-wide deployments.

Tekton has since become an integral part of delivering IT infrastructure and applications services for Ford. All cloud provisioning is now done exclusively on Tekton by thousands of pipelines.

By implementing a CI/CD pipeline, Ford has been able to drop the delivery time of applications because of automation. The automation and repeatability have also led to less errors -- and the errors that are made are easier to track and fix. Additionally, Ford has also been able to release more often. With continuous delivery and deployment, new features and improvements get into the hands of end-users as quickly as possible.

Community collaboration

Building on its success using Tekton, Ford approached Red Hat to start a joint engineering initiative that would help the company’s IT team contribute back to the upstream Tekton project. Based on Ford’s long-time use of open-source technologies, the company wanted to take the next step with contributing to the community.

Together with Red Hat, they formed a virtual team to push forward enterprise use-cases within the Tekton project. By contributing in the community, Ford was able to influence the project in ways that would best meet the needs of enterprise users. Specifically, they were focused on shaping the technology to better meet growing large-scale needs in the areas of security and shortening the feedback cycle.

The initial use-case to address was to prevent unnecessary execution of the pipelineRun based on whether a file was changed or not. Previously, Ford’s workaround was to check if files have been changed inside a custom task, which was cumbersome. 

To solve this issue, Ford contributed Tekton Triggers in the upstream, a component to enrich the github payload of an incoming request with the list of changed files related to a pull_request or push event. Then this list would be verified using a CEL interceptor to halt processing and/or pass down to the pipelineRun where the file list can be used by a task.

Another Ford contribution to the Tekton upstream is an interceptor that blocks the execution of a pull request trigger from being executed unless invoked by an owner, with owners being defined in an OWNERS file at the root of the repository. With this contribution a pull request of pipeline is run only after it is reviewed by an owner of the module. This helps to reduce resources and improve namespace security by running only approved pipelines, as Ford has thousands of developers contributing to different modules. The changes resulted in a more efficient and security-focused process. Ford’s contributions were released in Tekton Triggers v0.23.0, enabling other enterprises to take advantage of these features. 

Mentorship from Red Hat

Ford and Red Hat are committed to driving innovation within Tekton that can be used by other enterprises - as having the strongest project and tools to work with ultimately leads to more innovation across IT. By working and learning with Red Hat, Ford was able to more quickly understand and engage with the community. Red Hat served as a mentor to Ford, showing them how to get started with contributing and the steps to take when first getting involved.

According to Satish Puranam, Cloud Leader at Ford Motor Company, “It is easy to consume open source, but it wasn’t until we got involved with the upstream community that we understood the blood, sweat and tears being put into these projects - with a sizeable number of individual contributors. Working with Red Hat we gained valuable mentorship which helped us learn how to get plugged into and collaborate with the broader open source community.”

Through the continued initiative Ford is planning to expand their work within open-source contributions and get involved with other Tekton related projects to accelerate their digital transformation and make its leading product line-up even better.