DWP advances automation with Ansible Automation Platform

Around 20 million claimants and customers across the UK depend on the Department for Work and Pensions (DWP) for financial security. Its Benefit Payment Systems (BPS) team ensures people receive the benefits they need. Red Hat Ansible Automation Platform deploys development, test, pre-production, and production environments on premise and cloud. The team can deploy environments more consistently and more frequently.

Benefits:

• Cut deployment times from 50 minutes to 10
• Enabled consistent deployment across on premise and cloud environments
• Simplified troubleshooting with human-readable scripts

The UK’s Department for Work and Pensions (DWP) administers the State Pension and a range of working-age, disability, and ill health benefits to more than 20 million claimants and customers. It aims to improve people’s day-to-day lives and help them build financial resilience and a more secure and prosperous future. The DWP’s Benefit Payment Systems (BPS) team is responsible for ensuring people receive the benefits they need to survive, and, as such, it is seen as critical infrastructure for the country.

When BPS embarked on its automation journey in 2016, it adopted a deployment orchestration tool to support its on premise environments. Around 10 engineers created the automation code base for the tool. The team had to run the tool several times for each environment deployment. “You had to go onto the virtual machine instance to execute the tool over and over again; it would take around 50 minutes,” said Mark Laverick, Lead DevOps Engineer at Business Payment Systems, Department for Work and Pensions. “Sometimes the deployment would appear to be successful when actually it had some functionality that wasn’t working. And it took quite a lot of investigation to surface those issues.”

“The code was complex, which made it hard to identify where any issues were,” said Robert Brooks, Lead DevOps Engineer at Business Payment Systems, Department for Work and Pensions. “And we couldn’t get the support we needed from the supplier.”

Lack of support was primarily due to the tool reaching end of life. “It wasn’t able to meet our needs when we started building out more environments in the Amazon Web Services [AWS] cloud,” said Brad Clegg, Technical Delivery Lead at Business Payment Systems, Department for Work and Pensions.

Ansible Automation Platform emerged as the top choice for BPS for a number of reasons. First, it was already in use by the DWP Central IT department. Second, it offered a strategic advantage beyond solving the immediate issue. Not only would it address the current problem, but it would also empower the team to upskill their engineers in a modern approach, enabling them to tackle future challenges more effectively.

“Ansible Automation Platform was—and still is—the DWP’s strategic tool of choice for automating deployments,” said Brooks. “And while we didn’t have a lot of Ansible expertise at that point, it looked to be the right tool for us.” Unlike the legacy tool, Ansible Automation Platform would enable monitoring and provide an automation dashboard to refer not only to the result of an execution of Ansible roles but also to runtime metrics and other information.

The DWP’s Service Liability Engineering (SLE) team was also looking to adopt Ansible Automation Platform, so the two teams got together to move forward with the Red Hat automation platform.

The migration itself involved building an environment with feature parity on AWS that mirrored the functionality previously provided by their prior configuration management tool. This “greenfield” approach ensured a clean migration before transitioning back to the on-site production environment. Overall, the project not only achieved successful migration but also set the foundation for future automation success.

As DWP is essentially a financial institution, handling hundreds of millions in transactions, it also has Financial Operational (FinOps) goals to meet. To meet these goals and achieve immutable infrastructure within the non-production DWP environment, the team leverages two Ansible Automation Platform instances in the AWS cloud.

These instances are configured for daily self-rebuilds using automation scripts. This means the instances are terminated each evening and rebuilt fresh every morning. This “spin-up, spin-down” approach ensures a constantly up-to-date and immutable infrastructure for its non-production environment. “The Red Hat experts were invaluable,” said Brooks. “They helped us build Ansible Automation Platform in a mutable fashion as infrastructure as code (IaC) and answered general questions when necessary, about the platform and Ansible Playbooks.”

BPS built a new codebase from the ground up. It not only structured the code optimally by making it more modular but also made it as agnostic as possible to act as a single codebase for all environments, whether on premise or in the cloud.

Current Ansible Automation Platform use cases predominantly involve deploying releases of development, test, pre-production, and production environments, both on premise and in the cloud. BPS also uses it to deploy tooling such as Jenkins and build non-Linux images.

Cut deployment times from 50 minutes to 10

With Ansible Automation Platform, BPS has reduced deployment times from 50 minutes to 10. “We no longer have to run several iterations,” said Laverick. “We can see if a deployment has worked instantly through Ansible Automation Platform.” Historically, environments were provisioned locally from a user’s laptop. There was no visibility of whether it worked for other users. The team wouldn’t know any issues until an end user raised them.”

“We used to have a mixture of in-house and external consultants managing releases,” said Clegg. “Now it’s a single person and a single click of a button.”

Enabled consistent deployment across on premise and cloud environments

Thanks to Ansible Automation Platform, separate codebases for different environments are no longer necessary. “Under our previous tool, we had three separate deployment domains: cloud, pre-production, and production,” said Clegg. With Ansible Automation Platform, we have a single source of truth. Having a single playbook serving all three domains ensures consistency and gives us massive efficiency savings.”

The environments deployed are also more stable, allowing the team to deploy releases more frequently. “Ansible Automation Platform means our end users—the development and test teams—are getting new functionality and defect fixes rolled out much more frequently,” said Brooks.

Users were previously using long-standing instances of environments: “Having Ansible Automation Platform, we can deploy instances and recreate a brand new environment with confidence that the Ansible code will build it to a known configuration,” said Laverick.

Simplified troubleshooting with human-readable scripts

“Ansible Automation Platform is much more linear than our previous tool, which makes troubleshooting much easier,” said Brooks. “Everyone on the team can work with the tool, create a new playbook or role, and execute it.

”The previous tool was so complex that only two BPS resources were skilled in maintaining the codebase and troubleshooting issues. “The code for our previous tool was a web of different events and tasks,” said Laverick. “The code in Ansible playbooks is structured in a way that is human-readable. You can look at a playbook and completely understand what it’s doing.”

The success BPS has enjoyed with Ansible Automation Platform has unlocked its automation potential. The next steps in its roadmap include expanding orchestration within deployments, enabling a user-service deployment model, and integrating with APIs (Application Programmable Interfaces) and GitLab Pipelines.

“Ansible Automation Platform allows us to abstract a lot of complexity away from our users and empower them with self-service, so they’re no longer reliant on our technical teams,” said Brooks. “We’re also integrating a dedicated dashboard into our automation solution to help us drive our deployments.” In addition, BPS is exploring how it can take advantage of the event-driven automation offered by Ansible Automation Platform to enable self-healing in its environments.

There are also plans to migrate Ansible Automation Platform to a shared cloud service so all DWP engineering teams can take advantage of Ansible Automation Platform’s automation capabilities. Within DWP Digital, the HCS Automation Team currently manages their on premise Ansible Automation Platform. This same team will lead the development of a new shared Ansible Automation Platform environment hosted in AWS cloud.

“Ansible Automation Platform allows us to deploy environments consistently; it is highly repeatable, and highly automated and orchestrated – and we can deploy environments much faster than we could previously,” said Brooks. “It is an essential tool for not only our team but also the department.”