Xylem boosts datacenter engineer productivity with automation

Global water technology leader, Xylem, helps customers solve the world’s toughest water challenges across utility, industrial, commercial, and residential markets worldwide with innovative solutions. One such solution is the Regional Network Interface app, which utility companies use to manage water meters. Operational tasks, such as provisioning this app securely in the private datacenter, were manual and time-consuming. Xylem introduced Red Hat Ansible Automation Platform and Event-Driven Ansible, writing playbooks to automate security scans, storage tasks, and more. Now, engineers are more productive, workflows are standardized, and the company has repeatable playbooks to optimize more backend processes.

Benefits:

• Delivered time savings and reduced operational cost
• Developed standardized workflows
• Improved engineer satisfaction

Xylem is a global water solutions provider headquartered in Washington, DC. It offers water services to utilities, industrial, commercial, and residential markets, as well as providing hardware and software to improve water management.

The company’s software division is responsible for supporting an app called Regional Network Interface™ (RNI). This is hosted by Xylem and used by utility companies to manage water meters across municipalities.

“RNI instances are custom built by our network operations center. They’re tested, scanned by our security team, and hosted on virtual machines in our private datacenter,” said Michael Johnson, Principal Automation Engineer, Xylem.

These security scans, reports, and clean-ups were handled manually, which was time-consuming and could take several weeks. When a virtual machine was ready to deploy, the team opened a Jira ticket to request a security scan. The ticket went into a queue and generated an email notification. It was then assigned to a security engineer and addressed, which involved opening another tool, creating an asset using the virtual machine’s IP address, and starting a scan.

Once the scan was complete, the engineer checked the list of vulnerabilities, requested a fix, and repeated the whole process until all issues had been addressed. To improve operational efficiency, Xylem wanted to start automating datacenter processes. These security scans were identified as a process that would benefit from automation.

“Our long-term goal is to automate everything and empower our datacenter team to be more efficient,” said Johnson. “There’s room for error when processes are driven by a command line interface (CLI) and replacing them with playbooks will help to work in a more consistent way.”

Xylem introduced Red Hat Ansible Automation Platform into its private datacenter. This unified platform brings security features, integration, and flexibility into one place to automate and orchestrate workflows at scale.

“Automation was new to most of our datacenter teams. The presentation layer in Ansible Automation Platform was really helpful to make them feel more comfortable,” said Johnson. “People worry about triggering a playbook when they don’t know who wrote it or what it will do. I made diagrams to visualize the workflow, and the platform’s audit trail gives them clear visibility of every stage of the automation.”

Xylem used Ansible Automation Platform and the Event-Driven Ansible feature, which delivers advanced event-handling capabilities, to automate security scans. These are triggered when a user opens a Jira ticket. Event-Driven Ansible receives the payload via a webhook and initiates an API call to Ansible Automation Platform. This executes the next steps in the process and generates notifications to tell users when they have an action, and to update them when a scan is taking place.

While users have visibility of the process and remain in control, time-consuming tasks are automated, which speeds up the process considerably. Xylem also built safeguards into the playbook to prevent issues such as someone accidentally triggering too many scans in one go. “Certain conditions have to be met in order to trigger the next step,” said Johnson.

If the scan finds vulnerabilities, Xylem’s security team is notified but only the engineer to whom the ticket is assigned can trigger a rescan, which eliminates the risk of unauthorized actions. When issues have been fixed, simply adding a comment to the ticket enables the virtual machine to rescan.

This is just one example of automation at Xylem, which also spent six months working with the storage team to automate more than 30 manual tasks. “The storage team had the least amount of experience with automation and the biggest need for it,” said Johnson. “They had a lot of process documentation, which helped me to understand where we could automate things and move quickly.”

For example, the backup team can now pull automated reports on backups. The playbook took just 6 hours to write and will save 1 hour per week for its main report, while automated health-check reports are now delivered daily – and every 4 hours if an error is found. This saves around 2 hours a day compared to the previous manual process.

Automations are also in place to manage virtual machines, from provisioning and storage requirements to networking and ongoing maintenance. Xylem is currently working on auto-remediation for common alerts such as monitoring virtual machines, CPU, and disc alerts.

Delivered time savings and reduced operational costs

Every process automated with Red Hat Ansible Platform saves the datacenter team time, which helps Xylem to reduce operational costs while speeding up provisioning new RNIs to customers.

In another use case, storage reclaims, Ansible templates have to date saved over 40 hours and eliminated costly service outages.

“Every ticket for the storage team is different, and Red Hat Ansible Automation Platform saves time on all of them,” said Johnson. “It feels great knowing I can write a playbook to help my colleagues be more productive.”

Developed standardized workflows

Replacing manual processes with standardized workflows significantly removes almost all risk of human error, makes it easier to establish best practices across teams, and accelerates processes by at least 25%. Automation also improves accuracy and helps to optimize backend systems.

“Maintenance workflows are repeatable across operations. Storage automation can be reused in different teams, for example,” said Johnson. “We’ve got building blocks in Ansible Automation Platform that I can adapt for different personas. Everyone benefits from automation.”

Improved engineer satisfaction

Automating time-consuming tasks has freed up the private datacenter team to focus on higher value work, which is more satisfying. It also reduced overtime, and engineers have a better work-life balance.

“With Ansible Automation Platform, my colleagues can focus on strategic initiatives instead of firefighting all the time. They love having more time for exciting tasks now that the dull, repetitive work is automated,” said Johnson.

And by improving staff satisfaction, Xylem can recruit and retain the best talent.

Xylem may be early in its automation journey, but it’s already seeing great results. That includes a culture shift. “We’ve got our engineers excited about automation. Now, when we’re evaluating new tools for our environment, we always look at whether they can be automated. That shift in thinking will snowball across the organization and help with our long-term automation adoption strategy,” said Johnson.

Next, Xylem plans to automate patching, which will standardize 6 disparate ways of working into one workflow. It also has a long-term plan to automate disaster recovery, which is extremely complex. “Automating disaster recovery will be a huge win for us. Not only will it save time and money, but it will also really impress the rest of the organization,” said Johnson.