Adventures with Ansible: Lessons learned from real-world deployments

Ansible is a powerful IT automation tool. Like most powerful tools, it takes time to master, and you need to learn to use it well, and safely, in your environment.

Having used Ansible to automate deploying and managing enterprise applications, I’ve picked up a number of lessons that I consider best practices for automation with Ansible, and I would like to share them to help others. That is, after all, the open source way.

I’ve been doing enterprise software for a long time, well before Ansible even existed. I remember the days of quarterly production deployment that took place overnight with a team of people on call to get a release working. Deployments were expensive, complex, inconsistent, and frustrating on so many levels.

Getting away from that, and learning to automate software deployment more quickly and regularly, takes a lot of cultural and technical work. I’ll cover the cultural aspects in another post. Here I’d like to talk about a few of the practices I’ve picked up with Ansible to help make the most of its features and community.

Standards

Three things are important when it comes to automation:

1. standards,

2. standards, and…

3. oh, yeah, standards!

It might be easier and more fun to write a small playbook to do a particular admin task, but when you are developing hundreds of playbooks/roles for automated deployment of an entire software stack across many environments, it will hold a large amount of configuration elements and get quite complicated. Additionally your playbooks and roles will just grow if you do not reuse roles and develop standards. This Ansible playbook tutorial can help get you started on the path to success.

From day one, look for ways to standardize your playbooks, roles, and other practices with Ansible so that the team can reuse as much as possible and understand how things are put together.

A Galaxy far, far away

Ansible Galaxy, the Ansible community’s centralized repository of roles meant to be re-used by other Ansible users, is your friend. You should definitely reference Ansible Galaxy to see if someone else has already solved a problem that you need to tackle.

But! Exercise caution, too. Content from Galaxy can be reused, but it should be examined and understood before setting it loose in your environment. You could easily do a fair amount of harm to your environment in a very short time by using a role that does something unexpected.

Invent a solid inventory

Ansible can work with machines in your infrastructure at the same time by relying on its inventory to enumerate different groups of systems and define groups by workload or other characteristics. Ansible can even work with a dynamic inventory system for environments where the number of hosts varies due to demand.

Ansible Inventory can get very complex very fast when you are dealing with enterprise applications with a large amount of configuration information.

For example, we had a large number of XML files containing configuration information for our Middleware software. We created Jinja templates and moved the configuration elements from the XML files to our Ansible Inventory. The XML files were generated more easily and pushed to the target servers. Additionally, to deal with having multiple environments, we ended up using a custom-built Vars Plugin written in Python to load only specific configuration data within our custom inventory folder structure.

Hey cowboy, get ‘em handlers!

Handlers are a set of tasks that are run on notification, but only on notification, and only once per playbook. As an example, you can set up a playbook with handlers that are triggered from specific events, such as a restart of a service or application. Using handlers in an Ansible Role that manages a particular service can be great for creating clear and understandable tasks in the `handlers/main.yml` file that can be triggered by changes to the service.

When I first started working with Ansible, it didn’t have handlers, so I can tell you what life was like before and after the feature - don’t let it go to waste!

Idempotency - learn it, live it, love it

Idempotency is a way of life for IT automation. You need to understand it, and then live by it when you write automation code. So what is it? To quote the Ansible Glossary, an operation “is idempotent if the result of performing it once is exactly the same as the result of performing it repeatedly without any intervening actions.”

The upshot is that you can run your playbooks multiple times and the final result should remain the same - targeted servers will be in their “desired state”.  If your playbook fails on a small set of servers, you can fix the issue and then run your playbook again. Since the playbook is idempotent, targeted servers should be in the “desired state” and no other changes will occur.

What’s in a name?

Please, please, please always name your tasks. This seems really silly and almost like asking you to add comments to your code, but it is important to write a human-readable name for your task so that others can understand what should be happening in this task. Follow these simple rules:

1. Do not write traditional programming comments, for example “# this is a comment”, in your Ansible Playbooks. Remember Ansible is not a programming language, it’s an automation tool.

2. Instead, use the “Name” to describe the Play or Task using human readable explanations. Describe your intentions but try to refrain from having it too technical.

The biggest value with using the “Name” property on each Play or Task is that this information is displayed when running the Playbook. This helps with diagnosing Playbooks.  Comments, however, are not displayed in the run output.

My experience has been that playbooks/roles/tasks are written by many people in the company (let's start with Dev and Ops teams) and generally we need to know what our intentions are for each task.

Variable Naming and Precedence

Be careful when naming your variables within your Inventory, Playbooks and Ansible Roles. Remember, Ansible assigns every variable to a specific host (know as Host Variables). So be careful using generic names for your variables like ‘java_path: /my/path’. Chances are another application on the same host uses Java and has a different path.  Often times, multiple applications will exist on the same host but use different versions of Java that are located in different paths.

Choose distinct names for variables, prefix custom variables using your company name and make them human-readable too. For example, the ABC company created custom inventory variables for an application called AppOne that uses version Java 1.8.  

abc_appone_java_path: “/opt/appone/java”
abc_appone_java_version: “1.1”

Don’t be a copy-paster

Let me clarify this one. Do the right thing and create roles that can be reused, rather than copying and pasting things from one playbook to another. That way, you can edit the role once and dependencies will be automatically using the latest version, rather than hunting for outdated directives or suffering unexpected behavior when running a playbook months later.

Love thy environment

Use any IDE you want with Ansible. After all, it’s simply YAML syntax. However, you should have an IDE that can help you do your job with ease. Here’s some features I consider critical for an IDE when creating Ansible roles and playbooks:

• YAML syntax highlighting and lint support

• Python syntax highlighting and lint support - Ansible is written in Python, and believe me you will need to review the source code sometimes

• Integrated Version Control system

• JIRA integration - you can manage the changes to multiple tickets at one time

• Powerful Find & Replace - often you will need to change, remove or move that Ansible variable from multiple sources and you need to search and find all references

• Ruby language support - if you intend to also edit Vagrant files

• Groovy language support - if you intend to code your Jenkins jobs

By adopting these practices, you should be a step ahead and help avoid making the same mistakes that I have in using Ansible in enterprise software deployment and automation. You will be free to make new and interesting mistakes of your own! For more information on best practices with Ansible, you’ll want to consult the Ansible documentation’s best practices section, download the enterprise automation checklist, and keep an eye on the Red Hat Blog and Ansible Blog as well.