Satellite and Ansible Tower are each powerful tools, and many customers utilize both of them. It is possible to integrate these tools and in part 1 of this series we covered how to configure Ansible Tower to pull a dynamic inventory of hosts from Satellite.
One of Satellite’s features is the ability to provision new hosts, while one of Ansible Tower’s main features is the ability to configure hosts. By integrating these tools, Red Hat Enterprise Linux (RHEL) hosts provisioned by Satellite can be configured to automatically make a provisioning callback to Ansible Tower. This provisioning callback functionality allows you to run an Ansible playbook on the new RHEL host so it can be configured using the Ansible Tower infrastructure. The end result is the ability to go into Satellite, provision a new host, and automatically obtain a configured host via Ansible Tower. This can save system administrators time and allow them to meet the needs of their organization.
One of the prerequisites for setting up provisioning callbacks is having Satellite configured as a dynamic inventory source within Ansible Tower so, if you haven't already, first follow the steps outlined in the previous blog post.
We must also have Satellite setup for provisioning, and it should be configured to use host groups when provisioning. If you don’t currently have Satellite setup for provisioning, refer to the Provisioning Guide documentation.
In this post, we will show how to configure Satellite and Ansible Tower to run a playbook after Satellite provisions hosts.
Overview
Automation tools, such as Ansible Tower, generally fall into one of two categories of operation: push or pull. In push systems, like Ansible Tower, the connection is initiated by the automation server to the hosts. In pull systems, hosts initiate the connection to the automation server.
To illustrate why we need provisioning callbacks, consider the following scenario: we have a playbook doing host configuration that runs daily at midnight. We provision several new servers with Satellite at 8:00 AM. Satellite has been configured as a dynamic inventory source for Ansible Tower, so the new hosts will be included in the inventory. However, the next scheduled run of the playbook isn’t until midnight, which is 16 hours later. Obviously, this is not an ideal situation as we would like the playbook to run immediately after the new host is provisioned.
Provisioning callbacks allow hosts to instruct Ansible Tower to run a playbook immediately. So as soon as a new host is built, it can request Ansible Tower run a playbook.
The high level overview of the process is as follows:
-
An Ansible Tower machine credential is created for the root user, with the same root password defined in the existing Satellite host group used for provisioning. Ansible Tower will use this credential to authenticate to the newly provisioned host.
-
The Ansible Tower job template is configured with provisioning callbacks enabled. Callbacks can be initiated by making a request to a URL with the host config key shown on the job template page.
-
The Satellite host group is configured with several parameters defining how the client can make the Ansible Tower provisioning callback request (Ansible Tower URL, the host config key, and the template ID).
-
A new host is provisioned by Satellite and the provisioning templates cause a
/etc/systemd/system/ansible-callback.service
file to be created on the new host (for RHEL 7 and RHEL 8 hosts, RHEL 6 uses a script). When the new host boots for the first time, this systemctl service file causes it to make a request to the Ansible Tower provisioning callback URL with the correct host config key and template ID number. -
Ansible Tower receives the provisioning callback request and validates the host config key is correct. If so, it initiates a run of the template on the new host and uses the root user credential to authenticate. The template is run on the host, which configures the host.
Configuring Ansible Tower for Provisioning Callback
We’ll start on the Ansible Tower server by creating a playbook that will set the contents of /etc/motd
, create a user, and install a package. This is the playbook that will be configured to run on newly provisioned hosts.
In most situations, we would want to use a source configuration management system to store our playbooks; however, for the sake of simplicity in this post, we will use a playbook stored locally on our Ansible Tower server. We will log in to the Ansible Tower server over SSH and create the /var/lib/awx/projects/provision
directory with the following command:
# mkdir /var/lib/awx/projects/provision
We’ll then create our playbook, named provision.yaml, within this directory with the following content:
--- - name: Provision new host hosts: all tasks: - name: Set content in /etc/motd copy: content: Authorized use only! dest: /etc/motd mode: 644 owner: root group: root - name: Create brian user account user: name: brian uid: 10000 state: present - name: Install tmux package yum: name: tmux state: present
We’ll then login to the Tower web interface and go to Credentials in the menu. Next, we’ll click on the green + (Plus) button to create a new credential, give it a name (in this example we use provisioning_root for the name), and set the Credential Type to Machine. We’ll enter root for the Username and set the Password to the same password Satellite is configured to use for new hosts in the Satellite host group (pull up the host group in the Satellite web interface and reference the Operating System tab). This credential will allow Ansible Tower to authenticate to newly provisioned hosts.
After this, we’ll go to Projects in the Ansible Tower web Interface and click on the green + (Plus) to create a new project. We’ll name the project provision, change the SCM Type to Manual, and select the provision Playbook Directory:
Then we’ll go to Templates in the Ansible Tower web interface, click on the green + (Plus), and select Job Template. We’ll name the template provision, for Inventory select our Satellite inventory source, select provision for the Project, provision.yaml for the Playbook, and provisioning_root for the Credential. We’ll also select the Allow Provisioning Callbacks checkbox and click the wand to generate a Host Config Key. The Host Config Key is used to ensure that not just anyone can initiate the provisioning callback. Ansible Tower will only run the provisioning callback if it is made with the correct Host Config Key.
At this point, we’ll want to make a note of the generated Host Config Key as well as the Template ID, which can be determined by looking at the URL of the Template:
https://tower.example.com/#/templates/job_template/11
In this example, the template has an ID of 11.
Configuring Satellite for Provisioning Callback
We’ll now move over to the Satellite web interface. Once logged in, go to the Configure menu and select Host Groups. Then select the existing host group used for provisioning; in this example, it is the RHEL 8 host group.
From the edit host group screen, select the Parameters tab, and create the following 4 new parameters:
ansible_host_config_key
, which should be set to the Host Config Key from our Ansible Tower Template.
ansible_job_template_id
, which should be set to the template ID that we gathered from the template URL earlier.
ansible_tower_fqdn
, which should be set to the fully qualified domain name of the Ansible Tower server.
ansible_tower_provisioning
, which should be set to true.
Provisioning a Host to Validate
The final step will be to provision a new host from Satellite and validate the provisioning callback causes our provision playbook to be run automatically on the new host.
From the Satellite web interface, we’ll go to the Hosts menu, select Create Host, and make sure to use the host group we just added the parameters to (in this example, the RHEL 8 host group).
Once the host is provisioned, we’ll validate the template successfully ran on the host. To do so, from the Ansible Tower web interface, go to Templates in the menu. The provision template should be listed, and have a green box listed next to it in the leftmost spot:
Click on the green box to see details of the run, which in this example, was successful. Also note that this job was limited to only run on the newly provisioned host. This is expected, as provisioning callbacks only run the template on the host that made the request.
We’ll also log in to the new host and validate the state of the system matches what is in the playbook:
[root@provision-test-rhel8 ~]# rpm -qa | grep tmux tmux-2.7-1.el8.x86_64 [root@provision-test-rhel8 ~]# id brian uid=10000(brian) gid=10000(brian) groups=10000(brian) [root@provision-test-rhel8 ~]# cat /etc/motd Authorized use only!
Troubleshooting
If the Ansible Tower template doesn’t automatically run on a newly provisioned host, several things can be checked to troubleshoot what went wrong.
The first step is to determine if Ansible Tower received the provisioning callback request. To determine this, go to Templates in the Ansible Tower menu, and see if there is a green or red box for the template indicating a successful or failed run, respectively.
If you see a red box, click on it to see more details on why it failed (which could be related to a syntax issue in the playbook or other issues).
If you see a green box, click on it to see additional information. The job might have reported as successful, but not actually run. For example, if the playbook’s hosts line specifies specific hosts and groups, and the newly provisioned host isn’t included, the job will report successful. However, if you look at the details it will show skipping: no hosts matched.
If you don’t see any boxes next to the template, start by checking the parameters defined in the Satellite host group. Verify that the Ansible Tower URL, the host config key, and the template ID are all correct.
Next, login to the newly provisioned host. If it is a RHEL 7 or RHEL 8 host, it should have a /etc/systemd/system/ansible-callback.service
file that was created when the system was provisioned. On RHEL 6 hosts, it should have a /root/ansible_provisioning_call.sh
file.
If you don’t see one of these files, validate that the ansible_tower_provisioning
host group parameter was set to true, and that the provisioning templates on the Satellite server haven’t been modified.
On the newly provisioned host, if we look at the contents of the /etc/systemd/system/ansible-callback.service
file, it will show the URL that the system is calling for the provisioning callback:
[root@provision-test-rhel8 ~]# cat /etc/systemd/system/ansible-callback.service [Unit] Description=Provisioning callback to Ansible Tower Wants=network-online.target After=network-online.target [Service] Type=oneshot ExecStart=/usr/bin/curl -k -s --data "host_config_key=aa5ebe82-491c-4fbb-bd36-a6657549451e" https://tower.example.com/api/v2/job_templates/11/callback/ ExecStartPost=/usr/bin/systemctl disable ansible-callback [Install] WantedBy=multi-user.target
We can manually run the curl
command listed, which should initiate the provisioning callback:
[root@provision-test-rhel8 ~]# /usr/bin/curl -k -s --data "host_config_key=aa5ebe82-491c-4fbb-bd36-a6657549451e" https://tower.example.com/api/v2/job_templates/11/callback/
If there the host config key is incorrect, an error message will be shown:
[root@provision-test-rhel8 ~]# /usr/bin/curl -k -s --data "host_config_key=wrong-key-here" https://tower.example.com/api/v2/job_templates/11/callback/ {"detail":"You do not have permission to perform this action."}
If the template ID is incorrect (it was changed from 11 to 43 in this example), it will report:
[root@provision-test-rhel8 ~]# /usr/bin/curl -k -s --data "host_config_key=wrong-key-here" https://tower.example.com/api/v2/job_templates/43/callback/ {"detail":"Not found."}
It can also be helpful to take the -s (silent mode) off the curl command, which will then show if the host name cannot be resolved (it was changed from ansible.example.com to ansible-tower.example.com, which is not a valid hostname in the environment):
[root@provision-test-rhel8 ~]# /usr/bin/curl -k --data "host_config_key=wrong-key-here" https://ansible-tower.example.com/api/v2/job_templates/11/callback/ curl: (6) Could not resolve host: ansible-tower.example.com
It can also be helpful to review the /var/log/tower/tower.log
file on the Ansible Tower server, which might indicate the cause of the issue.
When an invalid host config key is used, this was logged:
2019-11-19 23:19:17,371 WARNING awx.api.generics status 403 received by user AnonymousUser attempting to access /api/v2/job_templates/11/callback/ from 192.168.0.138
When an invalid template ID was used, this was logged:
2019-11-19 23:19:49,093 WARNING awx.api.generics status 404 received by user AnonymousUser attempting to access /api/v2/job_templates/43/callback/ from 192.168.0.138 2019-11-19 23:19:49,095 WARNING django.request Not Found: /api/v2/job_templates/43/callback/
Summary and Closing
In conclusion, integrating Satellite and Ansible Tower together is a powerful way to streamline the new host provisioning and configuration process. This can allow system administrators to save time and respond to organizational needs faster.
About the author
Brian Smith is a Product Manager at Red Hat focused on RHEL automation and management. He has been at Red Hat since 2018, previously working with Public Sector customers as a Technical Account Manager (TAM).
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit