Getting started with Red Hat Satellite remote execution

Satellite 6.2 introduced an exciting new feature called remote execution, which allows administrators to remotely run an arbitrary command on Satellite clients quickly and easily. This post will cover an overview of how you can utilize remote execution.  

The Satellite documentation (for Satellite 6.2, and Satellite 6.3) walks through the process to set up remote execution, so that’s not covered in this post. We’ll pick up after remote execution has been enabled.  

Running Remote Execution Commands through the Satellite Web Interface

Once remote execution has been set up per the Satellite documentation, we are ready to run our first remote execution command. Start by logging in to the Satellite web interface, then going to the “Hosts” drop down, and selecting “All hosts”.

Next, select the hosts to run the remote command on, then click on the “Select Action” drop down, and select “Schedule Remote Job.”

On the next screen, make sure the “Job category” is set to “Commands”, and “Job template” is set to “Run Command - SSH Default”.   

We will set “command” to “uptime; hostname; whoami”. This will show the uptime of each host selected, as well as the hostname, and the name of the user the command is running as:

Next, click on “Submit” to run the command on each of the hosts. This will take us to a page that will show the status of the remote execution, and how many servers were able to successfully run the command:

In this case, the command was successfully run on all six hosts, and returned a zero return code. If the commands return a non-zero return code, it will display as failed (try running “/bin/false” as a command to see what a failure looks like).

We can see the actual output of the remote command by clicking on “Job Task”, then click “Sub tasks”, then click on each of the sub tasks to see the output for each respective host the command was run on:

Running Remote Execution Commands using the Hammer Command Line Interface

It is also possible to run remote execution commands from a shell on the Satellite server, using the hammer command.  

The basic syntax of running a remote command is:

# hammer job-invocation create --job-template "Run Command - SSH Default"   --search-query "query" --inputs command="command"

The “--search-query” option specifies which hosts the remote command should be run on. Here are a few examples:

• name = rhel7-test1.example.com        (run on this specific host, rhel7-test1.example.com)

• os = RedHat and os_major = 6            (run on RHEL 6 hosts)

• os = RedHat and os_major >= 6          (run on RHEL 6 and later hosts)

• hostgroup = development                     (run on the development hostgroup)

• registered_at > "4 July 2018"                (run on hosts registered after July 4, 2018)

The search query option is powerful and flexible, and I recommend referring to the Satellite documentation for more information on how to use this functionality.   

It is possible to interactively build search queries in the Satellite web interface by going to “Hosts”, “All Hosts”, and clicking on the “Filter” box, which will then bring up a list of fields that can be filtered on. It is also helpful to test search queries on the web interface to validate which servers are returned by a given search query.  

In addition, it is possible to save search queries through the Bookmark functionality so complex queries don’t need to be typed in every time they are used (the bookmark menu can be accessed by pressing the down arrow next to the Search button in the screenshot above).    

In this example, we will run the “df -m” command on the hosts within the RHEL7 hostgroup:

# hammer job-invocation create --job-template "Run Command - SSH Default"   --search-query "hostgroup = RHEL7" --inputs command="df -m"

The hammer command will return output specifying the job invocation ID number, how many subtasks were created, how many were successful, and how many failed:

Job invocation 31 created
[................................................................................] [100%]
4 task(s), 4 success, 0 fail

To see the output of the command on a given host, use this command:

# hammer job-invocation output --id id_number --host host

In our example, we know the job ID number was 31, so to see the output for the rhel7-test1.example.com server, we would run:  

​# hammer job-invocation output --id 31 --host rhel7-test1.example.com

Filesystem                      1M-blocks  Used Available Use% Mounted on
/dev/mapper/rhel_rhel7--test1-root  17394  1687 15708  10% /
devtmpfs                              967 0    967   0% /dev
tmpfs                                 979 0    979   0% /dev/shm
tmpfs                                 979 9    971   1% /run
tmpfs                                 979 0    979   0% /sys/fs/cgroup
/dev/vda1                            1014   173 842  18% /boot
tmpfs                                 196 0    196   0% /run/user/0
Exit status: 0

To see a list of which hosts the job ran on, run:

# hammer job-invocation info --id id_number

In our example, using ID number 31, the output is:

# hammer job-invocation info --id 31
ID:              31
Description:     Run df -m
Status:          succeeded
Success:         4
Failed:          0
Pending:         0
Total:           4
Start:           2018-07-06 21:53:30 UTC
Job Category:    Commands
Mode:          
Cron line:     
Recurring logic ID:
Hosts:         
- rhel7-test1.example.com
- rhel7-test3.example.com
- rhel7-test2.example.com
- openstack10.example.com

Using Search Queries for Remote Execution from the Satellite Web Interface

In the first Satellite web interface example, we simply manually selected which hosts we wanted to run the command on from the “Hosts”, “All Hosts” menu.   

It is also possible to use the powerful search query functionality to select which hosts to run the commands on from the Satellite Web Interface.   

To do this, from the Satellite Web Interface, go to “Hosts”, and select “Job templates”. Then click the “Run” button to the right of “Run Command - SSH Default”

From here, we can either use a previously saved search query bookmark, or we can specify a search query.  

Viewing Output for all Servers

With the Satellite web interface, it is necessary to click on each host to see the host’s output. Similarly, from the hammer command line, we need to run a command for each host to see each host’s output.

It is possible to create a simple bash loop to view the output from each host with a single command line. In this example, we will check the status of SELinux on all of the RHEL 7 servers:

# hammer job-invocation create --job-template "Run Command - SSH Default"   --search-query "hostgroup = RHEL7" --inputs command="getenforce"
Job invocation 32 created
[................................................................................] [100%]
4 task(s), 4 success, 0 fail

We then take the Job invocation ID returned (32), and run this one liner bash loop command:

# MY_ID=32; for host in `hammer job-invocation info --id $MY_ID | grep "^ - " | awk '{print $2}'`; do echo $host; hammer job-invocation output --id $MY_ID --host $host; echo;  done

rhel7-test1.example.com
Enforcing
Exit status: 0

rhel7-test3.example.com
Enforcing
Exit status: 0

rhel7-test2.example.com
Enforcing
Exit status: 0

openstack10.example.com
Enforcing
Exit status: 0

Based on this output, we can easily see that SELinux is enforcing on all hosts within the RHEL7 hostgroup.

It is also possible to kick off a remote execution from the Satellite web interface, and then use the Bash one-liner from the command line to view the output for all hosts. Simply kick off the remote execution from the Satellite web interface, then click the “Job Task” button on the status screen.  Then click the “Raw” tab, and make a note of the job invocation ID number (which is listed under “{"job_invocation"=>{"id"=>”). The final step is to run the one-liner bash loop from the Satellite servers command line:

# MY_ID=id_number; for host in `hammer job-invocation info --id $MY_ID | grep "^ - " | awk '{print $2}'`; do echo $host; hammer job-invocation output --id $MY_ID --host $host; echo;  done

Using and Creating Job Templates

All of the examples so far have been commands we specified using the “Run Command - SSH Default” job template, which allows any arbitrary command to be run on a remote host.

However, there are a number of job templates included with Satellite, and it is possible to create additional job templates.  

A job template is a powerful tool to make running remote commands consistent and easy to use, and makes it possible to add additional logic for handling things such as different versions of RHEL. To demonstrate what a job template is and how to use one, we will look at the “Service Action - SSH Default” job template included with Satellite. To view this job template, from the Satellite Web Interface, go to “Hosts”, click “Job templates”, and then click on “Service Action - SSH Default”.

On the Template tab, we can see the embedded ruby (ERB) code behind the template:

​<% if @host.operatingsystem.family == "Redhat" && @host.operatingsystem.major.to_i > 6 %>
systemctl <%= input("action") %> <%= input("service") %>
<% else %>
service <%= input("service") %> <%= input("action") %>
<% end -%>

From this, we can see that if the host is running a version of RHEL greater than version 6, it should run the “systemctl” command, followed by variables that will be replaced with the specified action and service name.

For all other hosts, it will run the “service” command, followed by variables that will be replaced with the specified service name and action.  

If we move over to the “Job” tab, we can see where the previously referenced <%= input("action") %> and <%= input("service") %> are specified. Here it is specified that the “action” can be restart, start, stop, or status, and the “service” can be any string that is specified.  

If we go back to “Hosts”, “Job Templates”, and then click “Run” to the right of “Service Action - SSH Default”, we can see what it looks like when we use a job template:

Just as is specified when we looked at this job template, it is prompting the user for the action to take (restart, start, stop, or status), and the name of the service. Once these are specified, the job template can be run and will take the specified action.  

Viewing Previously Run Remote Execution Tasks

It is possible to view a history of previously run remote execution tasks by accessing the “Monitor” menu in the Satellite web interface, and selecting “Jobs”.

This will display a list of previously run remote execution tasks, as well stats on the number of successes and failures, and how long ago the task was run:

It is also possible to initiate a remote execution task from this page by clicking on “Run Job” button.

From the command line, a history of previously run remote execution tasks can be viewed by running “hammer job-invocation list”.  

Final Thoughts

Satellite remote execution is a powerful feature that can help administrators manage their ever-growing environments. Remote execution is very flexible, and can be used from both the Satellite web Interface, and the hammer command line.  

To see a video demonstration of most of the topics covered in this post, see "Satellite Remote Execution" on the Red Hat Videos channel on YouTube.

If you’d like to see more about Red Hat Satellite, you can watch (after registration) the recording from our webinar, "Ensure Immediate Value in your Red Hat Satellite Upgrade."