Skip to main content

How to customize Grafana dashboards using Ansible

Learn how to use a custom JSON data source, Mockoon, FastApi, and Prometheus to customize your Grafana dashboard.
Image
close-up view of various graphs on a computer screen

You probably know that Grafana has excellent integration with Prometheus exporter. In fact, it comes with several prebuilt dashboards that can collect and display many types of metrics.

Now imagine that you also have your hosts organized in logical groups, and you have a nice Ansible inventory file where you list your hosts:

---
# Inventory host for a fictional network for the Nunez Barrios family
---
all:
  children:
    linux:
      hosts:
        macmini2:
        raspberrypi:
        dmaf5:
        mac-pro-1-1:
      vars:
        description: Linux servers for the Nunez Barrios family
    windows:
      hosts:
        windows01:
      vars:
        description: Windows Desktops/ laptops for the Nunez Barrios family
    osx:
      hosts:
        veronicalaptop
      vars:
        description: OSX Desktops/ laptops for the Nunez Barrios family
    pi:
      hosts:
        raspberrypi
      vars:
        description: Raspberry PI 4 servers for Nunez Barrios family

The Ansible playbooks will get access to this inventory like this:

$ ansible-inventory --inventory ~/grafana/Dashboards/hosts.yaml --list
{
    "_meta": {
        "hostvars": {
            "dmaf5": {
                "description": "Linux servers for the Nunez Barrios family"
            },
            "mac-pro-1-1": {
                "description": "Linux servers for the Nunez Barrios family"
            },
            "macmini2": {
                "description": "Linux servers for the Nunez Barrios family"
            },
            "raspberrypi": {
                "description": "Raspberry PI 4 servers for Nunez Barrios family"
            },
            "veronicalaptop": {
                "description": "OSX Desktops/ laptops for the Nunez Barrios family"
            },
            "windows01": {
                "description": "Windows Desktops/ laptops for the Nunez Barrios family"
            }
        }
    },
    "all": {
        "children": [
            "linux",
            "osx",
            "pi",
            "ungrouped",
            "windows"
        ]
    },
    "linux": {
        "hosts": [
            "dmaf5",
            "mac-pro-1-1",
            "macmini2",
            "raspberrypi"
        ]
    },
    "osx": {
        "hosts": [
            "veronicalaptop"
        ]
    },
    "pi": {
        "hosts": [
            "raspberrypi"
        ]
    },
    "windows": {
        "hosts": [
            "windows01"
        ]
    }
}

Normally, Ansible inventory files are well maintained and kept on a source-control repository like Git. Sometimes, sysadmins generate them dynamically. In this tutorial, I'll show you how to filter hosts by group in your Grafana dashboard using variables. I will also show how to populate those variables from the Ansible host inventory file (taking advantage of the groups).

By the time you complete this tutorial, you will have learned:

  • How to install plugins in Grafana running inside a Podman container.
  • How to create mock REST API endpoints to simulate a web service for quick prototyping using Mockoon.
  • How to connect the mock REST API with Grafana to populate missing values on a panel that uses Prometheus data for monitoring.
  • How to write a simple web service to provide a static host inventory file through a REST endpoint to Grafana.

You will need the following resources for this tutorial:

I'll begin with how to connect an Ansible inventory file with Grafana.

[ Get started with IT automation with the Ansible Automation Platform beginner's guide. ]

Choose a JSON data source: Pick your poison

There are several choices for the JSON data source. I will mention three of them and why I choose the one I did.

  • Grafana Simple JSON is the official plugin, but it is deprecated (the last update was at least 16 months ago on GitHub). Stay away from it.
  • simpod-json-data source is very complete and easy to use. The documentation is also well written and is actively maintained. However, I decided against it because:
    • It will force you to generate the JSON with a well-defined structure that the plugin understands. This is not an issue for this demonstration, as I will craft the resulting JSON from scratch, but if you have an existing web service exporting JSON, this is less than convenient.
    • The endpoint that shows the data (/query) requires you to use the POST method (so you can pass search arguments). Not a big deal, but if you feel more at home using GET with query arguments, you are out of luck.
  • JSON API is my favorite because:
    • You can use an existing REST endpoint and get the fields you want using JSONPath expressions. You can also define an alias for the retrieved values.
    • It is well documented and maintained.
    • Installation is very easy.

[ Want to test your sysadmin skills? Take a skills assessment today. ]

For the Grafana installation, install the JSON API:

$ podman exec --interactive --tty grafana /bin/bash
I have no name!@ae23d7e1123e:grafana-cli plugins install marcusolsson-json-datasource
✔ Downloaded marcusolsson-json-datasource v1.3.1 zip successfully
Please restart Grafana after installing plugins. Refer to Grafana documentation for instructions if necessary.
exit
podman restart grafana

Now Grafana is now ready to pull data in JSON format, which you will use to customize the dashboard.

Working in reverse: Define how the REST-API will look using a mock

Before you start coding, you need an idea of what the API to be consumed by the JSON API plugin should look like. The following must be true:

  1. GET / with 200 status code response. Use this for a "test connection" on the data source config page.
  2. GET /search, which you will use to filter servers by group.
  3. GET /query/{ansiblegroup}/, which you will use to return the list of all the available groups when searching by group.

So before writing any code to serve the Ansible inventory file contents, put together some fake data with the Grafana data source.

Faking it until it works using Mockoon

The definition of a mock is to mimic or resemble closely.

And that is exactly what you will do. Create a fake REST web service with a JSON good enough to connect to Grafana to simulate the web service. Once you're satisfied with the results, you can invest time writing the real REST endpoint.

First, install and run the Mockoon GUI:

$ sudo dnf install -y https://github.com/mockoon/mockoon/releases/download/v1.20.0/mockoon-1.20.0.rpm
...
Installed:
  mockoon-1.20.0-1.x86_64                                                                                                                                                                                                                                
Complete!
mockoon

It will look like something like this:

Image
Mockoon GUI
(Jose Vicente Nunez, CC BY-SA 4.0)

After experimenting with the GUI and following the documentation, create a REST mock:

Image
mock REST
(Jose Vicente Nunez, CC BY-SA 4.0)

Unsurprisingly, it looks a lot like the Ansible inventory file converted to JSON format. That is the whole point of this exercise.

Here is how it looks when you run queries against it using curl:

[ Download the curl command cheat sheet. ]

The next step is to construct the dashboard with data coming from Prometheus. First, create a new data source using the JSON API:

Image
JSON API datasource
(Jose Vicente Nunez, CC BY-SA 4.0)

Next, create a dashboard. In this example, it will monitor available RAM in kilobytes over time:

Image
Dashboard displays available memory
(Jose Vicente Nunez, CC BY-SA 4.0)

Now add a variable that will hold the groups and the hosts. Also, you will have to "enrich" the name of the machine a little to match the label used by the Prometheus node. (Hint: A tool like jsonpath.com will help ensure the JSONPath expression is correct. I found the inline editor to be picky sometimes.)

You need two query variables. Here are the expressions:

  1. Get the list of all the groups: jsonpath $.all.children[*]. It will return something like json ["linux", "osx", "pi", "ungrouped", "windows" ]. Save a single result, like the first group, on a variable called group.
  2. Then using that group variable (assume the members are linux), you can query the list of hosts. For example to get all the linux machines, json $.linux.hosts[*] will display something like json [ "dmaf5", "mac-pro-1-1", "macmini2", "raspberrypi"].
  3. But you need to make the group generic. Use the Grafana variables again, and the JSONPath becomes json $.linux.${group}[*].
  4. This displays the list of hosts by group.

If you notice, Prometheus returns either a job (node-exporter) or an instance (raspberrypi:9100) for a given host. You can enrich the host list to make it look like a Prometheus instance by passing an extra argument to the query (enrich=true). It returned host like raspberrypi:9100 instead of raspberrypi.

[ Learn how to manage your Linux environment for success. ]

Not so fast: You're not getting any data for your variable!

You can try to get the list of all the available groups in the inventory from a JSON fragment like this:

{    
  "all": {
        "children": [
            "linux",
            "osx",
            "pi",
            "ungrouped",
            "windows"
        ]
    }
}
Image
Empty Grafana query
(Jose Vicente Nunez, CC BY-SA 4.0)

But the variable creation doesn't work due to a bug involving parsing the nested children tag.

The JSON path renders the array-like results you're looking for, so what is going on here? (I'm using the Python jsonpath module to illustrate.)

(inventory)$ python
Python 3.9.9 (main, Nov 19 2021, 00:00:00) 
[GCC 10.3.1 20210422 (Red Hat 10.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from jsonpath import JSONPath
>>> import json
>>> data=json.loads('''{    
...   "all": {
...         "children": [
...             "linux",
...             "osx",
...             "pi",
...             "ungrouped",
...             "windows"
...         ]
...     }
... }''')
>>> query="$.all.children[*]"
>>> JSONPath(query).parse(data)
['linux', 'osx', 'pi', 'ungrouped', 'windows']

It is probably a bug on the plugin. To prove it, change the structure returned with Mockoon to remove the unnecessary children tag:

{    
  "all": [
            "linux",
            "osx",
            "pi",
            "ungrouped",
            "windows"
        ]
}

Try Grafana again, and you should see that the structure is now parsed properly:

Image
Grafana query for valid groups
(Jose Vicente Nunez, CC BY-SA 4.0)

That's an example of the power of working with a Mock REST API for rapid prototyping.

Once you have the group, you can write a more intuitive REST endpoint that uses the group you selected in the previous step. For example, if you call /query/linux, you could get:

[ "dmaf5:9100", "mac-pro-1-1:9100", "macmini2:9100", "raspberrypi:9100" ]

The second variable path looks like this (using the variable $group in the path expression):

/query${group}
Image
Grafana query using group variable
(Jose Vicente Nunez, CC BY-SA 4.0)

And the fields get as simple as $.[*]:

Image
Grafana instance with variable fields
(Jose Vicente Nunez, CC BY-SA 4.0)

The two variables look like this at the end:

Image
Define variables in Grafana
(Jose Vicente Nunez, CC BY-SA 4.0)

Now that you know what you want, you can focus on writing a web service with the required REST endpoints.

Getting real: Write a REST web service using FastAPI and Python

If you clone the project, then you can set it up to run the web service like this (passing the --reload-include '*.yaml' restarts the application if the inventory file changes):

$ sudo dnf install -y python3-pyyaml
$ python3 -m venv --system-site-packages ~/virtualenv/inventory
. ~/virtualenv/inventory/bin/activate
$ pip install --upgrade pip
$ pip install --upgrade build
$ pip install --upgrade wheel
$pip install --editable .
uvicorn --host 0.0.0.0 --reload --reload-include '*.yaml' inventory.main:app

You can quickly test the REST API on a different terminal:

The only change you need to make now is to create a new data source pointing to the web service (not the one running Mockoon) and see it in action:

Image
Grafana Prometheus JSON panel
(Jose Vicente Nunez, CC BY-SA 4.0)

What's next?

  • Grafana's ability to mix multiple data sources to create a comprehensive view of a system is a powerful tool. Each data source offers various parameters to filter and transform the data, and you saw how you could combine two by using variables.
  • FastAPI is a great tool to create REST API endpoints. I didn't cover automated testing, and this is something you should read about.
  • Mockoon offers several fancy features like partial proxy (when you don't want to mock a whole service) or serving files as a response. It is worth your time to take a look.
  • It is easy to start services with Podman. What if you want to coordinate the startup to start the Ansible web service you wrote and then launch Grafana? You can use docker compose for that.
  • I used Python JSONPath but did not explain its features in detail; it is worth adding to the list of tools in your arsenal.
Topics:   Monitoring   Ansible   Podman   Data visualization  
Author’s photo

Jose Vicente Nunez

Proud dad and husband, software developer and sysadmin. Recreational runner and geek. More about me

Try Red Hat Enterprise Linux

Download it at no charge from the Red Hat Developer program.