In this article, I will demonstrate how to monitor Ansible Automation Platform(AAP) running on OpenShift, using user-workload-monitoring with Prometheus and Grafana.

This article uses the following versions:

  • OpenShift v4.13.1
  • Grafana Operator v5.4.1
  • Ansible Automation Platform v2.4

I won't cover the installation of the Ansible Automation Platform here.

About this article

This article is aimed at users who want a more centralized view of the main usage metrics of the Ansible Automation Platform and identify possible situations of concern. It covers resources such as Grafana, datasources, dashboards, Prometheus, and ServiceMonitors to collect data dynamically.

Prerequisites

  • User with the cluster-admin cluster role
  • OpenShift 4.12 or +
  • Grafana Operator
  • User-Defined Projects enabled

Procedure

Use the following steps to begin monitoring Ansible Automation Platform(AAP) using Prometheus and Grafana.

Enable user-defined projects

Execute this command to add `enableUserWorkload: true` under `data/config.yaml`:

$ oc -n openshift-monitoring patch configmap cluster-monitoring-config -p '{"data":{"config.yaml":"enableUserWorkload: true"}}'

 

`Validate that the prometheus and thanos-ruler pods were created in the openshift-user-workload-monitoring project:

$ oc get pods -n openshift-user-workload-monitoring
NAME READY STATUS RESTARTS AGE
grafana-deployment-6847648746-4mbn9 1/1 Running 0 95m
grafana-operator-controller-manager-7f74d54f44-58pwk 1/1 Running 0 6h55m
prometheus-operator-cf59f9bdc-t7nvm 2/2 Running 0 7h6m
prometheus-user-workload-0 6/6 Running 0 7h6m
prometheus-user-workload-1 6/6 Running 0 7h6m
thanos-ruler-user-workload-0 4/4 Running 0 7h6m
thanos-ruler-user-workload-1 4/4 Running 0 7h6m

 

Install Grafana Operator

Using the WebConsole, in the left side menu, select OperatorHub, then search for Grafana Operator in the search field.

Make sure to change the project context to openshift-user-workload-monitoring at the top.

Click on the operator and click on Install.

01-03

 

Use the following settings:

  • In Update Channel, select v5.
  • In Installation Mode, select A specific namespace on the cluster and choose openshift-user-workload-monitoring below.
  • In Update approval, select Automatic.
  • Click Install.

02-03

Now, create a service account and assign permission to read metrics. Use the following commands:

$ oc project openshift-user-workload-monitoring

$ oc create sa grafana-sa

$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-sa

 

Collect the grafana-sa serviceaccount token and create a secret for the Grafana instance:

$ SECRET=`oc -n openshift-user-workload-monitoring describe sa grafana-sa | awk '/Tokens/{ print $2 }'`

$ TOKEN=`oc -n openshift-user-workload-monitoring get secret $SECRET --template='{{ .data.token | base64decode }}'`

$ cat <<EOF > grafana-secret-creds.yaml
kind: Secret
apiVersion: v1
metadata:
 name: credentials
 namespace: openshift-user-workload-monitoring
stringData:
 GF_SECURITY_ADMIN_PASSWORD: grafana <------ Set the password you want to authenticate with Grafana
 GF_SECURITY_ADMIN_USER: root <------ Set the desired user to authenticate in Grafana
 PROMETHEUS_TOKEN: '${TOKEN}' <------ This variable will receive the token collected above
type: Opaque
EOF

$ oc create -f grafana-secret-creds.yaml

 

Next, create the Grafana instance. It will read the credentials defined in the previously created secret, as seen below:

$ cat <<EOF > grafana-instance.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
 name: grafana
 labels:
  dashboards: "grafana"
  folders: "grafana"
spec:
 deployment:
  spec:
    template:
      spec:
        containers:
          - name: grafana
            env:
              - name: GF_SECURITY_ADMIN_USER
                valueFrom:
                  secretKeyRef:
                    key: GF_SECURITY_ADMIN_USER
                    name: credentials
              - name: GF_SECURITY_ADMIN_PASSWORD
                valueFrom:
                  secretKeyRef:
                    key: GF_SECURITY_ADMIN_PASSWORD
                    name: credentials
 config:
  auth:
    disable_login_form: "false"
    disable_signout_menu: "true"
  auth.anonymous:
    enabled: "false"
  log:
    level: warn
    mode: console
EOF

 

Apply and validate the created Instance:

$ oc -n openshift-user-workload-monitoring create -f grafana-instance.yaml

$ oc -n openshift-user-workload-monitoring get pods -l app=grafana
NAME READY STATUS RESTARTS AGE
grafana-deployment-c4959687c-7vg9d 1/1 Running 0 6m24s

 

Expose the grafana service using an edge-type route. Use the service called grafana-service. Here's an example:

$ oc -n openshift-user-workload-monitoring get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana-operator-operator-metrics-service ClusterIP 172.30.37.111 <none> 8443/TCP 7h1m
grafana-service ClusterIP 172.30.244.194 <none> 3000/TCP 6h16m
prometheus-operated ClusterIP None <none> 9090/TCP,10901/TCP 7h12m
prometheus-operator ClusterIP None <none> 8443/TCP 7h12m
prometheus-user-workload ClusterIP 172.30.159.129 <none> 9091/TCP,9092/TCP,10902/TCP 7h12m
prometheus-user-workload-thanos-sidecar ClusterIP None <none> 10902/TCP 7h12m
thanos-ruler ClusterIP 172.30.213.131 <none> 9091/TCP,9092/TCP,10901/TCP 7h12m
thanos-ruler-operated ClusterIP None <none> 10902/TCP,10901/TCP 7h12m

$ oc -n openshift-user-workload-monitoring create route edge grafana --service=grafana-service --insecure-policy=Redirect

 

Display the route exposed to Grafana:

$ oc -n openshift-user-workload-monitoring get route grafana -o jsonpath='{.spec.host}'

 

It's time to create the Grafana Datasource, which will connect to thanos-querier in the openshift-monitoring project and will use the grafana-sa serviceaccount token that is stored in secret credentials.

$ cat <<EOF > grafana-datasource.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
 name: grafana-ds
 namespace: openshift-user-workload-monitoring  
spec:
 valuesFrom:
  - targetPath: "secureJsonData.httpHeaderValue1"
    valueFrom:
      secretKeyRef:
        name: "credentials"
        key: "PROMETHEUS_TOKEN"
 instanceSelector:
  matchLabels:
    dashboards: "grafana"
 datasource:
  name: Prometheus
  type: prometheus
  access: proxy
  url: https://thanos-querier.openshift-monitoring.svc:9091
  isDefault: true
  jsonData:
    "tlsSkipVerify": true
    "timeInterval": "5s"
    httpHeaderName1: 'Authorization'
  secureJsonData:
    "httpHeaderValue1": "Bearer ${PROMETHEUS_TOKEN}"
  editable: true
EOF

 

Apply and validate the created Datasource:

$ oc -n openshift-user-workload-monitoring create -f grafana-datasource.yaml

$ oc -n openshift-user-workload-monitoring get GrafanaDatasource 
NAME NO MATCHING INSTANCES LAST RESYNC AGE
grafana-ds 119s 3d23h

 

To validate the created datasource using Grafana Console, use the edge route created previously and access it via a browser. Authenticate using the username and password added in secret credentials. 

Once authenticated, click ConfigurationData sources.

03-03

Creating User in Ansible Automation Platform

Access the AAP console and create a user for monitoring. Click on UsersAdd in the left side menu to do this.

04-03

To generate the token, authenticate to AAP using the created user and then click on Users > select the name of the created user > TokenAdd.

Define a description and scope as read and click Save. A popup will be displayed with the token; copy and save it.

05-03

Creating Prometheus ServiceMonitor

Create a ServiceMonitor to collect metrics from AAP and export them through the Prometheus and Thanos Querier.

First, create a secret to store the bearer token previously collected in AAP with the user aap-metrics. Here is the command:

$ oc create secret generic aap-monitor-creds --from-literal=token={{ YOUR AAP BEARER TOKEN }} -n aap

 

Next, create ServiceMonitor, which will discover the AAP service and collect the metrics in the path /api/v2/metrics.

$ cat <<EOF > svc-monitor-aap.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: aap-monitor
 namespace: aap
spec:
 endpoints:
 - interval: 30s
  scrapeTimeout: 10s
  honor_labels: true  
  path: /api/v2/metrics/
  port: http
  scheme: http
  bearerTokenSecret:  
    key: token
    name: aap-monitor-creds <------ Secret previously created with our Bearer Token      
 namespaceSelector:
  matchNames:
  - aap         
 selector:
  matchLabels:
    app.kubernetes.io/component: automationcontroller
EOF

 

Finally, apply and validate the created ServiceMonitor using the following two commands:

$ oc create -f svc-monitor-aap.yaml

$ oc get servicemonitor -n aap                             
NAME AGE
aap-monitor 31m

 

To validate using the WebConsole, in the left side menu, click on Targets in the Observe Session, and in Filter, select User.

06-03

While still in the Observe section, click on Metrics. You will identify whether the AAP metrics are arriving correctly. Use any metric starting with awx_, such as awx_instance_info.

07-02

Creating Grafana Dashboard

Create a Grafana dashboard, which will fetch the JSON externally from GitHub.

$ cat <<EOF > grafana-dashboard-aap.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
 name: grafana-dashboard-aap
 labels:
  app: grafana
spec:
 instanceSelector:
  matchLabels:
    dashboards: grafana  
 folder: "AAP"      
url: https://raw.githubusercontent.com/leoaaraujo/aap-dashboard/main/aap-dash.json
EOF

 

Next, apply and validate the created Grafana dashboard:

$ oc -n openshift-user-workload-monitoring create -f grafana-dashboard-aap.yaml

$ oc -n openshift-user-workload-monitoring get grafanadashboard
NAME NO MATCHING INSTANCES LAST RESYNC AGE
grafana-dashboard-aap 3s 145m

 

Viewing the Dashboard

Access Grafana, and in the left side menu, click on Dashboards and then on Browse.

It will display a folder named AAP and the dashboard AAP - Metrics. Click on the dashboard.

08-03

The dashboard looks like this:

09-02

10-3

Wrap up

I demonstrated creating monitoring for the Ansible Automation Platform using User-Defined Projects from the OpenShift Monitoring stack. I used a Grafana Dashboard to visualize usage metrics and statistics, such as subscription information, playbook metrics, users, and resource consumption within OpenShift.

References

For more details and other configurations, start with the reference documents below: