Building better RHEL notifications with Red Hat Insights

Red Hat Insights is one of the best tools Red Hat Enterprise Linux (RHEL) admins have at their disposal to manage systems at scale. With the addition of notifications and integrations, Insights is going to be an even better way to proactively manage systems and respond quickly to problems. 

Today, Insights can generate notifications when policies are triggered, systems drift from baseline, and when recommendations are discovered or resolved. And these features are ever-improving. In this article, I’m going to show you how to generate an alert from some Red Hat Enterprise Linux (RHEL) systems registered to Insights through the Hybrid Cloud Console, and do something useful with it instead of just giving you another red light on a dashboard.

The problem of too many data sources

In IT operations, your life is full of data sources. Server logs, warning lights, dashboards, alerting systems, log aggregators and pagers. When I still worked as a sysadmin, I had to watch our log aggregator, which had about half of our servers logging to it, our monitoring system, that wasn’t catching everything that was important, and also had to manually watch the rest of the fleet for errors. 

Most of the time that turned into, “hey did you see this?” type emails or phone calls. I was a single sysadmin at the helm of a moderately large fleet of servers. I got the job done, and as a sysadmin, I was good at putting out fires. I had to be. I would design new systems with as much attention to detail as I could, and then do my best to keep them up to date and free of errors. Assuming I knew about the errors in the first place, that is.

Today a sysadmin has many tools at their disposal to help solve these problems. All of them give yet another dashboard to look at or more cruft in the inbox. Some are checking for patches, others for vulnerabilities. Very few put everything in one place and give you useful notification tools. 

Red Hat Insights: One tool to rule them all

Red Hat Insights, included with many existing Red Hat subscriptions, can help, giving you a great way to take data directly from your Red Hat deployments and get meaningful alerts. Not just on software updates or vulnerabilities, but also suggestions for possible improvements and potential configuration issues. 

Red Hat Insights will even make recommendations for improvements on deployments of SAP for example, or Microsoft SQL Server on Red Hat Enterprise Linux. In many cases, Insights will even provide you with a ready-to-run Ansible playbook to resolve the issue.

But what if you don’t want another dashboard? There was a time when dashboards were the answer to everything. Our inboxes were all flooded with so many notifications that we went blind to them. A dashboard took all of that data and made it readable.

Insights can do that too, and it's a pretty good dashboard—but not if you’re not looking at it. So, how about some meaningful notifications instead? Or even better, how about a webhook into your existing request tracking system or issue tracker?

You can find our documentation on how to configure notifications and integrations in the Hybrid Cloud Console documentation on the Red Hat Customer Portal. But for a quick rundown, I'm going to walk through adding an integration in this article to get you started.

Adding an integration

First, you’ll need something to generate an alert from. I am going to set up an alert on “Resolved recommendations” using an Integration. What this will do is send an alert to a webhook. For the demo, I am sending a webhook to a Matrix chat group, but many services support webhooks.  

Click on the gear at the top of the Hybrid Cloud Console, next to your avatar (upper right), and click on Settings. Browse to Integrations on the left-hand side, and click the blue “Add Integration” button. You should have a screen similar to this: 

Name your integration, select Webhook as the type, enter the URL of the hook (and the security token if required by your target), and click Save.

Setting up alerts and notifications

Now we’ll need to tell the console to actually send alerts when recommendations are resolved. From the same Settings page, browse to Notifications -> Red Hat Enterprise Linux on the left. You will see a list of behavior groups, and a table of Events that you can select behaviors for. First, we’ll need to create a behavior group, so press the blue “Create new group” button.

Give your group a name, and then select what actions you would like this group to take. In this example, I am sending both an email and a webhook integration.

Now we’ll need to tell the console to send notifications of resolved recommendations to our behavior group. Press the edit icon next to the “Resolved recommendation” line on the table of events, select the behavior group that you added, and press the blue checkmark.

Now, as long as you have your notification user preferences configured to allow instant notifications from Advisor, you should receive an email and our webhook should be called after any recommendation is resolved. 

I received the following email notification:

And I received the following in my chat group:

Webhooks are JSON data, and whatever you have set up to receive the webhook should be able to parse it into something. I left it as JSON to illustrate that you can do whatever you like with this data. Of course, email notifications can be sent directly into an issue or request tracker as well, but nothing quite beats the versatility of raw data.

It’s worth noting that notifications from Insights may not be real-time, as Insights data is reported periodically from your systems, roughly once every 24 hours. Most remediations generated by Insights will force the insights-client to sync at the end. In my case, I remediated manually, and then I forced it by running the `insights-client` manually on a system that I had remediated an issue on for this test.

So, whether you’re managing a fleet of thousands of RHEL servers or just a few, I hope this article has helped save you some time and dashboard fatigue. Thanks for reading! 

Learn more

• Get more from your Red Hat Enterprise Linux subscription

• Red Hat Insights

• Red Hat Integration

• Tales from the field: A system administrator's guide to IT automation