Drift, a set of capabilities within Red Hat Insights, can help manage and troubleshoot issues across many systems. In this post, we will explore how you can set up configuration alerts and better use the features available to Red Hat Enterprise Linux (RHEL) subscribers.
Red Hat Insights is a configuration analysis service available as part of your Red Hat Enterprise Linux (RHEL) subscription. System administrators can use Drift with RHEL to compare configurations, define baselines, and ultimately perform root-cause analysis of issues during troubleshooting.
Since system configurations tend to vary and drift away from initially defined standard operating environments, operators need to be able to check quickly to determine if a problem can be related to any differences from the recommended configuration.
Let's explore how you can define and receive alerts on system configuration changes from a recommended baseline configuration. This is a recently introduced Drift service feature that uses Red Hat Insights Notifications service.
Red Hat Insights Notifications service
Notifications is a service common to other Red Hat Insights applications to send standardized messages to users of your organization or third-party applications, as defined by your organization notifications administrator.
For example, you might consider sending an email notification for all new Advisor recommendation hits that are found on your system inventory. In addition, you might also want to trigger a notification by sending a message to third-party applications such as ServiceNow or Slack using the webhook option. As an administrator, you can configure how you want events from each application to be handled by the Notifications service under Settings > Notifications.
Introducing Drift Notifications
We recently integrated Red Hat Insights Drift service to Notifications. Drift triggers a ‘Drift from baseline detected’ event each time a system configuration differs from one (or more) specified baseline(s). The event is then handled by Notifications service and routed to users or third party applications based on your organization configuration.
In this post, we assume the following Notifications configuration for the Drift events in the organization:
Figure 1 - Notifications configuration for ‘Drift from baseline detected’ events
Each Drift event results in an email notification sent to registered users in the organization and a post message to a configured webhook end-point.
Note: Some details are purposefully omitted from this post to limit its technical depth. ‘Registered users’ refers to all organization users who have subscribed to notifications in their User Preferences settings. ‘Webhook.site’ is a common website to test and debug Webhooks and HTTP requests. In this example, we have generated a unique endpoint URL and configured an Integration under Settings > Integrations for the organization.
With this Notifications configuration in mind, now let’s have a look at the Drift service and how we can trigger a Drift event each time our system configuration differs from a defined baseline.
Configuring Drift Notifications as part of Baseline
As mentioned in a previous blog post, Drift can be used to identify unexpected changes in system configurations quickly. For example, you can define a baseline in Drift to serve as a standard configuration to compare all other systems against.
If a system configuration drifts over time, performing a comparison to the defined baseline highlights all changes and when they were observed, thus providing an easier way to help ensure all system configurations are in line with the organization's standards.
Getting alerts on system configuration changes from baselines requires associating systems to baselines as part of the Baselines configuration screens. Do this by editing a baseline and selecting the ‘Systems’ tab (see Figure 2).
Figure 2 - System and baseline association
In this example, we added ‘rhel8desktop’ system to the list of associated systems for ‘Standard baseline’ baseline. This means that every time ‘rhel8desktop’ checks in, Drift will check for differences between ‘rhel8desktop’ system and ‘Standard Baseline’ defined baseline and then trigger a ‘Drift from baseline detected’ event if the system configuration differs from the defined baseline.
You can also use Drift APIs to set baseline and system associations. Find more information on the Red Hat Insights API documentation.
End-to-end Drift Notifications
Now that we have configured Drift to alert when ‘rhel8desktop’ configuration differs from our defined ‘Standard Baseline’ baseline, we will receive an email as well as a webhook notification when a difference is detected and triggered.
See examples of the email (Figure 3) and webhook (Figure 4) received after running the
sudo insights-client command on ‘rhel8desktop’ to simulate a scheduled daily run.
Figure 3 - Example of an email received for a ‘Drift from baseline detected’ notification.
The email notification contains information about the system and the baseline(s) it differs from. Clicking on the system name redirects to the Inventory profile for the system. Clicking on the baseline name points directly to the Comparison report for this system and the selected baseline in the Drift application. The Baseline feature and how to use it in Comparison to get a list of differences between system configuration and baseline was introduced in a previous blog post.
Figure 4 - Example of a webhook message received from a ‘Drift from baseline detected’ notification
The webhook notification contains information about the system (in the context section) and about the baseline it differs from (payload section). This can be used in multiple manners to troubleshoot, update CMDB records in third party applications, or even automate the trigger of specific alerts or processes according to your organization.
In this article, we showed how you can now associate systems to defined baselines in Drift, and get alerted wherever a system configuration differs from them. In the example, a ‘Drift from baseline detected’ event is triggered, and email and webhook notifications are sent according to the Notifications configuration for the organization. This new feature can help you maintain your systems configuration to match your organization's standard operating environments.
Other Red Hat Insights services such as Advisor and Policies also use the Notifications service to send alerts based on their findings. Additional event types triggered by Red Hat Insights applications will be added as we grow the service and on-board new features.
We invite you to test this new feature out in Drift and provide feedback and suggestions using the Red Hat Customer Portal feedback form.
About the author
Jerome Marc is a Red Hat Sr. Principal Product Manager with over 15 years of international experience in the software industry spanning product management and product marketing, software lifecycle management, enterprise-level application design and delivery, and solution sales.