Managing and troubleshooting issues on a number of systems is made easier with the use of Drift, a Red Hat Enterprise Linux (RHEL) configuration analysis tool within Red Hat Insights, freely available as part of your RHEL subscription. In this post, we would like to explore a new feature recently added to the Drift service, the historical system profile.

In case you missed it, check out our previous post for an introduction to creating system baselines and using them to analyze systems’ configuration drift. 

Historical System Profile

As its name suggests, historical system profile (HSP) provides the ability to inspect system profiles and RHEL configuration over time. In the Drift service, we can select and view system profiles over time when selecting systems for comparison.

20202020-10-19-Figure 1. Historical system profile selection for Drift comparison

 

The Insights documentation provides additional information on HSP and how to use it as part of Drift. For example, this feature can be useful when investigating changes that occured over time in a system as part of the root-cause analysis of an issue. In the following section, we are going to investigate how to use HSPs with Baselines.

Baselines feature in Drift

The baseline feature in Drift is used to compare current or historical RHEL configurations to a defined standard. This helps system administrators in daily tasks such as troubleshooting configuration differences and quickly spotting outliers.

2020-10-19-Figure 2. baseline comparison in Drift

The ability to compare system configuration over time was introduced at Red Hat Summit 2020. The feature lets us go back in time and analyze differences and similarities with previous RHEL configurations. Previously, baselines could be defined from scratch, as a copy of an existing system (current configuration), or as a copy of an existing baseline. We recently enhanced this feature with the ability to use previous RHEL configuration as a source for creating baselines.

Using historical system profiles in Drift

Let’s say we want to run a comparison on a system today, a historical profile for this system from a couple days ago, and another system.

2020-10-19-Figure 3. Example of System Comparison selection over time

Comparing the facts visually with Drift quickly reveals that there have been some updates, additions, and removals, allowing us to peruse the highlighted changes across systems.

2020-10-19-Figure 4. Comparison results from systems, including historical profile

 

Through this comparison, we notice someone has changed the RHEL configuration on one of the systems. In our example, the yum RPM was updated and is highlighted as a difference that we need to take a look at. We want to check other systems to see how many others have been changed, adding that new RPM. Looking back at our historical system profile from a couple days ago, the system was configured the way we wanted it at that time, so now we need to compare the other systems to this state.

For this comparison, we can create a new baseline for future comparisons to serve as standard. This new baseline serves as a reference point for comparison, allowing Drift to quickly highlight any changes so that we can easily notice where systems deviate from that standard.

Creating baseline from historical system profile

Drift recently added the ability for us to create a baseline not only from current system configuration, but directly from a saved historical system profile. These historical system profiles are created whenever a system checks in to cloud.redhat.com. Now we can create a baseline from the historical system profile we selected.

While historical system profiles are deleted after seven days (fixed value), we have now preserved this particular system state for future analysis by creating a baseline to reference from that historical system profile. Moving forward, we can perform a comparison of our systems against this baseline periodically to verify that no one has added anything unexpected to the configuration. 

Using the Insights API to generate a system historical change report

Red Hat Insights provides a set of APIs to integrate to. Endpoints and capabilities are fully documented in API documentation. For example, Drift service provides all CRUD operations on Baselines, as well as an API for generating comparison reports. With the addition of the Historical System Profiles feature, we introduced a new API providing previous RHEL configuration for a given system.

This API can be used to query all historical system profiles recorded in Insights against a system id. As an example of using it, we have created a script generating a report of all historical changes of RHEL configuration for a given system over a specified time range. This example is written in Python and can be used as a template for other report generation using Insights APIs. The script and instructions on how to use it can be found on this GitHub repository. Please feel free to experiment and contribute by adding new features as Pull Requests (PR).

Below is a screenshot of the generated report created by the script for one of our systems. We get a list of changes, added or removed configuration records, including the current and previous values recorded for each timestamp.

2020-10-19-Figure 6. Example of a generated historical change report

This report can be used to quickly spot recent changes that occured in a particular environment, and can be useful when troubleshooting issues on a system by providing its configuration change history.

Summary

In this post, we introduced and explored the historical system profile, a new feature recently added to the Drift service in Red Hat Insights. We looked at how it can be used for root cause analysis of issues due to RHEL configuration, and how it complements existing Drift features like Baselines. We also provided an example on using the Insights API to automate the generation of historical change reports for RHEL systems. We do hope you find these features and post useful.

We want to hear from you

We invite you to try Drift on Red Hat Insights and its new Historical System Profile feature. Let us know how you have used the service to more easily maintain your systems. How is Drift improving the efficiency of your operational workflow in your organization?  Can you think of any additional enhancements that would be beneficial for comparing and troubleshooting RHEL systems configurations?  Do reach out with suggestions using Red Hat Customer Portal feedback form.


About the author

Dana Walker is a Software Engineer on the Drift team for Red Hat Insights and cloud.redhat.com. A vocal proponent of inclusion and equity, Dana strives to ensure the long-term health and success of open source by nurturing a welcoming and supportive environment for underrepresented communities. She is a bibliophile, dog and cat mom, foodie, amateur linguist, natural living enthusiast, barefoot runner, acrophobic rock climber, and lover of all things geek.

Read full bio