Stay on top of systemd state with Red Hat Insights

Most Linux distributions use systemd as a system and service manager. It is responsible for managing the boot process, starting and stopping services, and handling system processes. It also provides a range of features and advantages over traditional init systems, such as parallel service startup, on-demand service activation, socket-based activation, and dependency-based service control.

Monitoring the state of systemd and Linux services allows you to check that critical services are running and available, that all required dependencies are met, and that services are starting and stopping correctly in the desired order. Monitoring systemd can also help identify security-related issues or suspicious behavior within services. 

Overall, monitoring systemd and the state of Linux services helps implement the stability, availability, and optimal performance of your system, enabling proactive management and timely response to issues such as service outages. 

Monitoring systemd at scale with Red Hat Insights

Monitoring the state of Linux services at scale presents several challenges that must be addressed for effective monitoring and management. Red Hat Insights analyzes all the crucial system metadata, including the list of running services along with its state. Without any additional configuration on the Red Hat Enterprise Linux (RHEL) hosts, Insights users can now create a custom Insights policy and receive an alert whenever there’s a failed service under systemd management within their environment. 

Users of Insights policies can easily create and manage rule conditions to evaluate against system facts and get automatic alerts whenever there’s a trigger. They can also use operational management to automate tasks such as:

• Raising an alert when conditions are met on system configuration
• Emailing a team when security packages are out of date on a system
• Creating an issue in external ticketing systems when policies are breached

Learn more about Insights policies from our previous post, Getting started with the Red Hat Insights policies capability. 

Configuring custom policy using the new systemd facts

First, navigate to the Red Hat Hybrid Cloud Console, available at console.redhat.com, and select Insights for RHEL on the main page to land in Red Hat Insights. Next, in the left-hand navigation, expand the Operations section and click on Policies. On this screen, you can easily create a new policy by following these steps:

Hit the Create Policy button to open up the Policy creation wizard, select 'From scratch', and hit the Next button.

On the second page, choose a suitable policy name and optional description.

After hitting Next, copy and paste the following condition: facts.systemd.failed > 0 and have it validated by clicking on the Validate condition button. This checks the syntax of the condition. 

Another option is to use the state attribute of the newly added systemd fact: facts.systemd.state = 'degraded'. Other values include initializing, starting, running, maintenance, and stopping. More details can be found here.

The fourth screen lets you configure the Trigger actions. Select Send a notification to notify the user everytime the policy hits.

On the last step, review the policy setup and toggle the "Policy is enabled" button to activate it.

Once applied, the Policy service will trigger a "Policy triggered" event every time there is a failed service picked up by Insights. This event can be turned into an email message or Slack notification, or it can create a ServiceNow ticket﹘for example, according to the user’s Configuring notifications on the Red Hat Hybrid Cloud Console.

This enhancement is a direct result of our discussions with Insights users; in particular with a large user in the financial sector that maintains thousands of RHEL hosts. 

Never hesitate to reach out to us with any questions and feedback at insights@redhat.com, or directly within Insights through our global feedback form. We are happy to help and listen to your needs.

Start mitigating failed systemd services at scale with Red Hat Insights today, included with your existing Red Hat Enterprise Linux subscription. Visit our Red Hat Insights product page for more information on how to get started.