Optimize your RHEL infrastructure: 5 performance areas to focus on

Would you rather wait for something to become a problem, or do some performance tweaking and optimization to avoid downtime or performance headaches altogether? The answer to that is easy, but deciding what to optimize or troubleshoot isn't always obvious. In this post we'll cover some common areas where you can use Red Hat Insights and its Advisor service to solve problems before they start.

It’s all about the performance!

Performance tuning and monitoring are some of the most important tasks Red Hat Enterprise Linux (RHEL) system administrators perform, along with patching. As many sysadmins know, incremental performance improvements can accumulate to deliver significant cost savings in resources that over time translate to significant business gains. Avoiding degradation of service or downtime may be even more valuable. 

Using some of the recommendations in this post may help save you money and time. However, knowing what can prevent downtime is not always obvious, so many sysadmins have come to rely on the performance recommendations directly from Red Hat Insights for Red Hat Enterprise Linux (RHEL).

Advisor: The heart and brain of Red Hat Insights

Red Hat Insights was built on the lessons and experience of Red Hat experts working directly with customers and products over the past 20-some years. Red Hat Insights provides continuous, in-depth analysis of registered Red Hat Enterprise Linux systems and it is included as part of your Red Hat Enterprise Linux subscription. Insights helps identify risks to your business, operations and security specifically based on how your systems are configured and performing. 

The Advisor service is both the heart and brain, providing warnings and recommendations based on best practices and known problematic configurations. Some users have referred to Insights as "like having an extra person on the team." Advisor recommendations are the composite of best practices and analytics over thousands of systems where the metrics across a variety of performance metrics can be taken into account. 

This post covers a variety of new Advisor recommendations recently released for performance tuning across key areas that can have a significant impact on improving your daily operations.

Get tuned in: RHEL 8 Performance tuning guide 

First, we started by analysing the performance tuning guide for RHEL 8. We have turned several aspects of this guide into automated recommendations, offering Insights users tailored instructions on how to fine-tune their RHEL 8 for best performance. Mainly to help identify the most appropriate tuned profile, configure the dynamic tickless option on latency-sensitive workloads, or to enable vhost_net kernel feature to optimize VM network performance.

RHEL 8 Tuning recommendations¹:

1. Decreased performance occurs when KVM and VMWare guests are not running with recommended tuned service configuration

2. Decreased throughput performance occurs when compute nodes are not running with recommended tuned service configuration

3. VM network performance is decreased when the vhost_net kernel feature is not enabled

4. CPU performance is decreased when the dynamic tickless option is not enabled

5. Samba server network performance is degraded when configuring specific parameters

Performance takes a team: Performance Co-Pilot 

We now have new Advisor recommendations using metrics captured by Performance Co-Pilot (PCP), providing an additional level of performance monitoring. We developed these in close collaboration with the engineers working on PCP.

The collaboration proved successful and as a result we have published a SYN flood traffic recommendation that leverages PCP data to detect a SYN flood attack. In order to get this working, the PCP team had to implement a new rule into their pmie service (part of the PCP project) to send appropriate log entries into /var/log/messages. These are subsequently picked up by the Insights client and processed by Advisor.

The team went even further, and built even more recommendations taking advantage of this integration - detecting applications using insufficient entropy pool for crypto operations, network connectivity loss after the system reboots, and/or performance issues when using a limited swap size. 

Performance recommendations:

1. Application failures will occur when the system receives SYN flood traffic

2. Network connectivity loss occurs after the system reboots when the pmlogger service is enabled

3. Applications using the random number generator for cryptographic operations fail when an insufficient entropy pool is present on the system

4. System stability loss occurs when the system swap size is smaller than the recommended size

Microsoft SQL Server performance best practices 

By analysing the Microsoft SQL Server performance best practices we have identified two important recommendations to prevent performance degradation. They both assist sysadmins with configuring crucial parameters on network configuration and mount parameters. In both cases it can lead to performance degradations of the SQL Server when running on top of RHEL.

Microsoft SQL Server recommendations:

1. Microsoft SQL filesystem IO performance decreases when noatime mount attribute is not applied

2. Microsoft SQL performance decreases when network setting recommendations are not applied

Avoid Azure performance degradation

There are several new recommendations  for Azure including helping users to prevent performance degradations when running RHEL in the Azure public cloud. 

Advisor now proactively detects a known performance issue in kernel when a VM uses hv_netvsc driver with IP forwarding mode, or when NICs go down after the system resumes from the hibernation state. Furthermore, Advisor offers a remediation for failures in monitoring and start operations when using the azure-lb resource agent in a Pacemaker cluster.

Azure recommendations:

1. Network performance loss will occur when NICs using CX4 VF go down after the system resumes from the hibernation state on the Azure VM

2. Network performance loss occurs when the Azure VM uses hv_netvsc driver with the IP forwarding mode

3. The monitoring operations and start operation on Azure VMs using the azure-lb resource agent in a Pacemaker cluster will fail

Optimal Insights client experience

Last but not least, Advisor can now also help Insights users with configuring their Insights client for an optimal experience. To accomplish this we have implemented two recommendations. One that helps customers keep their Insights client updated, receiving the latest bug fixes and another providing required data to the most recent recommendations. Second, we guide Insights users to use our advanced data collection mechanism, enabling Advisor to provide more detailed recommendations.

Insights client recommendations:

1. When the Insights Client is earlier than "3.1.0", it is not able to get recommendations dependent on core collection

2. When the Insights Client Core egg file is outdated, it's not able to get the latest recommendations and miss bug fixes for older recommendations

Performance matters

No matter which applications you run, including mission-critical applications like SQL Server and SAP, we all know that performance matters. 

But we sometimes forget that a small gain of one tenth of a second can then multiply across servers and over time. With customer facing applications this can have more dramatic impact on perception and revenue as many application experts point out. Red Hat Insights is committed to delivering recommendations to reduce operations, security and business risks across your systems. Improve your system performance and start saving time today by getting started with Red Hat Insights.