Forwarding Logs in Azure Red Hat OpenShift to Azure Blob Storage

This blog addresses some of the common questions around log management for Microsoft Azure Red Hat OpenShift (ARO). 

ARO is a fully managed turnkey application management platform. It is jointly engineered and supported by Red Hat and Microsoft through site reliability engineers. With integrated support, customers don't need to worry about the complexity of infrastructure management.

ARO generates several types of logs: 

• Infrastructure logs: ARO generates logs related to the underlying infrastructure components, such as virtual machines, networking, and storage. These logs help monitor the performance, availability, and stability of the infrastructure supporting the ARO cluster. Cluster logs from pods in ‘openshift-*’ are also part of infrastructure logs.
• Audit logs: ARO captures audit logs that track activities related to cluster management and administrative operations. These logs provide an audit trail of actions performed by users and service accounts, enabling compliance, security, and troubleshooting purposes.
• Application logs: Container logs generated by user applications running in the cluster, except infrastructure container applications.

Here is a quick video that explains how logs are forwarded to Azure Blob Storage. 

In order to set up log forwarding in ARO, you need to log in as cluster admin to configure operators and forward the logs. Here is a simplified overview of the process:

1: Install Loki Operator: Loki is a horizontally scalable, multi-tenant log aggregation system that indexes logs for efficient storage and retrieval. To capture logs, we'll use Vector, a stream aggregation agent that captures logs from a variety of sources such as pods, containers, and applications and  sends these logs to Loki for indexing.

2: Create Azure Blob storage and configure access keys or secrets within the OpenShift logging namespace, as shown in this example: 

apiVersion: v1
kind: Secret
metadata:
  name: logging-loki-azure
  namespace: openshift-logging
typeL Opaque
stringData:
  Account_name:  <account-name>
  account_key: <account-key>

3: Install Red Hat OpenShift Logging operator, which orchestrates and manages the aggregated logging stack as a cluster-wide service. 

The ClusterLogging custom resource (CR) defines a complete OpenShift Logging environment that includes all the components of the logging stack to collect, store, and visualize logs. The Red Hat OpenShift Logging Operator watches the OpenShift Logging CR and adjusts the logging deployment accordingly.

The Cluster Log Forwarder (CLF) is an API that enables you to send container, infrastructure, and audit logs to specific endpoints within or outside your cluster. You can use it to forward logs to other log aggregators.. When you forward logs externally, the Cluster Logging Operator creates or modifies a Fluentd config map to send logs using your desired protocol.

The Cluster Log Forwarder in OpenShift supports the following protocols for forwarding logs:

Fluentd forward protocol: This is the default protocol and is used to forward logs to a Fluentd server. It is a binary protocol that is well-suited for high-volume log forwarding and is based on a TCP or TLS connection.

• Syslog protocol: This protocol is simple and widely supported and can be used to forward logs to sys log server over a variety of transports such as TCP, UDP and UNIX sockets. It is well-suited for sending log messages over a network.
• HTTP protocols can be used to forward logs to remote servers.

By forwarding logs to Azure blob storage, you can centralize your log data, making it easier to manage and analyze. This not only enhances your ability to troubleshoot issues but also strengthens your security and compliance efforts, making you better equipped to provide the reliability and integrity of your applications running on Azure Red Hat OpenShift. 

Click here for more documentation on forwarding logs to third-party systems. 

Learn More about Azure Red Hat OpenShift:

• Visit Azure Red Hat OpenShift to get started
• Tutorial: Start building with Microsoft Azure Red Hat OpenShift
• Benefits of Azure Red Hat OpenShift