How to use rsyslog to create a Linux log aggregation server

Long ago, when I was a sysadmin for another company on another platform, we did not have a means of centralizing log files. Most mornings, when I arrived at work, I poured myself a cup of coffee and spent 20 to 30 minutes remoting to each server, reviewing the event logs, and then noting any issues detected. Let me assure you that this was not the most efficient way of handling logs. It was, however, the only option I had.

Linux, however, uses the Unix-based syslog tool to manage local log files. Most modern Linux distributions actually use a new-and-improved daemon called rsyslog. rsyslog is capable of forwarding logs to remote servers. The configuration is relatively simple and makes it possible for Linux admins to centralize log files for archiving and troubleshooting. In this article, I show you how to configure a server to act as the log file repository and then how to configure Linux servers to forward their logs to it.

Log files are a critical component of your server infrastructure. Sysadmins use log files to accomplish many tasks:

• Troubleshooting
• Service Level Agreement (SLA) audits
• Baselining
• Preventative maintenance

Configure the log host

I'll refer to the server hosting the log files as the log host. This server may need some unique configurations compared to other Linux devices. Consider the following settings for your log host server.

Separate partition for /var/log

Installation guidelines often advise admins to mount /var/log on a separate partition to avoid out of control logs from filling the storage where the root of the filesystem resides. Such a practice is critical when the server receives log files from a great many remote devices.

Configuration file

The actual rsyslog configuration is managed via a configuration file in the /etc directory. You will need to edit several lines. Settings may be slightly different, depending on the distribution. Back up the original configuration file, and then open the /etc/rsyslog.conf file with your favorite text editor.

First, uncomment the two lines for UDP:

# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514

You can also use TCP as the Transport protocol.

# Provides TCP syslog reception
$ModLoad imtcp.so
$InputTCPServerRun 514

You will find these lines near the top of the configuration file in the Modules section.

Next, configure a template for the incoming logs. If you don't configure a template, all of the log entries from the remote servers mix with the log host server's local logs.

$template DynamicFile,"/var/log/%HOSTNAME%/forwarded-logs.log" 
*.* -?DynamicFile

This template places all logs from a given host in a directory named for that host. For example, if you have a server named WebServer1, a directory named WebServer1 is created, and all of that server's logs are stored in that directory. Templates are set in a Templates section of the config file. If no specific section is defined, just make sure the templates are defined before the Rules.

There are many different template options available online.

Once you've uncommented the Transport layer protocol and set a template, save your changes to the file. Don't forget to restart rsyslog.

# systemctl restart rsyslog

Firewall

rsyslog uses port 514 for network connectivity, whether it's using TCP or UDP. You need to open port 514 in the firewall on the log host server. Assuming you're using UDP, the firewall configuration looks like this:

# firewall-cmd --add-port=514/udp --permanent

# firewall-cmd --reload

Use the following command to confirm your configuration:

# firewall-cmd --list-all

Logrotate

You may find it useful to configure logrotate, as well. logrotate helps admins manage large numbers of log files by implementing archiving, compression, deletion, and other necessary log file management tasks. That tool has been effectively covered by Edem Afenyo in the Enable Sysadmin article Setting up Logrotate in Linux, so there is no reason for me to cover it here.

Document the IP address of the log host server

Use ifconfig or ip addr to document the log host server's IP address. You will use this address in the client configuration files.

Bonus note: I recommend using IP addresses in configuration files such as /etc/rsyslog.conf instead of hostnames. Doing so simplifies the configuration and removes name resolution from the computer's connection process and the sysadmin's troubleshooting process. It eliminates an entire later of complexity.

[ You might also like: Setting up logrotate in Linux ]

Configure a log client

I will start this section by configuring a basic Linux server to forward its log files to the log host server configured earlier. This configuration is quite a bit shorter. Don't forget to back up the original /etc/rsyslog.conf file.

Open the /etc/rsyslog.conf configuration file with your favorite editor (I prefer Vim, but whatever works for you is fine). Move to the Rules section and create a new line before any other entries. To be clear, you are adding this line before the facility.severity entries. Doing so allows all matches to be forwarded to the destination server.

Add the following information, where IP is the IP address of the log host server:

*.* @IP

Note: If you're using TCP, the syntax is *.* @@IP

Save your changes to the configuration file, and then restart the rsyslog service:

# systemctl restart rsyslog

Test the configuration

The logger command is used to manually create a log file entry. Run the following command to generate an entry:

# logger Test

Verify the log file entry by using the tail command to display the most recent entries in the /var/log/messages log on the local server:

# tail /var/log/messages

You should see the Test message.

Next, switch to the log host server, and then display the contents of /var/log:

# ls /var/log

You should see a directory named for the remote server you configured. If you ls the contents of that directory, you should see logs forwarded from the server. You can use the tail command to display the contents of the logs in this server's subdirectory. You should see the Test message repeated here, too.

Configure your remaining servers

At this point, you can configure your remaining Linux servers to forward their logs to the log host. You could distribute a new version of the /etc/rsyslog.conf file by using rsync, SSH, or even set the configuration with Ansible.

Forward specific logs

The example settings above forward all logs to the log host system (hence the *.* syntax). You can choose to only forward entries for individual facilities or forward entries for different facilities to different log host servers.

Let's say you wanted to send cron logs to hostlogserver1 (where the sysadmins can review the entries) and FTP logs to hostlogserver2 (where the netadmins can check the entries). The configuration looks something like this:

cron.* @10.1.1.15

ftp.* @10.1.1.20

Where hostlogserver1 has an IP address of 10.1.1.15 and hostlogserver2 has an IP address of 10.1.1.20.

You can also forward the same entries to two different servers:

*.* @10.1.1.15

*.* @10.1.1.20

Other platforms

Recall that in the introduction, I pointed out that this log file mechanism is really a Unix system, not a Linux-specific function. That means that virtually any Unix-based device that maintains logs can participate.

Configure routers to forward logs

I was recently working in a client's lab environment that used a router configured with VyOS. VyOS is a Linux-based router, VPN, firewall, and NAT distribution. We used the above process (editing the /etc/rsyslog.conf configuration file) to set the router to forward its logs to a central server. Network administrators can easily archive router, VPN, and other logs by using rsyslog.

Another OS used in the lab environment was pfSense. pfSense is a FreeBSD derivative operating system that can also forward logs to a central location by using rsyslog. Again, it was a straightforward configuration, though this time it was via a web-based GUI.

[ Free online course: Red Hat Enterprise Linux technical overview. ] 

Wrap up

Consider the options available to sysadmins with rsyslog log forwarding. Your network team could centralize logs for all internal and perimeter routers, VPN appliances, and firewalls. Sysadmins could organize logs based on areas of responsibility or geography. The centralization may impact your security, service desk, and server admin teams. rsyslog is a basic but essential Linux service, and taking advantage of its ability to centralize logs allows you to get the most out of it.