How to use rsyslog to create a Linux log aggregation server
Long ago, when I was a sysadmin for another company on another platform, we did not have a means of centralizing log files. Most mornings, when I arrived at work, I poured myself a cup of coffee and spent 20 to 30 minutes remoting to each server, reviewing the event logs, and then noting any issues detected. Let me assure you that this was not the most efficient way of handling logs. It was, however, the only option I had.
Linux, however, uses the Unix-based syslog tool to manage local log files. Most modern Linux distributions actually use a new-and-improved daemon called rsyslog. rsyslog
is capable of forwarding logs to remote servers. The configuration is relatively simple and makes it possible for Linux admins to centralize log files for archiving and troubleshooting. In this article, I show you how to configure a server to act as the log file repository and then how to configure Linux servers to forward their logs to it.
Log files are a critical component of your server infrastructure. Sysadmins use log files to accomplish many tasks:
- Troubleshooting
- Service Level Agreement (SLA) audits
- Baselining
- Preventative maintenance
Configure the log host
I'll refer to the server hosting the log files as the log host. This server may need some unique configurations compared to other Linux devices. Consider the following settings for your log host server.
Separate partition for /var/log
Installation guidelines often advise admins to mount /var/log
on a separate partition to avoid out of control logs from filling the storage where the root of the filesystem resides. Such a practice is critical when the server receives log files from a great many remote devices.
Configuration file
The actual rsyslog
configuration is managed via a configuration file in the /etc
directory. You will need to edit several lines. Settings may be slightly different, depending on the distribution. Back up the original configuration file, and then open the /etc/rsyslog.conf
file with your favorite text editor.
First, uncomment the two lines for UDP:
# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514
You can also use TCP as the Transport protocol.
# Provides TCP syslog reception
$ModLoad imtcp.so
$InputTCPServerRun 514
You will find these lines near the top of the configuration file in the Modules section.
Next, configure a template for the incoming logs. If you don't configure a template, all of the log entries from the remote servers mix with the log host server's local logs.
$template DynamicFile,"/var/log/%HOSTNAME%/forwarded-logs.log"
*.* -?DynamicFile
This template places all logs from a given host in a directory named for that host. For example, if you have a server named WebServer1, a directory named WebServer1
is created, and all of that server's logs are stored in that directory. Templates are set in a Templates section of the config file. If no specific section is defined, just make sure the templates are defined before the Rules.
There are many different template options available online.
Once you've uncommented the Transport layer protocol and set a template, save your changes to the file. Don't forget to restart rsyslog
.
# systemctl restart rsyslog
Firewall
rsyslog
uses port 514 for network connectivity, whether it's using TCP or UDP. You need to open port 514 in the firewall on the log host server. Assuming you're using UDP, the firewall configuration looks like this:
# firewall-cmd --add-port=514/udp --permanent
# firewall-cmd --reload
Use the following command to confirm your configuration:
# firewall-cmd --list-all
Logrotate
You may find it useful to configure logrotate
, as well. logrotate
helps admins manage large numbers of log files by implementing archiving, compression, deletion, and other necessary log file management tasks. That tool has been effectively covered by Edem Afenyo in the Enable Sysadmin article Setting up Logrotate in Linux, so there is no reason for me to cover it here.
Document the IP address of the log host server
Use ifconfig
or ip addr
to document the log host server's IP address. You will use this address in the client configuration files.
Bonus note: I recommend using IP addresses in configuration files such as /etc/rsyslog.conf
instead of hostnames. Doing so simplifies the configuration and removes name resolution from the computer's connection process and the sysadmin's troubleshooting process. It eliminates an entire later of complexity.
[ You might also like: Setting up logrotate in Linux ]
Configure a log client
I will start this section by configuring a basic Linux server to forward its log files to the log host server configured earlier. This configuration is quite a bit shorter. Don't forget to back up the original /etc/rsyslog.conf
file.
Open the /etc/rsyslog.conf
configuration file with your favorite editor (I prefer Vim, but whatever works for you is fine). Move to the Rules section and create a new line before any other entries. To be clear, you are adding this line before the facility.severity entries. Doing so allows all matches to be forwarded to the destination server.
Add the following information, where IP is the IP address of the log host server:
*.* @IP
Note: If you're using TCP, the syntax is *.* @@IP
Save your changes to the configuration file, and then restart the rsyslog
service:
# systemctl restart rsyslog
Test the configuration
The logger
command is used to manually create a log file entry. Run the following command to generate an entry:
# logger Test
Verify the log file entry by using the tail
command to display the most recent entries in the /var/log/messages
log on the local server:
# tail /var/log/messages
You should see the Test message.
Next, switch to the log host server, and then display the contents of /var/log
:
# ls /var/log
You should see a directory named for the remote server you configured. If you ls
the contents of that directory, you should see logs forwarded from the server. You can use the tail
command to display the contents of the logs in this server's subdirectory. You should see the Test message repeated here, too.
Configure your remaining servers
At this point, you can configure your remaining Linux servers to forward their logs to the log host. You could distribute a new version of the /etc/rsyslog.conf
file by using rsync, SSH, or even set the configuration with Ansible.
Forward specific logs
The example settings above forward all logs to the log host system (hence the *.* syntax). You can choose to only forward entries for individual facilities or forward entries for different facilities to different log host servers.
Let's say you wanted to send cron logs to hostlogserver1 (where the sysadmins can review the entries) and FTP logs to hostlogserver2 (where the netadmins can check the entries). The configuration looks something like this:
cron.* @10.1.1.15
ftp.* @10.1.1.20
Where hostlogserver1 has an IP address of 10.1.1.15 and hostlogserver2 has an IP address of 10.1.1.20.
You can also forward the same entries to two different servers:
*.* @10.1.1.15
*.* @10.1.1.20
Other platforms
Recall that in the introduction, I pointed out that this log file mechanism is really a Unix system, not a Linux-specific function. That means that virtually any Unix-based device that maintains logs can participate.
Configure routers to forward logs
I was recently working in a client's lab environment that used a router configured with VyOS. VyOS is a Linux-based router, VPN, firewall, and NAT distribution. We used the above process (editing the /etc/rsyslog.conf
configuration file) to set the router to forward its logs to a central server. Network administrators can easily archive router, VPN, and other logs by using rsyslog
.
Another OS used in the lab environment was pfSense. pfSense is a FreeBSD derivative operating system that can also forward logs to a central location by using rsyslog
. Again, it was a straightforward configuration, though this time it was via a web-based GUI.
[ Free online course: Red Hat Enterprise Linux technical overview. ]
Wrap up
Consider the options available to sysadmins with rsyslog
log forwarding. Your network team could centralize logs for all internal and perimeter routers, VPN appliances, and firewalls. Sysadmins could organize logs based on areas of responsibility or geography. The centralization may impact your security, service desk, and server admin teams. rsyslog
is a basic but essential Linux service, and taking advantage of its ability to centralize logs allows you to get the most out of it.
Damon Garn
Damon Garn owns Cogspinner Coaction, LLC, a technical writing, editing, and IT project company based in Colorado Springs, CO. Damon authored many CompTIA Official Instructor and Student Guides (Linux+, Cloud+, Cloud Essentials+, Server+) and developed a broad library of interactive, scored labs. He regularly contributes to Enable Sysadmin, SearchNetworking, and CompTIA article repositories. Damon has 20 years of experience as a technical trainer covering Linux, Windows Server, and security content. He is a former sysadmin for US Figure Skating. He lives in Colorado Springs with his family and is a writer, musician, and amateur genealogist. More about me