How to find and interpret system log files on Linux

Log files and journals are important to a system administrator's work. They reveal a great deal of information about a system and are instrumental during troubleshooting and auditing.

Log files contain events and messages generated by the kernel, applications, and users that log into the system.

Use rsyslog

Syslog and rsyslog have long been used to provide logging on Linux servers. Systemd became the default service manager with Red Hat Enterprise Linux (RHEL) 7, and it introduced its own logging system called systemd-journald. systemd-journald continues to be the logging mechanism on RHEL 8 and 9 while keeping rsyslog for backward compatibility.

The rsyslog service keeps various log files in the /var/log directory. You can open these files using native commands such as tail, head, more, less, cat, and so forth, depending on what you are looking for.

For example, to display boot and other kernel messages, view /var/log/messages:

[server]$ cat /var/log/messages

Use grep and other filtering tools to gather more specific events from a file. You can also use tail to view files as they are updated:

[server]$ tail -f /var/log/messages

In the command above, the -f option updates the output when new log file entries are added.

Check the /var/log/secure file to view users and their activities:

[server]$ tail -f /var/log/secure

Use systemd-journald

The systemd-journald service does not keep separate files, as rsyslog does. The idea is to avoid checking different files for issues. Systemd-journald saves the events and messages in a binary format that cannot be read with a text editor. You can query the journal with the journalctl command.

To show all event messages, use:

[server]$ journalctl

This is similar to the /var/log/messages in the rsyslog service.

[ Download the free eBook Manage your Linux environment for success. ]

To view the last 10 event messages, use:

[server]$ journalctl -n

You can view the last n entries by using journalctl -n {number}. For example, to view the last 20 entries, type:

[server]$ journalctl -n 20

To output new journal entries as they are written to the journal, use:

[server]$ journalctl -f

Run the following command to display the kernel message log from the last boot:

[server]$ journalctl -k

The journalctl command has several choices that can make querying the journal easier. You can query the log based on applications, time frame, systemd units, priority, and many other options. Run the journalctl –help command to list the available options.

To view journal entries based on their critical priority, use:

[server]$ journalctl -p crit

To query all messages related to a particular user, find the user's ID (UID) and use that to perform the query. For example, to check all logs related to the sadmin user, run:

[server]$ id sadmin
uid=1000(sadmin) gid=1000(sadmin) groups=1000(sadmin)
[server]$journalctl _UID=1000

To view journal entries for today, use:

[[server]$ journalctl --since today

To view journal entries related to the sshd daemon, run:

[server]$ journalctl -u sshd

The same applies to other services running under systemd that can be stopped and started with systemctl.

To check for messages related to the httpd service for the past hour, you can run:

[server]$ journalctl -u httpd –since "1 hour ago"

Manage log forwarding

RHEL 8 and 9 servers use both rsyslog and systemd-journald, and they complement each other to perform logging. Systemd-journald does not have a mechanism to forward logs to external systems and monitoring applications. A configuration modifies this in the /etc/systemd/journald.conf. The ForwardToSyslog parameter defines whether entries in the journal should be forwarded to syslog. When enabled, syslog then captures the entries as they come through systemd-journald and forwards them accordingly.

Wrap up

Current RHEL distributions rely on systemd and the related journald logging tool. However, rsyslog still plays a major role in logging for many administrators—particularly when it comes to log forwarding and centralization. Sysadmins must know how to use both log mechanisms effectively. These commands will help you learn and use system logging for troubleshooting and audits. Work with them both and you will have a much better understanding of what is happening on your Linux systems.