LFCS Admin Exam preparation guide – Locate and analyze system log files

LFCS Admin Exam preparation guide – Locate and analyze system log files

LFCS Admin Exam preparation guide series, main page can be found here.

This post is part of the Operation of Running Systems from the domain competency list for the exam. The full list can be found in the link above paragraph or the Linux Foundation page here.

To be able to troubleshoot Linux based Operation Systems we need to know where to look for log files and how to read them.

The common location is /var/log, running ls command we can see there a few of them. The auth.log contain information about all login, logout, user management operations like change password or add to the group.

system logs location listing

Another useful log is dpkg.log, where we can check all the package operations like an update, upgrade, installation, and removal. Next important log is syslog which contains all information about what is happening inside our operating system: what jobs were run as daily tasks.

To see what is inside the log files we can use a few of the commands which we learn already: cat, less, head, and tail. Probably the most useful will be the less command as we can scroll up and down, and search for information. One point to remember /var/log contains system files, we will have to use sudo to be able to read them.

auth.log content example

But what and how the logs file is created, all this thanks to program call rsyslog which utilizes all the features of Syslog protocol. The configuration file for rsyslog is stored inside the /etc/rsyslog.conf file. There are separate files inside /etc/rsyslog.d directory which stores individual configuration files for a separate application to store their own log settings. Listing of the rsyslog.conf file shows to use that default logging rules are defined inside the /etc/rsyslog.d/50-default.conf file.

rsyslog logging definition inside the 50-default.conf file

The first line is defining that auth, and authpriv message to be written to the /var/log/auth.log file. The wildcard (*) means that all severity levels to be logged. The next line defines that all events (*.*) except the security events will be sent to the /var/log/syslog file. The minus in front of the filename tells the rsyslogd to not sync file after every write to increase the performance. The following line defines where to store the kernel event messages. The is another rule in the configuration file *.emerg :omusrmsg:*, which specifies that all emergency events to be sent to all current logging users to the system.

Linux introduces the logrotate utility, which helps to maintain the Syslog file size by automatically splitting into an archive file base on time or the file size. We can find the archive file in the same /var/log directory, they are numbered and the gz extension is added.

The Linux systemd service package includes the systemd-journald utility for storing logs. This program saves the event messages differently than the Syslog protocol. The messages are not stored inside the text file but inside the database in binary format. To retrieve the messages, we can use the journalctl command. For example, the journalctl -a _HOSTNAME=lfcsexam02 will display messages for the specified host.

journalctl command example

Thank you for reading, and keep learning!