Weight: 3
Description: Candidates should be able to configure the syslog daemon. This objective also includes configuring the logging daemon to send log output to a central log server or accept log output as a central log server. Use of the systemd journal subsystem is covered. Also, awareness of rsyslog and syslog-ng as alternative logging systems is included.
Key Knowledge Areas:
Configuration of the syslog daemon
Understanding of standard facilities, priorities and actions
Configuration of logrotate
Awareness of rsyslog and syslog-ng
Terms and Utilities:
syslog.conf
syslogd
klogd
/var/log/
logger
logrotate
/etc/logrotate.conf
/etc/logrotate.d/
journalctl
/etc/systemd/journald.conf
/var/log/journal/
A Linux system has many subsystems and applications running. We use system logging to gather data about our running system from the moment it boots. Sometimes we just need to know that all is well. At other times we use this data for auditing, debugging, knowing when a disk or other resource is running out of capacity, and many other purposes.
Syslog is a standard for sending and receiving notification messages–in a particular format–from various network devices. The messages include time stamps, event messages, severity, host IP addresses, diagnostics and more.
The syslog daemon is a server process that provides a message logging facility for application and system processes.Unfortunately linux logging is one of aspects of linux which is transition fase. The traditional syslog facility and its syslogd daemon has been supplemented by other logging facilities such as rsyslog, syslog-ng, and the systemd journal subsystem.
The syslog.conf file is the main configuration file for the syslogd. Whenever syslogd receives a log message, it acts based on the message's type (or facility) and its priority (called selector fields).
type (facility).priority (severity) destination(where to send the log)
Facilities are simply categories. Some facilities in Linux are: auth, user, kern, cron, daemon, mail, local1, local2, ...
auth
: Security/authentication messages
user
: User-level messages
kern
: Kernel messages
corn
: Clock daemon
daemon
: System daemons
mail
: Mail system
local0 – local7
: Locally used facilities
priorities Unlike facilities, which have no relationship to each other, priorities are hierarchical. Possible priorities in Linux are: emerg/panic, alert, crit, err/error, warn/warning, notice, info, debug
emerg :
System is unusable
alert :
Action must be taken immediately
critical :
Critical conditions
err :
Error conditions
warning :
Warning conditions
notice :
Normal but significant conditions
info :
Informational messages
debug :
Debug-level messages
if we log some specific priority , all the more important things will be logged too
Action: Each line in this file specifies one or more facility/priority selectors followed by an action . On the action field we can have things like:
action | example | notes |
filename | /var/log/messages | Writes logs to specified file |
username | user2 | Will notify that person on the screen |
@ip | @192.168.10.42 | Will send logs to specified log server and that server decides how to treat logs based on its configs. |
In the following syslog.conf line, mail.notice is the selector and /var/log/mail is the action (i.e., “write messages to /var/log/mail”):
mail.notice /var/log/mail
Within the selector, “mail” is the facility (message category) and “notice” is the level of priority. You can see part of syslog.conf (CentOS6) :
#### RULES ##### Log all kernel messages to the console.# Logging much else clutters up the screen.#kern.* /dev/console# Log anything (except mail) of level info or higher.# Don't log private authentication messages!*.info;mail.none;authpriv.none;cron.none /var/log/messages# The authpriv file has restricted access.authpriv.* /var/log/secure# Log all the mail messages in one place.mail.* -/var/log/maillog# Log cron stuffcron.* /var/log/cron# Everybody gets emergency messages*.emerg *# Save news errors of level crit and higher in a special file.uucp,news.crit /var/log/spooler# Save boot messages also to boot.loglocal7.* /var/log/boot.log
*
: wildcard . signifying “any facility” or "any priority"dash - : means it can use memory cache (:don't waist time constantly writing to the disk )
equal sign = : to log ONLY one specific level of priority.
facility.=priority action
There is also /etc/rsyslog.d/ directory and it is better for different softwares and admins to add their specific configs there, instead of editing the main configuration file (See Ubuntu16).
How do boot-time kernel messages get logged before a file system is even mounted? The kernel stores messages in a ring buffer in memory. The klogd
daemon processes these messages directly to a console, or a file such as /var/log/dmesg, or through the syslog facility.
Almost all logfiles are located under /var/log directory and its sub-directories on Linux(CentOS6).
[[email protected] ~]# ls /var/loganaconda.ifcfg.log cron messages-20200217 tallyloganaconda.log cron-20200217 ntpstats vmware-cafanaconda.program.log cups pm-powersave.log vmware-install.loganaconda.storage.log dmesg ppp vmware-tools-upgrader.loganaconda.syslog dmesg.old prelink vmware-vgauthsvc.log.0anaconda.xlog dracut.log sa vmware-vmsvc.loganaconda.yum.log gdm samba wpa_supplicant.logaudit httpd secure wtmpboot.log lastlog secure-20200217 Xorg.0.logbtmp maillog spice-vdagent.log Xorg.0.log.oldbtmp-20200217 maillog-20200217 spooler yum.logConsoleKit messages spooler-20200217 yum.log-20200217
You can use your favorite text editor or less or tail commands in conjunction with grep to read these log files.
We can creating rsyslog listener and catch other systems log messages. That is pretty easy.
###CentOS 6vi /etc/rsyslog.conf###Change from#$UDPServerRun 514#$ModLoad imtcp#$InputTCPServerRun 514### to$UDPServerRun 514$ModLoad imtcp$InputTCPServerRun 514
###ubuntu 16vim /etc/default/rsyslog### Change FromRSYSLOGD_OPTIONS=""### ToRSYSLOGD_OPTIONS="-r"
and finally do not forget to restart the service systemctl restart rsyslog
.
Systemd also has its own journaling program called journald and it stores things in binary files. We can't go and see text files (like what we did in syslog/rsyslog), so we have to use special tool called journalctl to access them(CentOS7):
[[email protected] ~]# journalctl-- Logs begin at Mon 2020-02-10 02:51:48 EST, end at Tue 2020-02-18 09:19:13 EST. --Feb 10 02:51:48 localhost.localdomain systemd-journal[106]: Runtime journal is using 8.Feb 10 02:51:48 localhost.localdomain kernel: Initializing cgroup subsys cpusetFeb 10 02:51:48 localhost.localdomain kernel: Initializing cgroup subsys cpuFeb 10 02:51:48 localhost.localdomain kernel: Initializing cgroup subsys cpuacctFeb 10 02:51:48 localhost.localdomain kernel: Linux version 3.10.0-693.el7.x86_64 (builFeb 10 02:51:48 localhost.localdomain kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-Feb 10 02:51:48 localhost.localdomain kernel: Disabled fast string operationsFeb 10 02:51:48 localhost.localdomain kernel: e820: BIOS-provided physical RAM map:Feb 10 02:51:48 localhost.localdomain kernel: BIOS-e820: [mem 0x0000000000000000-0x0000Feb 10 02:51:48 localhost.localdomain kernel: BIOS-e820: [mem 0x000000000009ec00-0x0000Feb 10 02:51:48 localhost.localdomain kernel: BIOS-e820: [mem 0x00000000000dc000-0x0000
As we mentioned earlier , linux logging is one of aspects of linux which is in under change. Distributions with systemd has journald, beside that some of them still preserve rsyslog and some other not. Try to find out your linux logging system
The config file of journalctl is located at /etc/systemd/journald.conf (CentOS7)
[[email protected] ~]# cat /etc/systemd/journald.conf# This file is part of systemd.## systemd is free software; you can redistribute it and/or modify it# under the terms of the GNU Lesser General Public License as published by# the Free Software Foundation; either version 2.1 of the License, or# (at your option) any later version.## Entries in this file show the compile time defaults.# You can change settings by editing this file.# Defaults can be restored by simply deleting this file.## See journald.conf(5) for details.[Journal]#Storage=auto#Compress=yes#Seal=yes#SplitMode=uid#SyncIntervalSec=5m#RateLimitInterval=30s#RateLimitBurst=1000#SystemMaxUse=#SystemKeepFree=#SystemMaxFileSize=#RuntimeMaxUse=#RuntimeKeepFree=#RuntimeMaxFileSize=#MaxRetentionSec=#MaxFileSec=1month#ForwardToSyslog=yes#ForwardToKMsg=no#ForwardToConsole=no#ForwardToWall=yes#TTYPath=/dev/console#MaxLevelStore=debug#MaxLevelSyslog=debug#MaxLevelKMsg=notice#MaxLevelConsole=info#MaxLevelWall=emerg
The Linux logger command provides an easy way to generate some logs(centOS6)
[[email protected] ~]# logger local1.emerg Hello! This is my log!
and it will appear at /var/log/syslog (or /var/log/messages):
[[email protected] ~]# tail -5 /var/log/messagesFeb 18 06:54:42 server1 NetworkManager[2183]: <info> nameserver '172.16.43.2'Feb 18 06:54:42 server1 NetworkManager[2183]: <info> domain name 'localdomain'Feb 18 06:54:42 server1 dhclient[29269]: bound to 172.16.43.137 -- renewal in 693 seconds.Feb 18 06:54:42 server1 nm-dispatcher.action: Script '/etc/NetworkManager/dispatcher.d/13-named' exited with error status 1.Feb 18 06:59:34 server1 payam: Hello! This is my log!
With the amount of logging that is possible, we need to be able to control the size of log files. This is done using the logrotate
utility , which is usually run as a cron job.
The important files to pay attention to are:
/usr/sbin/logrotate -- the logrotate command itself (the executable)
/etc/cron.daily/logrotate -- the shell script that runs logrotate on a daily basis (note that it might be /etc/cron.daily/logrotate.cron on some systems)
/etc/logrotate.conf -- the log rotation configuration file
Another important file is /etc/logrotate.d, included in the process through this line in the /etc/logrotate.conf file:
Use the /etc/logrotate.conf configuration file to specify how your log rotating and archiving should happen.
[[email protected] ~]# cat /etc/logrotate.conf# see "man logrotate" for details# rotate log files weeklyweekly# keep 4 weeks worth of backlogsrotate 4# create new (empty) log files after rotating old onescreate# use date as a suffix of the rotated filedateext# uncomment this if you want your log files compressed#compress# RPM packages drop log rotation information into this directoryinclude /etc/logrotate.d# no packages own wtmp and btmp -- we'll rotate them here/var/log/wtmp {monthlycreate 0664 root utmpminsize 1Mrotate 1}/var/log/btmp {missingokmonthlycreate 0600 root utmprotate 1}# system-specific logs may be also be configured here.
Each log file may be handled daily, weekly, monthly, or when it grows too large.
parameter | meaning |
missingok | don’t write an error message if the log file is missing |
daily, weekly, monthly | rotate logs daily, weekly, monthly |
rotate N | keep the latest N logs and delete the older ones |
compress | compress the log (creates gz files) |
create mode owner group | Immediately after rotation (before the postrotate script is run) the log file is created with this acces and owner |
minsize N | Log files are rotated when they grow bigger than size bytes, but not before the additionally specified time interval(daily,...) |
this file contains some default settings and sets up rotation for a few logs that are not owned by any system packages. It also uses an include
statement to pull in configuration from any file in the /etc/logrotate.d
directory(CentOS6).
Any packages we install that need help with log rotation will place their Logrotate configuration here.
[[email protected] ~]# ls /etc/logrotate.d/ConsoleKit cups dracut httpd named ppp psacct syslog wpa_supplicant yum[[email protected] ~]# cat /etc/logrotate.d/httpd/var/log/httpd/*log {missingoknotifemptysharedscriptsdelaycompresspostrotate/sbin/service httpd reload > /dev/null 2>/dev/null || trueendscript}
These are the meaning of some of these parameters:
parameter | meaning |
missingok | don’t write an error message if the log file is missing |
notifempty | don’t rotate the log file if it is empty. |
shared scripts | Run prerotate and postrotate scripts for every log file which is rotated |
delaycompress | Postpone compression of the previous log file to the next rotation cycle |
That's all!
.
.
.
.
https://developer.ibm.com/tutorials/l-lpic1-108-2/
https://stackify.com/syslog-101/
https://www.ibm.com/support/knowledgecenter/en/SSB23S_1.1.0.15/gtpc1/hsyslog.html
https://linux.die.net/man/5/syslog.conf
https://www.linuxjournal.com/article/5476
https://jadi.gitbooks.io/lpic1/content/1082_system_logging.html
https://en.wikipedia.org/wiki/Syslog
https://renenyffenegger.ch/notes/Linux/logging/klogd/index
https://www.tecmint.com/create-centralized-log-server-with-rsyslog-in-centos-7/
https://linux.die.net/man/8/logrotate
.