201.3. Kernel runtime management and troubleshooting

201.3 Kernel runtime management and troubleshooting

Weight: 4

Description: Candidates should be able to manage and/or query a 2.6.x, 3.x or 4.x kernel and its loadable modules. Candidates should be able to identify and correct common boot and run time issues. Candidates should understand device detection and management using udev. This objective includes troubleshooting udev rules.

Key Knowledge Areas:

  • Use command-line utilities to get information about the currently running kernel and kernel modules

  • Manually load and unload kernel modules

  • Determine when modules can be unloaded

  • Determine what parameters a module accepts

  • Configure the system to load modules by names other than their file name.

  • /proc filesystem

  • Content of /, /boot/ , and /lib/modules/

  • Tools and utilities to analyze information about the available hardware

  • udev rules

Terms and Utilities:

  • /lib/modules/kernel-version/modules.dep

  • module configuration files in /etc/

  • /proc/sys/kernel/

  • /sbin/depmod

  • /sbin/rmmod

  • /sbin/modinfo

  • /bin/dmesg

  • /sbin/lspci

  • /usr/bin/lsdev

  • /sbin/lsmod

  • /sbin/modprobe

  • /sbin/insmod

  • /bin/uname

  • /usr/bin/lsusb

  • /etc/sysctl.conf, /etc/sysctl.d/

  • /sbin/sysctl

  • udevmonitor

  • udevadm monitor

  • /etc/udev/

Kernel Modules

Linux Kernel is modular since version 2.0. Being modular make simplicity and efficiency, the code which is not needed wouldn't be loaded. Kernel Modules have different conditions. A module might be compiled, or not. If it has already compiled it might be loaded or not. So Loading an unload but compiled kernel is easy, otherwise required kernel manipulation.We use CentOS7 for demonstartion:

lsmod

To list Kernel Loaded modules use lsmod

root@server1:~# lsmod
Module                  Size  Used by
bnep                   20480  2
nfnetlink_queue        24576  0
nfnetlink_log          20480  0
nfnetlink              16384  2 nfnetlink_log,nfnetlink_queue
bluetooth             557056  7 bnep
vmw_vsock_vmci_transport    28672  2
vsock                  36864  3 vmw_vsock_vmci_transport
vmw_balloon            20480  0
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
ghash_clmulni_intel    16384  0
snd_ens1371            28672  4
snd_ac97_codec        131072  1 snd_ens1371
gameport               16384  1 snd_ens1371
ac97_bus               16384  1 snd_ac97_codec
pcbc                   16384  0
snd_pcm               102400  2 snd_ac97_codec,snd_ens1371
aesni_intel           167936  0
aes_x86_64             20480  1 aesni_intel
snd_seq_midi           16384  0
snd_seq_midi_event     16384  1 snd_seq_midi
crypto_simd            16384  1 aesni_intel
snd_rawmidi            32768  2 snd_seq_midi,snd_ens1371
snd_seq                65536  2 snd_seq_midi_event,snd_seq_midi
glue_helper            16384  1 aesni_intel
cryptd                 24576  3 crypto_simd,ghash_clmulni_intel,aesni_intel
intel_rapl_perf        16384  0
snd_seq_device         16384  3 snd_seq,snd_rawmidi,snd_seq_midi
snd_timer              32768  2 snd_seq,snd_pcm
input_leds             16384  0
joydev                 20480  0
serio_raw              16384  0
i2c_piix4              24576  0
shpchp                 36864  0
snd                    77824  15 snd_seq,snd_ac97_codec,snd_timer,snd_rawmidi,snd_ens1371,snd_seq_device,snd_pcm
soundcore              16384  1 snd
vmw_vmci               69632  2 vmw_balloon,vmw_vsock_vmci_transport
nfit                   49152  0
mac_hid                16384  0
parport_pc             32768  0
ppdev                  20480  0
lp                     20480  0
parport                49152  3 lp,parport_pc,ppdev
autofs4                40960  2
vmw_pvscsi             24576  0
vmxnet3                61440  0
hid_generic            16384  0
usbhid                 53248  0
hid                   118784  2 hid_generic,usbhid
vmwgfx                241664  3
ttm                    98304  1 vmwgfx
drm_kms_helper        151552  1 vmwgfx
mptspi                 24576  2
syscopyarea            16384  1 drm_kms_helper
mptscsih               40960  1 mptspi
sysfillrect            16384  1 drm_kms_helper
psmouse               139264  0
sysimgblt              16384  1 drm_kms_helper
mptbase               102400  2 mptscsih,mptspi
fb_sys_fops            16384  1 drm_kms_helper
ahci                   36864  0
e1000                 143360  0
drm                   352256  6 vmwgfx,ttm,drm_kms_helper
libahci                32768  1 ahci
scsi_transport_spi     32768  1 mptspi
pata_acpi              16384  0
fjes                   77824  0

lets try to load and unloaded module " jfs " that old journaling file system. for loading an unloaded but compiled module there are two solutions. An old solution is insmod , lets try it first

root@server1:~# insmod jfs
insmod: ERROR: could not load module jfs: No such file or directory

But it doesn't work :(( so let get more information about "jfs" module with modinfo:

root@server1:~# modinfo jfs
filename:       /lib/modules/4.10.0-40-generic/kernel/fs/jfs/jfs.ko
alias:          fs-jfs
license:        GPL
author:         Steve Best/Dave Kleikamp/Barry Arndt, IBM
description:    The Journaled Filesystem (JFS)
srcversion:     3249CD436BA490FF745560A
depends:        
intree:         Y
vermagic:       4.10.0-40-generic SMP mod_unload 
parm:           nTxBlock:Number of transaction blocks (max:65536) (int)
parm:           nTxLock:Number of transaction locks (max:65536) (int)
parm:           commit_threads:Number of commit threads (int)

modinfo shows a bunch of information about a module. Like its filename, dependencies and possible parameters.Till now we have realize that "jfs" module has been compiled and its available but it is not loaded. inorder to load a module with insmod we have to specify the full patch of module:

root@server1:~# insmod /lib/modules/4.10.0-40-generic/kernel/fs/jfs/jfs.ko
root@server1:~# lsmod | grep jfs
jfs                   184320  0

Haha its loaded now, Lets remove it by using rmmod:

root@server1:~# lsmod | grep jfs
jfs                   184320  0
root@server1:~# 
root@server1:~# rmmod jfs
root@server1:~# lsmod | grep jfs

But as you see using insmod is not that much easy, insmod has another problem and that is dependencies. "jfs" module has no dependency so we were able to load it very easily, if a module has some dependencies, insmod dosn't load those required dependencies automatically. So it would be administrator task. All these shortages cause we come to this conclusion that insmod is not perfect and we need a modern tool, which automatically know the patch and load required dependencies, that is modprobe.

root@server1:~# lsmod | grep lp
lp                     20480  0
glue_helper            16384  1 aesni_intel
parport                49152  1 lp
drm_kms_helper        151552  1 vmwgfx
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
drm                   352256  6 vmwgfx,ttm,drm_kms_helper
root@server1:~# modinfo lp
filename:       /lib/modules/4.10.0-40-generic/kernel/drivers/char/lp.ko
license:        GPL
alias:          char-major-6-*
srcversion:     5273B3FB623D57CACC37C09
depends:        parport
intree:         Y
vermagic:       4.10.0-40-generic SMP mod_unload 
parm:           parport:array of charp
parm:           reset:bool
root@server1:~# modinfo parport
filename:       /lib/modules/4.10.0-40-generic/kernel/drivers/parport/parport.ko
license:        GPL
srcversion:     CED357203546F0957229054
depends:        
intree:         Y
vermagic:       4.10.0-40-generic SMP mod_unload 
root@server1:~# rmmod lp
root@server1:~# rmmod parport
root@server1:~# insmod /lib/modules/4.10.0-40-generic/kernel/drivers/char/lp.ko
insmod: ERROR: could not insert module /lib/modules/4.10.0-40-generic/kernel/drivers/char/lp.ko: Unknown symbol in module
root@server1:~# modprobe lp
root@server1:~# lsmod | grep lp
lp                     20480  0
parport                49152  1 lp
glue_helper            16384  1 aesni_intel
drm_kms_helper        151552  1 vmwgfx
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
drm                   352256  6 vmwgfx,ttm,drm_kms_helper

also for removing use modprobe -r lp. But how modprobe realize module dependencies and act such a smart tool? modules.dep

root@server1:~# cd /lib/modules/
root@server1:/lib/modules# ls
4.10.0-28-generic  4.10.0-40-generic
root@server1:/lib/modules# cd 4.10.0-40-generic/
root@server1:/lib/modules/4.10.0-40-generic# ls
build          modules.alias.bin    modules.dep.bin  modules.symbols
initrd         modules.builtin      modules.devname  modules.symbols.bin
kernel         modules.builtin.bin  modules.order    vdso
modules.alias  modules.dep          modules.softdep
root@server1:/lib/modules/4.10.0-40-generic# depmod -a

we can use depmod -a to create and update modules.dep with all new modules and dependencies. Now that we have learned alot of tools, let play game with cdrom parameters [in ubunto cdrom is not a module, so use other distro]:

[root@server1 ~]# lsmod | grep cdrom
cdrom                  42556  1 sr_mod
[root@server1 ~]# modinfo cdrom
filename:       /lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/cdrom/cdrom.ko.xz
license:        GPL
rhelversion:    7.4
srcversion:     BE3BD0D17D080229D55B173
depends:        
intree:         Y
vermagic:       3.10.0-693.el7.x86_64 SMP mod_unload modversions 
signer:         CentOS Linux kernel signing key
sig_key:        DA:18:7D:CA:7D:BE:53:AB:05:BD:13:BD:0C:4E:21:F4:22:B6:A4:9C
sig_hashalgo:   sha256
parm:           debug:bool
parm:           autoclose:bool
parm:           autoeject:bool
parm:           lockdoor:bool
parm:           check_media_type:bool
parm:           mrw_format_restart:bool
[root@server1 ~]# modprobe cdrom lookdoor=1

We as users like modules to load automatically, mean while parameters should be loaded automatically, to make parameters persistence create a file in /etc/modprobe.d :

 root@server1:~# cd /etc/modprobe.d/
root@server1:/etc/modprobe.d# ls
alsa-base.conf           blacklist-framebuffer.conf   blacklist-watchdog.conf
blacklist-ath_pci.conf   blacklist-modem.conf         fbdev-blacklist.conf
blacklist.conf           blacklist-oss.conf           iwlwifi.conf
blacklist-firewire.conf  blacklist-rare-network.conf  mlx4.conf
root@server1:/etc/modprobe.d# vi cdroom-lookdoor.conf

and insert:

options cdroom lookdoor=1

If you have noticed there is blacklist.conf file in /etc/modfprobe.d , if you don't wana let a specific module be loaded use it:

# This file lists those modules which we don't want to be loaded by
# alias expansion, usually so some other driver will be loaded for the
# device instead.

# evbug is a debug tool that should be loaded explicitly
blacklist evbug

# these drivers are very simple, the HID drivers are usually preferred
blacklist usbmouse
blacklist usbkbd

# replaced by e100
blacklist eepro100

# replaced by tulip
blacklist de4x5

# causes no end of confusion by creating unexpected network interfaces
blacklist eth1394

# snd_intel8x0m can interfere with snd_intel8x0, doesn't seem to support much
# hardware on its own (Ubuntu bug #2011, #6810)
blacklist snd_intel8x0m

# Conflicts with dvb driver (which is better for handling this device)
blacklist snd_aw2

# causes failure to suspend on HP compaq nc6000 (Ubuntu: #10306)
blacklist i2c_i801

# replaced by p54pci
blacklist prism54

# replaced by b43 and ssb.
blacklist bcm43xx

# most apps now use garmin usb driver directly (Ubuntu: #114565)
blacklist garmin_gps

# replaced by asus-laptop (Ubuntu: #184721)
blacklist asus_acpi

# low-quality, just noise when being used for sound playback, causes
# hangs at desktop session start (Ubuntu: #246969)
blacklist snd_pcsp

# ugly and loud noise, getting on everyone's nerves; this should be done by a
# nice pulseaudio bing (Ubuntu: #77010)
blacklist pcspkr

# EDAC driver for amd76x clashes with the agp driver preventing the aperture
# from being initialised (Ubuntu: #297750). Blacklist so that the driver
# continues to build and is installable for the few cases where its
# really needed.
blacklist amd76x_edac

go to /sys/module/<module name> directory to see current running environment parameters.

sysctl

We discussed that by manipulating /proc/sys/kernel we can change some current running kernel parameters. But as we said these settings are not permanent and they all gone after reboot. sysctl lets us make permanent settings . sysctl is loaded from /sbin/sysctl and read setting from /etc/sysctl.conf. As user we can change settings inside sysctl.conf file or make our custom file with desired parameters in /etc/sysctl.d/ directory. sysctl has a command interface and let us to see and write paramets trough /proc/sys/kernel/.

sysctl -a to show all tuneables.

root@server1:~# sysctl -a | wc
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.ens33.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
    947    2960   35450

that is a large file you can see.To set a new value to a parameter use sysctl -w :

root@server1:~# sysctl -a | grep swap
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.ens33.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.swappiness = 60
root@server1:~# sysctl -w vm.swappiness=80
vm.swappiness = 80
root@server1:~# sysctl -a | grep swap
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.ens33.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.swappiness = 80

Before quit lets take a look at sysctl.conf :

# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3

##############################################################3
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1

###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1

#net.ipv4.ip_forwarding = 1

as you can see includes /etc/sysctl.d/ directory setting files but don't for get that setting in sysctl.conf file always over write /etc/sysctl.d/ directory and Win :) You can uncomment or add setting weather in sysctl.conf or go and create custom file in /etc/sysctl.d/ directory but Take care of contrasts.

udev

During boot process when kernel is loaded and Device Driver start setting up hardware, next step of hardware initializing is udev. So kernel initial device loading and send uevents to udev daemon. udev keep these event and handle them base on attributes it has received in the event. Then udev take action based on its rules .there are two types of rules in two different locations :

/lib/udev/rules.d "default rules"

/etc/udev/rules.d "custom rules/ user defined"

After Finishing boot process udev is active and receive kernel messages an uevent messages from hot-pluggable devices.

udevadm monitor , udevadm

udev has a monitoring tool, and it monitor and shows event received from the kernel after hardware initializing and uevent which are udev events after processing udev rules, lets attach a usb and monitor what it shows:

root@server1:~# udevadm monitor 
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[3520.586975] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1 (usb)
KERNEL[3520.589831] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0 (usb)
UDEV  [3520.610064] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1 (usb)
KERNEL[3520.653543] add      /module/usb_storage (module)
UDEV  [3520.655921] add      /module/usb_storage (module)
KERNEL[3520.655983] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33 (scsi)
KERNEL[3520.656293] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/scsi_host/host33 (scsi_host)
KERNEL[3520.656671] add      /bus/usb/drivers/usb-storage (drivers)
UDEV  [3520.659555] add      /bus/usb/drivers/usb-storage (drivers)
KERNEL[3520.664336] add      /module/uas (module)
KERNEL[3520.664472] add      /bus/usb/drivers/uas (drivers)
UDEV  [3520.665071] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0 (usb)
UDEV  [3520.666828] add      /module/uas (module)
UDEV  [3520.669972] add      /bus/usb/drivers/uas (drivers)
UDEV  [3520.671676] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33 (scsi)
UDEV  [3520.673836] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/scsi_host/host33 (scsi_host)
KERNEL[3521.702618] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0 (scsi)
KERNEL[3521.703797] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0 (scsi)
KERNEL[3521.703877] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/scsi_disk/33:0:0:0 (scsi_disk)
KERNEL[3521.703990] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/scsi_device/33:0:0:0 (scsi_device)
KERNEL[3521.706049] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/scsi_generic/sg2 (scsi_generic)
KERNEL[3521.707046] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/bsg/33:0:0:0 (bsg)
UDEV  [3521.709816] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0 (scsi)
UDEV  [3521.715283] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0 (scsi)
UDEV  [3521.728137] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/scsi_disk/33:0:0:0 (scsi_disk)
UDEV  [3521.728500] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/scsi_device/33:0:0:0 (scsi_device)
UDEV  [3521.732039] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/bsg/33:0:0:0 (bsg)
UDEV  [3521.732504] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/scsi_generic/sg2 (scsi_generic)
KERNEL[3522.394218] add      /devices/virtual/bdi/8:16 (bdi)
UDEV  [3522.405809] add      /devices/virtual/bdi/8:16 (bdi)
KERNEL[3522.446634] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/block/sdb (block)
UDEV  [3522.871787] add      /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/block/sdb (block)
KERNEL[3523.061202] add      /module/nls_iso8859_1 (module)
UDEV  [3523.066143] add      /module/nls_iso8859_1 (module)

You can see both kernel messages and udev events. And finally our usb flash is mounted as sdb. For having more readable data:

root@server1:~# udevadm info --query=all --name=/dev/sdb
P: /devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/block/sdb
N: sdb
S: disk/by-id/usb-Kingston_DT_101_II_0013729982D5B97196320049-0:0
S: disk/by-label/MYLINUXLIVE
S: disk/by-path/pci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0
S: disk/by-uuid/3879-A38A
E: DEVLINKS=/dev/disk/by-uuid/3879-A38A /dev/disk/by-label/MYLINUXLIVE /dev/disk/by-id/usb-Kingston_DT_101_II_0013729982D5B97196320049-0:0 /dev/disk/by-path/pci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0
E: DEVNAME=/dev/sdb
E: DEVPATH=/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/block/sdb
E: DEVTYPE=disk
E: ID_BUS=usb
E: ID_FS_LABEL=MYLINUXLIVE
E: ID_FS_LABEL_ENC=MYLINUXLIVE
E: ID_FS_TYPE=vfat
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=3879-A38A
E: ID_FS_UUID_ENC=3879-A38A
E: ID_FS_VERSION=FAT32
E: ID_INSTANCE=0:0
E: ID_MODEL=DT_101_II
E: ID_MODEL_ENC=DT\x20101\x20II\x20\x20\x20\x20\x20\x20\x20
E: ID_MODEL_ID=1625
E: ID_PATH=pci-0000:02:03.0-usb-0:1:1.0-scsi-0:0:0:0
E: ID_PATH_TAG=pci-0000_02_03_0-usb-0_1_1_0-scsi-0_0_0_0
E: ID_REVISION=PMAP
E: ID_SERIAL=Kingston_DT_101_II_0013729982D5B97196320049-0:0
E: ID_SERIAL_SHORT=0013729982D5B97196320049
E: ID_TYPE=disk
E: ID_USB_DRIVER=usb-storage
E: ID_USB_INTERFACES=:080650:
E: ID_USB_INTERFACE_NUM=00
E: ID_VENDOR=Kingston
E: ID_VENDOR_ENC=Kingston
E: ID_VENDOR_ID=0951
E: MAJOR=8
E: MINOR=16
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=3522768898

use udevadm info attribute-walk --name=/dev/sdb | less to have summary of device attributes:

root@server1:~# udevadm info --attribute-walk --name=/dev/sdb

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/block/sdb':
    KERNEL=="sdb"
    SUBSYSTEM=="block"
    DRIVER==""
    ATTR{alignment_offset}=="0"
    ATTR{badblocks}==""
    ATTR{capability}=="51"
    ATTR{discard_alignment}=="0"
    ATTR{events}=="media_change"
    ATTR{events_async}==""
    ATTR{events_poll_msecs}=="-1"
    ATTR{ext_range}=="256"
    ATTR{inflight}=="       0        0"
    ATTR{range}=="16"
    ATTR{removable}=="1"
    ATTR{ro}=="0"
    ATTR{size}=="15679488"
    ATTR{stat}=="     344    15135    17586     3160        2        0        2      268        0     1792     3428"

  looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0':
    KERNELS=="33:0:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS=="sd"
    ATTRS{device_blocked}=="0"
    ATTRS{device_busy}=="0"
    ATTRS{dh_state}=="detached"
    ATTRS{eh_timeout}=="10"
    ATTRS{evt_capacity_change_reported}=="0"
    ATTRS{evt_inquiry_change_reported}=="0"
    ATTRS{evt_lun_change_reported}=="0"
    ATTRS{evt_media_change}=="0"
    ATTRS{evt_mode_parameter_change_reported}=="0"
    ATTRS{evt_soft_threshold_reached}=="0"
    ATTRS{inquiry}==""
    ATTRS{iocounterbits}=="32"
    ATTRS{iodone_cnt}=="0x590"
    ATTRS{ioerr_cnt}=="0x2"
    ATTRS{iorequest_cnt}=="0x590"
    ATTRS{max_sectors}=="240"
    ATTRS{model}=="DT 101 II       "
    ATTRS{queue_depth}=="1"
    ATTRS{queue_type}=="none"
    ATTRS{rev}=="PMAP"
    ATTRS{scsi_level}=="0"
    ATTRS{state}=="running"
    ATTRS{timeout}=="30"
    ATTRS{type}=="0"
    ATTRS{vendor}=="Kingston"

  looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0':
    KERNELS=="target33:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33':
    KERNELS=="host33"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0':
    KERNELS=="1-1:1.0"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb-storage"
    ATTRS{authorized}=="1"
    ATTRS{bAlternateSetting}==" 0"
    ATTRS{bInterfaceClass}=="08"
    ATTRS{bInterfaceNumber}=="00"
    ATTRS{bInterfaceProtocol}=="50"
    ATTRS{bInterfaceSubClass}=="06"
    ATTRS{bNumEndpoints}=="02"
    ATTRS{supports_autosuspend}=="1"

  looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1':
    KERNELS=="1-1"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="00"
    ATTRS{bDeviceProtocol}=="00"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="300mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0110"
    ATTRS{bmAttributes}=="80"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="6"
    ATTRS{devpath}=="1"
    ATTRS{idProduct}=="1625"
    ATTRS{idVendor}=="0951"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="Kingston"
    ATTRS{maxchild}=="0"
    ATTRS{product}=="DT 101 II"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="unknown"
    ATTRS{serial}=="0013729982D5B97196320049"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="3230"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1':
    KERNELS=="usb1"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{authorized_default}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="09"
    ATTRS{bDeviceProtocol}=="00"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="0mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0410"
    ATTRS{bmAttributes}=="e0"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="1"
    ATTRS{devpath}=="0"
    ATTRS{idProduct}=="0002"
    ATTRS{idVendor}=="1d6b"
    ATTRS{interface_authorized_default}=="1"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="Linux 4.10.0-40-generic ehci_hcd"
    ATTRS{maxchild}=="6"
    ATTRS{product}=="EHCI Host Controller"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="unknown"
    ATTRS{serial}=="0000:02:03.0"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="88"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0':
    KERNELS=="0000:02:03.0"
    SUBSYSTEMS=="pci"
    DRIVERS=="ehci-pci"
    ATTRS{acpi_index}=="16777752"
    ATTRS{broken_parity_status}=="0"
    ATTRS{class}=="0x0c0320"
    ATTRS{companion}==""
    ATTRS{consistent_dma_mask_bits}=="32"
    ATTRS{d3cold_allowed}=="0"
    ATTRS{device}=="0x0770"
    ATTRS{dma_mask_bits}=="32"
    ATTRS{driver_override}=="(null)"
    ATTRS{enable}=="1"
    ATTRS{irq}=="17"
    ATTRS{label}=="ehci"
    ATTRS{local_cpulist}=="0-3"
    ATTRS{local_cpus}=="00000000,00000000,00000000,0000000f"
    ATTRS{msi_bus}=="1"
    ATTRS{numa_node}=="-1"
    ATTRS{revision}=="0x00"
    ATTRS{subsystem_device}=="0x0770"
    ATTRS{subsystem_vendor}=="0x15ad"
    ATTRS{uframe_periodic_max}=="100"
    ATTRS{vendor}=="0x15ad"

  looking at parent device '/devices/pci0000:00/0000:00:11.0':
    KERNELS=="0000:00:11.0"
    SUBSYSTEMS=="pci"
    DRIVERS==""
    ATTRS{broken_parity_status}=="0"
    ATTRS{class}=="0x060401"
    ATTRS{consistent_dma_mask_bits}=="32"
    ATTRS{d3cold_allowed}=="0"
    ATTRS{device}=="0x0790"
    ATTRS{dma_mask_bits}=="32"
    ATTRS{driver_override}=="(null)"
    ATTRS{enable}=="1"
    ATTRS{irq}=="0"
    ATTRS{local_cpulist}=="0-3"
    ATTRS{local_cpus}=="00000000,00000000,00000000,0000000f"
    ATTRS{msi_bus}=="1"
    ATTRS{numa_node}=="-1"
    ATTRS{revision}=="0x02"
    ATTRS{subsystem_device}=="0x0790"
    ATTRS{subsystem_vendor}=="0x15ad"
    ATTRS{vendor}=="0x15ad"

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""

Oh. That is a long long list of attributes, but we notice that udev consider all of these attributes when it checks it rules!

Thanks udev, but udev dose another great job for us, can you remember? udev populate /dev/ and create Hardware Abstraction, which is used my other programs. Udev has made world easier place for us and other programs.Thank you udev :)

dmesg

In LPIC1 Course we discussed that during boot process (when operating system hasn't been started yet) dmesg logs kernel ring buffer. Let go deeper, during the boot process kernel is loaded into RAM. At this stage there is a Device Driver presents in kernel which set up to drive relevant hardware. As any other elements of kernel, device driver produce messages. These messages have information about modules presents and value of any parameters. As booting process is so fast, dmesg help us to review what has happened during boot process.

Even after booting kernel might generate some messages, for example when an I/O devicefacing the problem, or a hot-pluggable device like USB is attached to system. Again dmesg can be used to review what kernel has generated during doing its jobs.

tion="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session" pid=638 comm="apparmor_parser"
[    5.013133] audit: type=1400 audit(1512370178.278:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session//chromium" pid=638 comm="apparmor_parser"
[    5.032577] audit: type=1400 audit(1512370178.302:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince" pid=640 comm="apparmor_parser"
[    5.032579] audit: type=1400 audit(1512370178.302:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince//sanitized_helper" pid=640 comm="apparmor_parser"
[    5.032580] audit: type=1400 audit(1512370178.302:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince-previewer" pid=640 comm="apparmor_parser"
[    5.032581] audit: type=1400 audit(1512370178.302:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince-previewer//sanitized_helper" pid=640 comm="apparmor_parser"
[    5.032582] audit: type=1400 audit(1512370178.302:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince-thumbnailer" pid=640 comm="apparmor_parser"
[    5.032583] audit: type=1400 audit(1512370178.302:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince-thumbnailer//sanitized_helper" pid=640 comm="apparmor_parser"
[    5.046032] audit: type=1400 audit(1512370178.314:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/cups-browsed" pid=668 comm="apparmor_parser"
[    5.662430] random: crng init done
[    5.668377] Adding 1045500k swap on /dev/sda5.  Priority:-1 extents:1 across:1045500k FS
[    6.567089] IPv6: ADDRCONF(NETDEV_UP): ens33: link is not ready
[    6.574513] IPv6: ADDRCONF(NETDEV_UP): ens33: link is not ready
[    6.575691] e1000: ens33 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[    6.576992] IPv6: ADDRCONF(NETDEV_CHANGE): ens33: link becomes ready
[    6.866049] NET: Registered protocol family 40
[   12.878693] Bluetooth: Core ver 2.22
[   12.878785] NET: Registered protocol family 31
[   12.878787] Bluetooth: HCI device and connection manager initialized
[   12.878792] Bluetooth: HCI socket layer initialized
[   12.878795] Bluetooth: L2CAP socket layer initialized
[   12.878805] Bluetooth: SCO socket layer initialized
[   12.887893] Netfilter messages via NETLINK v0.30.
[   13.014635] device ens33 entered promiscuous mode
[ 2076.880855] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 2076.880856] Bluetooth: BNEP filters: protocol multicast
[ 2076.880860] Bluetooth: BNEP socket layer initialized
[ 2080.858589] ISO 9660 Extensions: Microsoft Joliet Level 3
[ 2080.875428] ISO 9660 Extensions: RRIP_1991A
[ 2098.732330] usb 1-1: new high-speed USB device number 2 using ehci-pci
[ 2098.920575] usb 1-1: device descriptor read/64, error 18
[ 2099.260910] usb 1-1: device descriptor read/64, error 18
[ 2099.597146] usb 1-1: new high-speed USB device number 3 using ehci-pci
[ 2099.839122] usb 1-1: device descriptor read/64, error 18
[ 2100.182250] usb 1-1: device descriptor read/64, error 18
[ 2100.522218] usb 1-1: new high-speed USB device number 4 using ehci-pci
[ 2100.546399] usb 1-1: Invalid ep0 maxpacket: 9
[ 2100.779187] usb 1-1: new high-speed USB device number 5 using ehci-pci
[ 2100.813183] usb 1-1: Invalid ep0 maxpacket: 9
[ 2100.819476] usb usb1-port1: unable to enumerate USB device

result above is abstracted, how ever you can see that dmesg dosn't workout well to give information about modules and parameters.

There are some user space commands from lpic1, lets just review:

command

Description

lspci

List PCi devices on the computer

lsusb

Shows Connected USB Devices

lsdev

List all Devices and Device Information

Last updated