201.3. Kernel runtime management and troubleshooting
201.3 Kernel runtime management and troubleshooting
Weight: 4
Description: Candidates should be able to manage and/or query a 2.6.x, 3.x or 4.x kernel and its loadable modules. Candidates should be able to identify and correct common boot and run time issues. Candidates should understand device detection and management using udev. This objective includes troubleshooting udev rules.
Key Knowledge Areas:
Use command-line utilities to get information about the currently running kernel and kernel modules
Manually load and unload kernel modules
Determine when modules can be unloaded
Determine what parameters a module accepts
Configure the system to load modules by names other than their file name.
/proc filesystem
Content of /, /boot/ , and /lib/modules/
Tools and utilities to analyze information about the available hardware
udev rules
Terms and Utilities:
/lib/modules/kernel-version/modules.dep
module configuration files in /etc/
/proc/sys/kernel/
/sbin/depmod
/sbin/rmmod
/sbin/modinfo
/bin/dmesg
/sbin/lspci
/usr/bin/lsdev
/sbin/lsmod
/sbin/modprobe
/sbin/insmod
/bin/uname
/usr/bin/lsusb
/etc/sysctl.conf, /etc/sysctl.d/
/sbin/sysctl
udevmonitor
udevadm monitor
/etc/udev/
Kernel Modules
Linux Kernel is modular since version 2.0. Being modular make simplicity and efficiency, the code which is not needed wouldn't be loaded. Kernel Modules have different conditions. A module might be compiled, or not. If it has already compiled it might be loaded or not. So Loading an unload but compiled kernel is easy, otherwise required kernel manipulation.We use CentOS7 for demonstartion:
lets try to load and unloaded module " jfs " that old journaling file system. for loading an unloaded but compiled module there are two solutions. An old solution is insmod , lets try it first
root@server1:~# insmod jfs
insmod: ERROR: could not load module jfs: No such file or directory
But it doesn't work :(( so let get more information about "jfs" module with modinfo:
root@server1:~# modinfo jfs
filename: /lib/modules/4.10.0-40-generic/kernel/fs/jfs/jfs.ko
alias: fs-jfs
license: GPL
author: Steve Best/Dave Kleikamp/Barry Arndt, IBM
description: The Journaled Filesystem (JFS)
srcversion: 3249CD436BA490FF745560A
depends:
intree: Y
vermagic: 4.10.0-40-generic SMP mod_unload
parm: nTxBlock:Number of transaction blocks (max:65536) (int)
parm: nTxLock:Number of transaction locks (max:65536) (int)
parm: commit_threads:Number of commit threads (int)
modinfo shows a bunch of information about a module. Like its filename, dependencies and possible parameters.Till now we have realize that "jfs" module has been compiled and its available but it is not loaded. inorder to load a module with insmod we have to specify the full patch of module:
But as you see using insmod is not that much easy, insmod has another problem and that is dependencies. "jfs" module has no dependency so we were able to load it very easily, if a module has some dependencies, insmod dosn't load those required dependencies automatically. So it would be administrator task. All these shortages cause we come to this conclusion that insmod is not perfect and we need a modern tool, which automatically know the patch and load required dependencies, that is modprobe.
also for removing use modprobe -r lp. But how modprobe realize module dependencies and act such a smart tool? modules.dep
root@server1:~# cd /lib/modules/
root@server1:/lib/modules# ls
4.10.0-28-generic 4.10.0-40-generic
root@server1:/lib/modules# cd 4.10.0-40-generic/
root@server1:/lib/modules/4.10.0-40-generic# ls
build modules.alias.bin modules.dep.bin modules.symbols
initrd modules.builtin modules.devname modules.symbols.bin
kernel modules.builtin.bin modules.order vdso
modules.alias modules.dep modules.softdep
root@server1:/lib/modules/4.10.0-40-generic# depmod -a
we can use depmod -a to create and update modules.dep with all new modules and dependencies. Now that we have learned alot of tools, let play game with cdrom parameters [in ubunto cdrom is not a module, so use other distro]:
We as users like modules to load automatically, mean while parameters should be loaded automatically, to make parameters persistence create a file in /etc/modprobe.d :
root@server1:~# cd /etc/modprobe.d/
root@server1:/etc/modprobe.d# ls
alsa-base.conf blacklist-framebuffer.conf blacklist-watchdog.conf
blacklist-ath_pci.conf blacklist-modem.conf fbdev-blacklist.conf
blacklist.conf blacklist-oss.conf iwlwifi.conf
blacklist-firewire.conf blacklist-rare-network.conf mlx4.conf
root@server1:/etc/modprobe.d# vi cdroom-lookdoor.conf
and insert:
options cdroom lookdoor=1
If you have noticed there is blacklist.conf file in /etc/modfprobe.d , if you don't wana let a specific module be loaded use it:
# This file lists those modules which we don't want to be loaded by
# alias expansion, usually so some other driver will be loaded for the
# device instead.
# evbug is a debug tool that should be loaded explicitly
blacklist evbug
# these drivers are very simple, the HID drivers are usually preferred
blacklist usbmouse
blacklist usbkbd
# replaced by e100
blacklist eepro100
# replaced by tulip
blacklist de4x5
# causes no end of confusion by creating unexpected network interfaces
blacklist eth1394
# snd_intel8x0m can interfere with snd_intel8x0, doesn't seem to support much
# hardware on its own (Ubuntu bug #2011, #6810)
blacklist snd_intel8x0m
# Conflicts with dvb driver (which is better for handling this device)
blacklist snd_aw2
# causes failure to suspend on HP compaq nc6000 (Ubuntu: #10306)
blacklist i2c_i801
# replaced by p54pci
blacklist prism54
# replaced by b43 and ssb.
blacklist bcm43xx
# most apps now use garmin usb driver directly (Ubuntu: #114565)
blacklist garmin_gps
# replaced by asus-laptop (Ubuntu: #184721)
blacklist asus_acpi
# low-quality, just noise when being used for sound playback, causes
# hangs at desktop session start (Ubuntu: #246969)
blacklist snd_pcsp
# ugly and loud noise, getting on everyone's nerves; this should be done by a
# nice pulseaudio bing (Ubuntu: #77010)
blacklist pcspkr
# EDAC driver for amd76x clashes with the agp driver preventing the aperture
# from being initialised (Ubuntu: #297750). Blacklist so that the driver
# continues to build and is installable for the few cases where its
# really needed.
blacklist amd76x_edac
go to /sys/module/<module name> directory to see current running environment parameters.
sysctl
We discussed that by manipulating /proc/sys/kernel we can change some current running kernel parameters. But as we said these settings are not permanent and they all gone after reboot. sysctl lets us make permanent settings . sysctl is loaded from /sbin/sysctl and read setting from /etc/sysctl.conf. As user we can change settings inside sysctl.conf file or make our custom file with desired parameters in /etc/sysctl.d/ directory. sysctl has a command interface and let us to see and write paramets trough /proc/sys/kernel/.
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#
#kernel.domainname = example.com
# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3
##############################################################3
# Functions previously found in netbase
#
# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1
# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1
# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1
# Uncomment the next line to enable packet forwarding for IPv6
# Enabling this option disables Stateless Address Autoconfiguration
# based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1
###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1
#net.ipv4.ip_forwarding = 1
as you can see includes /etc/sysctl.d/ directory setting files but don't for get that setting in sysctl.conf file always over write /etc/sysctl.d/ directory and Win :) You can uncomment or add setting weather in sysctl.conf or go and create custom file in /etc/sysctl.d/ directory but Take care of contrasts.
udev
During boot process when kernel is loaded and Device Driver start setting up hardware, next step of hardware initializing is udev. So kernel initial device loading and send uevents to udev daemon. udev keep these event and handle them base on attributes it has received in the event. Then udev take action based on its rules .there are two types of rules in two different locations :
/lib/udev/rules.d "default rules"
/etc/udev/rules.d "custom rules/ user defined"
After Finishing boot process udev is active and receive kernel messages an uevent messages from hot-pluggable devices.
udevadm monitor , udevadm
udev has a monitoring tool, and it monitor and shows event received from the kernel after hardware initializing and uevent which are udev events after processing udev rules, lets attach a usb and monitor what it shows:
use udevadm info attribute-walk --name=/dev/sdb | less to have summary of device attributes:
root@server1:~# udevadm info --attribute-walk --name=/dev/sdb
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0/block/sdb':
KERNEL=="sdb"
SUBSYSTEM=="block"
DRIVER==""
ATTR{alignment_offset}=="0"
ATTR{badblocks}==""
ATTR{capability}=="51"
ATTR{discard_alignment}=="0"
ATTR{events}=="media_change"
ATTR{events_async}==""
ATTR{events_poll_msecs}=="-1"
ATTR{ext_range}=="256"
ATTR{inflight}==" 0 0"
ATTR{range}=="16"
ATTR{removable}=="1"
ATTR{ro}=="0"
ATTR{size}=="15679488"
ATTR{stat}==" 344 15135 17586 3160 2 0 2 268 0 1792 3428"
looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0/33:0:0:0':
KERNELS=="33:0:0:0"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{device_blocked}=="0"
ATTRS{device_busy}=="0"
ATTRS{dh_state}=="detached"
ATTRS{eh_timeout}=="10"
ATTRS{evt_capacity_change_reported}=="0"
ATTRS{evt_inquiry_change_reported}=="0"
ATTRS{evt_lun_change_reported}=="0"
ATTRS{evt_media_change}=="0"
ATTRS{evt_mode_parameter_change_reported}=="0"
ATTRS{evt_soft_threshold_reached}=="0"
ATTRS{inquiry}==""
ATTRS{iocounterbits}=="32"
ATTRS{iodone_cnt}=="0x590"
ATTRS{ioerr_cnt}=="0x2"
ATTRS{iorequest_cnt}=="0x590"
ATTRS{max_sectors}=="240"
ATTRS{model}=="DT 101 II "
ATTRS{queue_depth}=="1"
ATTRS{queue_type}=="none"
ATTRS{rev}=="PMAP"
ATTRS{scsi_level}=="0"
ATTRS{state}=="running"
ATTRS{timeout}=="30"
ATTRS{type}=="0"
ATTRS{vendor}=="Kingston"
looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33/target33:0:0':
KERNELS=="target33:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0/host33':
KERNELS=="host33"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1/1-1:1.0':
KERNELS=="1-1:1.0"
SUBSYSTEMS=="usb"
DRIVERS=="usb-storage"
ATTRS{authorized}=="1"
ATTRS{bAlternateSetting}==" 0"
ATTRS{bInterfaceClass}=="08"
ATTRS{bInterfaceNumber}=="00"
ATTRS{bInterfaceProtocol}=="50"
ATTRS{bInterfaceSubClass}=="06"
ATTRS{bNumEndpoints}=="02"
ATTRS{supports_autosuspend}=="1"
looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1':
KERNELS=="1-1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="00"
ATTRS{bDeviceProtocol}=="00"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="300mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0110"
ATTRS{bmAttributes}=="80"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="6"
ATTRS{devpath}=="1"
ATTRS{idProduct}=="1625"
ATTRS{idVendor}=="0951"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="Kingston"
ATTRS{maxchild}=="0"
ATTRS{product}=="DT 101 II"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="unknown"
ATTRS{serial}=="0013729982D5B97196320049"
ATTRS{speed}=="480"
ATTRS{urbnum}=="3230"
ATTRS{version}==" 2.00"
looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1':
KERNELS=="usb1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{authorized_default}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="09"
ATTRS{bDeviceProtocol}=="00"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="0mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0410"
ATTRS{bmAttributes}=="e0"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="1"
ATTRS{devpath}=="0"
ATTRS{idProduct}=="0002"
ATTRS{idVendor}=="1d6b"
ATTRS{interface_authorized_default}=="1"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="Linux 4.10.0-40-generic ehci_hcd"
ATTRS{maxchild}=="6"
ATTRS{product}=="EHCI Host Controller"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="unknown"
ATTRS{serial}=="0000:02:03.0"
ATTRS{speed}=="480"
ATTRS{urbnum}=="88"
ATTRS{version}==" 2.00"
looking at parent device '/devices/pci0000:00/0000:00:11.0/0000:02:03.0':
KERNELS=="0000:02:03.0"
SUBSYSTEMS=="pci"
DRIVERS=="ehci-pci"
ATTRS{acpi_index}=="16777752"
ATTRS{broken_parity_status}=="0"
ATTRS{class}=="0x0c0320"
ATTRS{companion}==""
ATTRS{consistent_dma_mask_bits}=="32"
ATTRS{d3cold_allowed}=="0"
ATTRS{device}=="0x0770"
ATTRS{dma_mask_bits}=="32"
ATTRS{driver_override}=="(null)"
ATTRS{enable}=="1"
ATTRS{irq}=="17"
ATTRS{label}=="ehci"
ATTRS{local_cpulist}=="0-3"
ATTRS{local_cpus}=="00000000,00000000,00000000,0000000f"
ATTRS{msi_bus}=="1"
ATTRS{numa_node}=="-1"
ATTRS{revision}=="0x00"
ATTRS{subsystem_device}=="0x0770"
ATTRS{subsystem_vendor}=="0x15ad"
ATTRS{uframe_periodic_max}=="100"
ATTRS{vendor}=="0x15ad"
looking at parent device '/devices/pci0000:00/0000:00:11.0':
KERNELS=="0000:00:11.0"
SUBSYSTEMS=="pci"
DRIVERS==""
ATTRS{broken_parity_status}=="0"
ATTRS{class}=="0x060401"
ATTRS{consistent_dma_mask_bits}=="32"
ATTRS{d3cold_allowed}=="0"
ATTRS{device}=="0x0790"
ATTRS{dma_mask_bits}=="32"
ATTRS{driver_override}=="(null)"
ATTRS{enable}=="1"
ATTRS{irq}=="0"
ATTRS{local_cpulist}=="0-3"
ATTRS{local_cpus}=="00000000,00000000,00000000,0000000f"
ATTRS{msi_bus}=="1"
ATTRS{numa_node}=="-1"
ATTRS{revision}=="0x02"
ATTRS{subsystem_device}=="0x0790"
ATTRS{subsystem_vendor}=="0x15ad"
ATTRS{vendor}=="0x15ad"
looking at parent device '/devices/pci0000:00':
KERNELS=="pci0000:00"
SUBSYSTEMS==""
DRIVERS==""
Oh. That is a long long list of attributes, but we notice that udev consider all of these attributes when it checks it rules!
Thanks udev, but udev dose another great job for us, can you remember? udev populate /dev/ and create Hardware Abstraction, which is used my other programs. Udev has made world easier place for us and other programs.Thank you udev :)
dmesg
In LPIC1 Course we discussed that during boot process (when operating system hasn't been started yet) dmesg logs kernel ring buffer. Let go deeper, during the boot process kernel is loaded into RAM. At this stage there is a Device Driver presents in kernel which set up to drive relevant hardware. As any other elements of kernel, device driver produce messages. These messages have information about modules presents and value of any parameters. As booting process is so fast, dmesg help us to review what has happened during boot process.
Even after booting kernel might generate some messages, for example when an I/O devicefacing the problem, or a hot-pluggable device like USB is attached to system. Again dmesg can be used to review what kernel has generated during doing its jobs.
tion="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session" pid=638 comm="apparmor_parser"
[ 5.013133] audit: type=1400 audit(1512370178.278:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session//chromium" pid=638 comm="apparmor_parser"
[ 5.032577] audit: type=1400 audit(1512370178.302:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince" pid=640 comm="apparmor_parser"
[ 5.032579] audit: type=1400 audit(1512370178.302:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince//sanitized_helper" pid=640 comm="apparmor_parser"
[ 5.032580] audit: type=1400 audit(1512370178.302:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince-previewer" pid=640 comm="apparmor_parser"
[ 5.032581] audit: type=1400 audit(1512370178.302:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince-previewer//sanitized_helper" pid=640 comm="apparmor_parser"
[ 5.032582] audit: type=1400 audit(1512370178.302:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince-thumbnailer" pid=640 comm="apparmor_parser"
[ 5.032583] audit: type=1400 audit(1512370178.302:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/evince-thumbnailer//sanitized_helper" pid=640 comm="apparmor_parser"
[ 5.046032] audit: type=1400 audit(1512370178.314:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/cups-browsed" pid=668 comm="apparmor_parser"
[ 5.662430] random: crng init done
[ 5.668377] Adding 1045500k swap on /dev/sda5. Priority:-1 extents:1 across:1045500k FS
[ 6.567089] IPv6: ADDRCONF(NETDEV_UP): ens33: link is not ready
[ 6.574513] IPv6: ADDRCONF(NETDEV_UP): ens33: link is not ready
[ 6.575691] e1000: ens33 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 6.576992] IPv6: ADDRCONF(NETDEV_CHANGE): ens33: link becomes ready
[ 6.866049] NET: Registered protocol family 40
[ 12.878693] Bluetooth: Core ver 2.22
[ 12.878785] NET: Registered protocol family 31
[ 12.878787] Bluetooth: HCI device and connection manager initialized
[ 12.878792] Bluetooth: HCI socket layer initialized
[ 12.878795] Bluetooth: L2CAP socket layer initialized
[ 12.878805] Bluetooth: SCO socket layer initialized
[ 12.887893] Netfilter messages via NETLINK v0.30.
[ 13.014635] device ens33 entered promiscuous mode
[ 2076.880855] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 2076.880856] Bluetooth: BNEP filters: protocol multicast
[ 2076.880860] Bluetooth: BNEP socket layer initialized
[ 2080.858589] ISO 9660 Extensions: Microsoft Joliet Level 3
[ 2080.875428] ISO 9660 Extensions: RRIP_1991A
[ 2098.732330] usb 1-1: new high-speed USB device number 2 using ehci-pci
[ 2098.920575] usb 1-1: device descriptor read/64, error 18
[ 2099.260910] usb 1-1: device descriptor read/64, error 18
[ 2099.597146] usb 1-1: new high-speed USB device number 3 using ehci-pci
[ 2099.839122] usb 1-1: device descriptor read/64, error 18
[ 2100.182250] usb 1-1: device descriptor read/64, error 18
[ 2100.522218] usb 1-1: new high-speed USB device number 4 using ehci-pci
[ 2100.546399] usb 1-1: Invalid ep0 maxpacket: 9
[ 2100.779187] usb 1-1: new high-speed USB device number 5 using ehci-pci
[ 2100.813183] usb 1-1: Invalid ep0 maxpacket: 9
[ 2100.819476] usb usb1-port1: unable to enumerate USB device
result above is abstracted, how ever you can see that dmesg dosn't workout well to give information about modules and parameters.
There are some user space commands from lpic1, lets just review:
