# 4.3 Where Data is Stored

### **4.3 Where Data is Stored**&#x20;

**Weight:** 3

**Description:** Where various types of information are stored on a Linux system.

**Key Knowledge Areas:**

* Programs and configuration
* Processes
* Memory addresses
* System messaging
* Logging

**The following is a partial list of the used files, terms and utilities:**

* ps, top, free
* syslog, dmesg
* /etc/, /var/log/
* /boot/, /proc/, /dev/, /sys/

## How a computer uses memory

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2FOglRDibhRVXoDTsdcxDp%2Fimage.png?alt=media&#x26;token=281e2239-c586-4c3d-8224-2ef27ca7ed17" alt=""><figcaption></figcaption></figure>

This image shows a simple hierarchical view of the main components of computer architecture. At the top are the **input devices**, such as the mouse, keyboard, scanner, camera, microphone, and video input, whose job is to transfer data and commands from the user to the system. After entering the system, this data is usually sent to lower layers for processing.

In the middle section, **storage devices** are shown, including read-only memory (ROM), removable storage, hard drives, and network storage. These types of memory provide **permanent storage**, meaning the data remains even after the system is turned off. Below this part is **RAM**, which acts as temporary storage and includes both physical RAM and virtual memory. This is where programs keep their data while running, and it offers much faster access compared to permanent storage.&#x20;

At the bottom are the **cache** and **CPU**. The cache is the fastest type of memory and stores the data the CPU needs most often, while the CPU is the central unit that processes and executes instructions. The vertical arrow on the left represents the **data access speed hierarchy**: the closer we are to the CPU, the faster (but smaller) the memory; the further we move toward permanent storage, the larger but slower it becomes. This structure shows how different components work together to transfer and process data in a computer system.

### free

The \`free\` command in Linux is the one that facilitates with providing the overview of system memory utilization. It displays all the details regarding the RAM usage such as how is the total, what is used, and free memory including buffers and cached data, aiding in real-time monitoring of memory resources. It acts as essential command for administrators and users to assess system performance, allocate resources effectively, and identify potential memory-related issues promptly.

#### **Syntax:** <a href="#syntax" id="syntax"></a>

The basic syntax of the "free" command is as follows:

```
free [OPTION]
```

**Basic Usage of 'free' Command**

```
free
```

```
[payam@earth ~]$ free
               total        used        free      shared  buff/cache   available
Mem:        15888864     7154488     5575548     1143072     4646800     8734376
Swap:              0           0           0
```

Free command without any option shows the used and free space of swap and physical memory in **KB**

When no option is used then free command produces the columnar output as shown above where column:

1. **total** displays the total installed memory (MemTotal and SwapTotal *i.e* present in /proc/meminfo).
2. **used** displays the used memory.
3. **free** displays the unused memory.
4. **shared** displays the memory used by tmpfs(Shmen i.e. present in '/proc/meminfo' and displays zero in case not available).
5. **buffers** displays the memory used by kernel buffers.
6. **cached** displays the memory used by the page cache and slabs(Cached and Slab available in '/proc/meminfo').
7. **buffers/cache** displays the sum of buffers and cache.

**Common Options of 'free' Command**

| **Options**   | **Description**                                                                                                               |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| -k, --kilo    | Displays memory usage in kilobytes (default).                                                                                 |
| -m, --mega    | Displays memory usage in megabytes.                                                                                           |
| -g, --giga    | Displays memory usage in gigabytes.                                                                                           |
| --tera        | Displays memory usage in terabytes.                                                                                           |
| -h, --human   | Automatically scales all output columns to the shortest three-digit unit and displays the units (B, K, M, G, T).              |
| -c, --count   | Displays the output 'c' number of times; works with the -s option.                                                            |
| -l, --lohi    | Shows detailed low and high memory statistics.                                                                                |
| -o, --old     | Disables the display of the buffer-adjusted line.                                                                             |
| -s, --seconds | Continuously displays the output after 's' seconds delay. Uses the usleep system call for microsecond resolution delay times. |
| -t, --total   | Adds an additional line in the output showing column totals.                                                                  |
| --help        | Displays a help message and exits.                                                                                            |
| -V, --version | Displays version information and exits.                                                                                       |

## List Running Processes in Linux

We can use multiple commands to list the running processes in Linux like **ps**, **top**, **htop**,and **atop** commands in Linux. We can also have a combination of commands to list the running processes in Linux.

### ps

The `ps` command in Linux is used to display information about the currently running processes on the system.

* `ps` stands for process status.
* It shows details like PID, user, CPU, memory usage, and the command that started the process.
* By default, it displays processes running in the current shell.
* Use options to view more detailed or system-wide process information.
* Common formats include standard (`ps`), user-based (`ps -u`), and full system (`ps -ef` or `ps aux`).
* Often combined with `grep` to find specific processes.
* Useful for monitoring and troubleshooting running applications and services.

syntax:

The `ps` command provides a snapshot of the current processes on your system. The basic syntax is as follows:

```
ps [options]
```

Without any options, \`**`` ps` ``** displays information about the processes associated with the current terminal session. However, to harness the full potential of the \`**`` ps` ``** command, various options can be used to customize the output.

```
[payam@earth ~]$ ps
    PID TTY          TIME CMD
  17172 pts/0    00:00:00 bash
  18663 pts/0    00:00:00 ps
```

Result contains four columns of information. Where,&#x20;

* **PID -** the unique process ID&#x20;
* **TTY -** terminal type that the user is logged into&#x20;
* **TIME -** amount of CPU in minutes and seconds that the process has been running&#x20;
* **CMD -** name of the command that launched the process.&#x20;

#### ps command options

Some commonly used options:

| **Options**          | **Description**                                                                                                                                        |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **`a`**              | List all ruining processes for all users.                                                                                                              |
| **`-A, -e`**         | Lists all processes on the entire system, offering a complete overview of running tasks and programs.                                                  |
| **`-a`**             | List all processes except session leaders (instances where the process ID is the same as the session ID) and processes not associated with a terminal. |
| **`-d`**             | Lists all processes except session leaders, providing a filtered view of processes running on the system.                                              |
| **`--deselect, -N`** | Lists all processes except those that meet specific user-defined conditions.                                                                           |
| **`f`**              | Displays the hierarchy of processes in a visual ASCII art format, illustrating parent-child relationships.                                             |
| **`-j`**             | Presents the output in the jobs format, providing detailed information such as process ID, session ID, and command.                                    |
| **`T`**              | Lists all processes associated with the current terminal, aiding in focusing on tasks related to a specific terminal.                                  |
| **`r`**              | Only lists running processes, useful for monitoring system performance.                                                                                |
| **`u`**              | Expands the output to include additional information like CPU and memory usage.                                                                        |
| **`-u`**             | Specifies a username, listing processes associated with that user.                                                                                     |
| **`x`**              | Includes processes without a TTY, showing background processes not tied to a specific terminal session.                                                |

### top

the \``` top` ``  (**t**able **o**f **p**rocesses) command is a dynamic and interactive tool that provides real-time information about system processes. It offers a comprehensive view of running processes, system resource utilization, and other critical system metrics. This article explores how to effectively use the `top` command to monitor and manage processes.

#### Launching `top` <a href="#launching-top" id="launching-top"></a>

```
top
```

```
top - 10:58:42 up  2:32,  2 users,  load average: 1.40, 1.10, 0.87
Tasks: 367 total,   1 running, 366 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us,  1.0 sy,  0.0 ni, 93.9 id,  0.1 wa,  0.5 hi,  0.2 si,  0.0 st
MiB Mem :  15516.5 total,   4627.3 free,   7730.6 used,   4681.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   7785.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                
   7799 payam     20   0 1376.7g 439592 147300 S  17.9   2.8   3:21.63 chrome                                                                                                                                                                 
   6318 payam     20   0 5811236 278764 138288 S   8.9   1.8   5:44.53 gnome-shell                                                                                                                                                            
   7263 payam     20   0   33.2g 490484 293088 S   3.3   3.1   3:13.89 chrome                                                                                                                                                                 
   7355 payam     20   0   49.2g 252328 117636 S   2.6   1.6   3:14.22 chrome                                                                                                                                                                 
  11145 payam     20   0 1376.6g 195044 131940 S   2.0   1.2   2:07.53 chrome                                                                                                                                                                 
  17117 payam     20   0  770188  57012  44828 S   2.0   0.4   0:02.27 gnome-terminal-                                                                                                                                                        
  11803 payam     20   0 2835736 183248 134372 S   1.3   1.2   0:31.60 WebExtensions                                                                                                                                                          
  12220 payam     20   0 3213812 524624 149172 S   1.0   3.3   4:40.07 Isolated Web Co                                                                                                                                                        
  18900 payam     20   0  226072   4548   3460 R   0.7   0.0   0:00.08 top                                                                                                                                                                    
    922 root      20   0   82796   4612   4156 S   0.3   0.0   0:00.64 irqbalance                                                                                                                                                             
   6816 payam     20   0  527916  12920   6752 S   0.3   0.1   0:05.11 ibus-daemon                                                                                                                                                            
   7539 payam     20   0   32.7g 149928 113032 S   0.3   0.9   0:40.77 chrome                                                                                                                                                                 
  14104 root       0 -20       0      0      0 D   0.3   0.0   0:07.10 kworker/u33:3+i915_flip                                                                                                                                                
  14333 payam     20   0 1378.3g 319776 147536 S   0.3   2.0   0:40.73 chrome                                                                                                                                                                 
  15775 root       0 -20       0      0      0 I   0.3   0.0   0:04.55 kworker/u33:1-i915_flip                                                                                                                                                
  17887 root      20   0       0      0      0 I   0.3   0.0   0:00.40 kworker/u32:1-i915                                                                                                                                                     
      1 root      20   0  175392  18172  11328 S   0.0   0.1   0:01.73 systemd                                                                                                                                                                
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd                                                                                                                                                               
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_                                                                                                                                                        
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_g                                                                                                                                                        
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-sync_                                                                                                                                                        
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-slub_                                                                                                                                                        
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-netns                                                                                                                                                        
      9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri        
```

**Hoe to Intrepret top Command Output**

The **`top`** command output is divided into several sections, each with specific information about system performance and processes. This section provides a breakdown of the output based on the information it shows.

#### Uptime <a href="#uptime" id="uptime"></a>

When you first open the `top` command, the initial line, often referred to as the header or summary line, displays information similar to what you see when you use the `uptime` command. It shows:

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2FFARnsh8K2sMZtpQat3nO%2Fimage.png?alt=media&#x26;token=5e4dc434-19e4-47bf-aa30-dd79ec60d928" alt=""><figcaption></figcaption></figure>

1. **System Time**: The current time on the system.
2. **Uptime**: How long the system has been running since the last boot.
3. **Users**: The number of users currently logged into the system.
4. **Load Average**: This is shown in three numbers separated by commas. These numbers represent the average number of processes waiting for CPU time over the last 1 minute, 5 minutes, and 15 minutes, respectively. A value of 1.0 means the system's CPU is fully utilized; higher values indicate potential overloading.

#### tasks

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2FKZwmI3nHlFtTSDfdwxsD%2Fimage.png?alt=media&#x26;token=16fd4202-9ff9-465a-b7f3-e2017bb27022" alt=""><figcaption></figcaption></figure>

* **total**: Indicates the total count of processes currently being tracked by the system.
* **running**: Represents the number of processes currently actively using CPU time.
* **sleeping**: Refers to processes that are currently idle and waiting for a signal to wake up.
* **stopped**: Denotes processes that have been manually stopped, typically through a signal.
* **zombie**: Indicates processes that have completed execution but still have an entry in the process table.

#### %Cpu(s): <a href="#cpus" id="cpus"></a>

The `%Cpu(s)` line in the `top` command provides information about CPU usage and statistics on a Linux system. It typically includes:

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2F00gj4WeWMj7zLxAJ2FwY%2Fimage.png?alt=media&#x26;token=51bc2238-245d-45d0-9ee9-8747bab30b81" alt=""><figcaption></figcaption></figure>

* **us**: Percentage of CPU time spent running user processes.
* **sy**: Percentage of CPU time spent running kernel (system) processes.
* **ni**: Percentage of CPU time spent running processes with a nice value (priority adjusted).
* **id**: Percentage of CPU time spent idle (no work being done).
* **wa**: Percentage of CPU time spent waiting for I/O operations to complete.
* **hi**: Percentage of CPU time spent servicing hardware interrupts.
* **si**: Percentage of CPU time spent servicing software interrupts.
* **st**: Percentage of CPU time stolen from this virtual machine by the hypervisor (if virtualized).

#### MiB Memory: <a href="#mib-memory" id="mib-memory"></a>

The "MiB Memory" line in the `top` command provides information about memory usage and statistics on a Linux system. It typically includes:

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2Fgo5jEy14IkPJi8zxP4WK%2Fimage.png?alt=media&#x26;token=9adb1cc9-ab8c-407c-878b-2e6ff93a17c1" alt=""><figcaption></figcaption></figure>

* **total**: Total amount of physical memory (RAM) available in MiB.
* **used**: Amount of RAM currently in use by processes and the kernel.
* **free**: Amount of RAM not being used at all.
* **buff/cache**: Amount of memory used for buffering data and caching filesystems.

#### Mib Swap: <a href="#m" id="m"></a>

The "MiB Swap" line in the `top` command provides information about swap usage and statistics on a Linux system. It typically includes:

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2FHqL8XJaU3lbSy1fjS29K%2Fimage.png?alt=media&#x26;token=fd0cf178-8707-4459-b9e0-b58ac438ac82" alt=""><figcaption></figcaption></figure>

* **total**: Total amount of swap space available in MiB.
* **used**: Amount of swap space currently in use.
* **free**: Amount of swap space that is not being used.
* **available**: Estimate of how much memory is available for starting new applications without swapping.

***

## Linux Directory Structure

In Linux, everything is treated as a file even if it is a normal file, a directory, or even a device such as a printer or keyboard. All the directories and files are stored under one root directory which is represented by a forward slash /.

* The Linux directory layout follows the Filesystem Hierarchy Standard (FHS).
* This standard defines how directories are organized and what types of files should be stored in each.
* Since Linux is based on UNIX, it inherits many of UNIX’s filesystem conventions.
* Similar directory structures are also found in other UNIX-like operating systems such as BSD and macOS.

We know that in a Windows-like operating system, files are stored in different folders on different data drives like C: D: E:, whereas in the Linux/Unix operating system, files are stored in a tree-like structure starting with the root directory, as shown in the diagram below.&#x20;

The Linux/Unix file system hierarchy base begins at the root and everything starts with the root directory.&#x20;

#### **Top-level directories associated with the root directory** <a href="#these-are-the-common-toplevel-directories-associated-with-the-root-directory" id="these-are-the-common-toplevel-directories-associated-with-the-root-directory"></a>

These top-level directories under the root (`/`) form the foundation of the Linux file system, each serving a specific role in organizing system files, user data, and configurations.

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2FDIoOiUaqsIyHY37v72ba%2Fimage.png?alt=media&#x26;token=8b7d3e35-751a-4a90-a566-48768dd4176e" alt=""><figcaption></figcaption></figure>

## Linux File System Structure <a href="#linux-file-system-structure" id="linux-file-system-structure"></a>

The architecture of a file system comprises three layers mentioned below.

#### 1. Logical File System: <a href="#id-1-logical-file-system" id="id-1-logical-file-system"></a>

The Logical File System acts as the interface between the user applications and the file system itself. It facilitates essential operations such as opening, reading, and closing files. Essentially, it serves as the user-friendly front-end, ensuring that applications can interact with the file system in a way that aligns with user expectations.

#### 2. Virtual File System: <a href="#id-2-virtual-file-system" id="id-2-virtual-file-system"></a>

The Virtual File System (VFS) is a crucial layer that enables the concurrent operation of multiple instances of physical file systems. It provides a standardized interface, allowing different file systems to coexist and operate simultaneously. This layer abstracts the underlying complexities, ensuring compatibility and cohesion between various file system implementations.

#### 3. Physical File System: <a href="#id-3-physical-file-system" id="id-3-physical-file-system"></a>

The Physical File System is responsible for the tangible management and storage of physical memory blocks on the disk. It handles the low-level details of storing and retrieving data, interacting directly with the hardware components. This layer ensures the efficient allocation and utilization of physical storage resources, contributing to the overall performance and reliability of the file system.

Together, these layers form a cohesive architecture, orchestrating the organized and efficient handling of data in the Linux operating system.

### Virtual File Systems

### /dev

The */dev/* directory consists of files that represent devices that are attached to the local system. However, these are not regular files that a user can read and write to; these files are called devices files or special files:

```
ls -l /dev
```

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2FIApTFzJhyz03Aqshupw6%2Fimage.png?alt=media&#x26;token=bb33f9ee-7b8f-46f0-b2af-7cce923fb9b7" alt=""><figcaption></figcaption></figure>

**Device files are abstractions of standard devices that applications interact with via I/O system calls.** The device files that correspond to hardware devices fall into two main categories. Mainly character special files and block special files.

{% hint style="success" %}

### 3. The Difference Between Character Special Files and Block Special Files? <a href="#bd-what-is-the-difference-between-character-special-files-and-block-special-files" id="bd-what-is-the-difference-between-character-special-files-and-block-special-files"></a>

Character special files are simple interfaces to character devices. Likewise, block special files are simple interfaces to block devices. The difference between these devices depends on how the operating system reads data off of them. **A driver communicates with a character device by sending single characters as data such as bytes.** In addition, character devices don’t require buffering when communicating with a driver. On the other hand, a driver accesses data from block devices through a cache. Moreover, **a driver communicates with a block device by sending an entire block of data**. For example, character devices are sound cards or serial ports, whereas block devices are hard disks or USBs. We identify block and character devices by the letter that appears in front of the permissions. The letter *‘b’* that is displayed in the first column denotes a block device. On the other hand, the letter ‘c’ shown in the first column symbolizes a character device.
{% endhint %}

### /proc

Proc file system (procfs) is a virtual file system created on the fly when the system boots and is dissolved at the time of system shutdown. It contains useful information about the processes that are currently running, it is regarded as a control and information center for the kernel. The proc file system also provides a communication medium between kernel space and user space.

To List all the files and directories under the \`/proc\` directory.

```
ls -l /proc
```

This command will list all the files and directories under the `/proc` directory with detailed information like permissions, ownership, size, and time of modifications. This information is useful for understanding the current state of our system and diagnosing problems that are related to the running processes.

<figure><img src="https://4191340380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fohrzk4YVVEIEOw3rrQKr%2Fuploads%2FZKdHMTkBwqSv3uYVA1ac%2Fimage.png?alt=media&#x26;token=5193a312-731a-4e61-8f27-9ae489937217" alt=""><figcaption></figcaption></figure>

### /sys

*/sys* is another virtual directory like */proc* and */dev* and also contains information from devices connected to your computer.

#### **Key Differences Between `/proc` and `/sys`**

| Feature      | `/proc` (procfs)                                 | `/sys` (sysfs)                                             |
| ------------ | ------------------------------------------------ | ---------------------------------------------------------- |
| Purpose      | Process and system runtime information           | Kernel and hardware interaction                            |
| Type         | Virtual filesystem (procfs)                      | Virtual filesystem (sysfs)                                 |
| Content      | Process details, kernel parameters, system stats | Hardware devices, kernel subsystems, driver configurations |
| Read/Write   | Mostly read-only (except `/proc/sys/`)           | Allows modifying hardware and kernel settings              |
| Example File | `/proc/cpuinfo` (CPU details)                    | `/sys/class/net/eth0/address` (MAC address of eth0)        |
| Main Use     | Monitoring and debugging system state            | Configuring kernel and hardware parameters                 |

### Physical File Syetems

**Top-level directories**

| Directories | Description                                          |
| ----------- | ---------------------------------------------------- |
| **/etc**    | system configuration files.                          |
| **/home**   | home directory. It is the default current directory. |
| **/opt**    | optional or third-party software.                    |
| **/tmp**    | temporary space, typically cleared on reboot.        |
| **/usr**    | User related programs.                               |
| **/var**    | log files.                                           |

other directories in the Linux system:

| **Directories**      | **Description**                                  |
| -------------------- | ------------------------------------------------ |
| **/bin**             | binary or executable programs. Needed for system |
| **/usr/bin**         | Most Programs                                    |
| **/sbin  /usr/sbin** | System Config tools                              |
| **/usr/share/bin**   | Programs for other apps, like Nginx,Squic, ...   |

{% hint style="success" %}
it is good to know that locally compiled programs usually goes to `/usr/local/bin` or `/usr/local/etc`&#x20;
{% endhint %}

***

## Linux Logging Basics

Operating system logs provide a wealth of diagnostic information about your computers, and Linux is no exception. Everything from kernel events to user actions is logged by Linux, allowing you to see almost any action performed on your servers. In this guide, we’ll explain what Linux logs are, where they’re located, and how to interpret them.

### /var/log

Linux has a special directory for storing logs called `/var/log`. This directory contains logs from the OS itself, services, and various applications running on the system. Here’s what this directory looks like on a typical Ubuntu system.

```
[root@earth log]# ls -l
total 52184
drwxr-xr-x. 2 root   root       4096 Nov 17 11:15 anaconda
drwx------. 2 root   root         23 Nov  2 00:42 audit
-rw-------. 1 root   root     486287 Dec 10 08:27 boot.log
-rw-rw----. 1 root   utmp       6912 Dec  9 16:49 btmp
drwxr-x---. 2 chrony chrony        6 Nov  2 21:35 chrony
-rw-------. 1 root   root      63396 Dec 10 14:01 cron
drwxr-xr-x. 2 lp     sys          57 Nov 11 23:27 cups
-rw-r--r--. 1 root   root     602866 Dec 10 14:25 dnf.librepo.log
-rw-r--r--. 1 root   root      65431 Dec 10 14:25 dnf.log
-rw-r--r--. 1 root   root    1047061 Dec  9 16:02 dnf.log.1
-rw-r--r--. 1 root   root     974034 Nov 17 16:27 dnf.log.2
-rw-r--r--. 1 root   root     263985 Dec 10 14:25 dnf.rpm.log
-rw-r-----. 1 root   root      78855 Dec 10 08:27 firewalld
drwx--x--x. 2 root   gdm           6 Nov 12 05:09 gdm
-rw-r--r--. 1 root   root      17760 Dec 10 13:07 hawkey.log
-rw-rw-r--. 1 root   utmp     292292 Dec 10 12:18 lastlog
-rw-------. 1 root   root          0 Nov 17 11:10 maillog
-rw-------. 1 root   root   45148154 Dec 10 14:41 messages
drwx------. 2 root   root          6 May 16  2022 ppp
drwx------. 2 root   root          6 Nov 17 11:09 private
drwxr-xr-x. 2 root   root          6 Nov  6 11:53 qemu-ga
lrwxrwxrwx. 1 root   root         39 Nov 17 11:09 README -> ../../usr/share/doc/systemd/README.logs
drwx------. 3 root   root         17 Nov  2 09:37 samba
-rw-------. 1 root   root     158295 Dec 10 14:39 secure
drwx------. 2 root   root          6 May 15  2022 speech-dispatcher
-rw-------. 1 root   root          0 Nov 17 11:10 spooler
drwxr-x---. 2 sssd   sssd         26 Nov 12 04:14 sssd
-rw-------. 1 root   root          0 Nov 17 11:09 tallylog
drwxr-xr-x. 2 root   root         44 Nov  6 20:02 tuned
drwxr-xr-x. 2 root   root         68 Nov 18 13:13 vmware
-rw-r--r--. 1 root   root      57541 Nov 18 13:18 vmware-installer
-rw-r--r--. 1 root   root     172477 Nov 18 13:15 vnetlib
-rw-rw-r--. 1 root   utmp      55296 Dec 10 08:40 wtmp

```

Some of the most important Linux system logs include:

* `/var/log/syslog` and `/var/log/messages` store all global system activity data, including startup messages. Debian-based systems like Ubuntu store this in /`var/log/syslog`, while Red Hat-based systems like `RHEL` or `CentOS` use `/var/log/messages`.
* `/var/log/auth.log` and `/var/log/secure` store all security-related events such as logins, root user actions, and output from pluggable authentication modules (PAM). Ubuntu and Debian use `/var/log/auth.log`, while Red Hat and CentOS use `/var/log/secure`.
* `/var/log/kern.log` stores kernel events, errors, and warning logs, which are particularly helpful for troubleshooting custom kernels.
* `/var/log/cron` stores information about scheduled tasks (cron jobs). Use this data to verify your cron jobs are running successfully.

Some applications also write log files to this directory. For example, the Apache web server writes logs to the /var/log/apache2 directory (on Debian), while MySQL writes logs to the /var/log/mysql directory. Some applications also log via Syslog, which we’ll explain in the next section.

### syslog

Syslog is a standard for creating and transmitting logs. The word “syslog” can refer to any of the following:

1. The syslog service receives and processes syslog messages and listens for events by creating a socket located at `/dev/log`, which applications can write to. It can write messages to a local file or forward messages to a remote server. There are different syslog implementations, including rsyslogd and syslog-ng.
2. The Syslog protocol (RFC 5424) is a transport protocol specifying how to transmit logs over a network. It’s also a data format defining how messages are structured. By default, it uses port 514 for plaintext messages and port 6514 for encrypted messages.
3. A syslog message is any log formatted in the syslog message format and consists of a standardized header and message containing the log’s contents.

> Since Syslog can forward messages to remote servers, it’s often used to forward system logs to log management solutions.

**Syslog Daemons difference**

Depending on the distribution or system you use, you may get a different open source syslog daemon by default. Some prefer the standard syslogd, others go with rsyslog and others install syslog-ng by default. All of them do what you expect, and manage the syslog process and calls within the server.

| Name      | Description                                                                                                                                                                                                      | Config file                   |
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
| syslogd   | The first one, originally created in the 80's to handle the syslog protocol. It is still the default on OpenBSD.                                                                                                 | **/etc/syslog.conf**          |
| syslog-ng | Created in the late 90's as a robust replacement to to syslogd. Added support for TCP, encryption and many other features. Syslog-ng was the standard and included on Suse, Debian and Fedora for many years.    | /etc/syslog-ng/syslog-ng.conf |
| rsyslog:  | Created in 2004 as a competitor to syslog-ng, because the default syslog daemon on Ubuntu, RHEL and many other distributions. If you have a common and updated Linux distribution, you are likely using rsyslog. | **/etc/rsyslog.conf**         |

### Logging with systemd

Many Linux distributions ship with systemd—a process and service manager. Systemd implements its own logging service called journald, which can replace or complement Syslog. Journald logs in a significantly different manner than systemd, which is why it has its own section in LPIC cources.

### dmesg

**dmesg** command also called “driver message” or “display message” is used to examine the kernel ring buffer and print the message buffer of the kernel. The output of this command contains the messages produced by the device drivers.

```
[root@earth log]# dmesg
[    0.000000] Linux version 5.14.0-611.11.1.el9_7.x86_64 (mockbuild@iad1-prod-build001.bld.equ.rockylinux.org) (gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), GNU ld version 2.35.2-67.el9) #1 SMP PREEMPT_DYNAMIC Wed Dec 3 13:51:50 UTC 2025
[    0.000000] The list of certified hardware and cloud instances for Enterprise Linux 9 can be viewed at the Red Hat Ecosystem Catalog, https://catalog.redhat.com.
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt1)/vmlinuz-5.14.0-611.11.1.el9_7.x86_64 root=/dev/mapper/vg--os-lv--root ro resume=/dev/mapper/vg--os-lv--swap rd.lvm.lv=vg-os/lv-root rd.lvm.lv=vg-os/lv-swap rhgb quiet crashkernel=1G-2G:192M,2G-64G:256M,64G-:512M
[    0.000000] x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000039f98fff] usable
[    0.000000] BIOS-e820: [mem 0x0000000039f99000-0x000000003a898fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003a899000-0x00000000434aefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000434af000-0x00000000452fefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000452ff000-0x0000000045b2efff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x0000000045b2f000-0x0000000045bfefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x0000000045bff000-0x0000000045bfffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000045c00000-0x0000000049ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000004a200000-0x000000004a3fffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000004b000000-0x00000000503fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000004afbfffff] usable
```

When the computer boots up, there are lot of messages(log) generated during the system start-up. So you can read all these messages by using dmesg command. The contents of the kernel ring buffer are also stored in **'/var/log/dmesg'** fil&#x65;**.**

The **dmesg** command can be useful when the system encounters any problem during its start-up, so by reading the contents of **dmesg** command you can find out where the problem occurred(as there are many steps in the system boot-up sequence).

Syntax:

```
dmesg [options]
```

#### **Common Options for the dmesg Command**

| Option             | Description                                           |
| ------------------ | ----------------------------------------------------- |
| -C, --clear        | Clears the kernel ring buffer.                        |
| -c, --read-clear   | Prints the contents of the buffer and then clears it. |
| -D, --console-off  | Disables printing of kernel messages to the console.  |
| -E, --console-on   | Enables printing of kernel messages to the console.   |
| -F, --file \<file> | Reads kernel messages from the specified file.        |
| -h, --help         | Displays help text for dmesg and its options.         |
| -k, --kernel       | Prints only kernel messages.                          |
| -t, --notime       | Suppresses timestamps in the output.                  |
| -u, --userspace    | Prints userspace messages.                            |

That's all.

.

.

.

***

sources:

<https://www.geeksforgeeks.org/linux-unix/free-command-linux-examples/>\
<https://www.geeksforgeeks.org/linux-unix/ps-command-in-linux-with-examples/>\
<https://www.geeksforgeeks.org/linux-unix/top-command-in-linux-with-examples/>\
<https://www.geeksforgeeks.org/linux-unix/linux-directory-structure/>\
<https://opensource.com/article/19/3/virtual-filesystems-linux>\
<https://www.baeldung.com/linux/dev-directory>\
<https://www.geeksforgeeks.org/linux-unix/proc-file-system-linux/>\
<https://www.geeksforgeeks.org/linux-unix/linux-file-system/>\
[https://itprohelper.com/differences-between-proc-and-sys-in-linux/#:\~:text=Key%20Differences%20Between%20%2Fproc%20and%20%2Fsys\&text=Use%20%2Fproc%20to%20monitor%20system,hardware%20and%20configure%20kernel%20settings.](https://itprohelper.com/differences-between-proc-and-sys-in-linux/)\
[https://www.loggly.com/ultimate-guide/linux-logging-basics/#:\~:text=Some%20of%20the%20most%20important,%2Fvar%2Flog%2Fmessages%20.](https://www.loggly.com/ultimate-guide/linux-logging-basics/)\
<https://www.geeksforgeeks.org/linux-unix/dmesg-command-linux-driver-messages/>
