204.2. Adjusting Storage Device Access

204.2 Adjusting Storage Device Access

Weight: 2
Description: Candidates should be able to configure kernel options to support various drives. This objective includes software tools to view & modify hard disk settings including iSCSI devices.
Key Knowledge Areas:
    Tools and utilities to configure DMA for IDE devices including ATAPI and SATA
    Tools and utilities to configure Solid State Drives including AHCI and NVMe
    Tools and utilities to manipulate or analyse system resources (e.g. interrupts)
    Awareness of sdparm command and its uses
    Tools and utilities for iSCSI
    Awareness of SAN, including relevant protocols (AoE, FCoE)
Terms and Utilities:
    hdparm, sdparm
    nvme
    tune2fs
    fstrim
    sysctl
    /dev/hd*, /dev/sd*, /dev/nvme*
    iscsiadm, scsi_id, iscsid and iscsid.conf
    WWID, WWN, LUN numbers
In this course many standards and storage concepts are discussed. Before begging lets classified some topics and then we realize where each tools for.
Storage Types
Description
HDD
Hard Disk Drive, traditional mechanical hard disks stores data on magnetic platter.
SSD
Solid state drives (SSDs) are storage devices that contain non-volatile flash memory. they are superior to mechanical hard disk drives in terms of performance, power use, and availability.
NVMe
non-volatile memory expres (NVMe) devices are flash memory chips connected to a system via the PCI-E bus.
and from another aspect :
Storage Connectivity Protocols
Description
IDE/ATA
Popular Interface, used to connect hard disks and optical drives. speed: 133 MB/s
Serial ATA
Serial Version of IDE/ATA typically used for internal connectivity.speed up to 16 Gb/s
SCSI
Popular standard for compute-to-storage connectivity.speed up to 640 MB/s
SAS
Serial attached SCSI, is a point-to-point serial protocol that provides alternative to SCSI.speed up to 12 Gb/s
FC
Fibre Channel a widely-used-protocol for high-spedd communication to the storage device. Latest version speed up to 32 Gb/s
IP
IP is a network protocol that has been traditionally used for compute-to-compute traffic.ISCSI and FCIP protocols are common examples of using IP for compute-to-storage communications.
Now lets get familiar with some linux tools for specific Hard Drives:

hdparm

hdparm is a command line program to set and view ATA hard disk drive hardware parameters. First (in 2005) hdparm utility developed by Mrk Lord to test Linux drivers for IDE hard drives. Since then, the program has developed into a valuable tool for diagnosis and tuning of hard drives It can set parameters such as drive caches, sleep mode, power management and DMA settings.As hdparm interacts directly with hardware it can cause data lost and other file system damages.
hdparm has to be run with root privileges, otherwise it will either not be found or the requested actions will not be executed properly.
1
[email protected]:~# hdparm -h
2
3
hdparm - get/set hard disk parameters - version v9.48, by Mark Lord.
4
5
Usage: hdparm [options] [device ...]
6
7
Options:
8
-a Get/set fs readahead
9
-A Get/set the drive look-ahead flag (0/1)
10
-b Get/set bus state (0 == off, 1 == on, 2 == tristate)
11
-B Set Advanced Power Management setting (1-255)
12
-c Get/set IDE 32-bit IO setting
13
-C Check drive power mode status
14
-d Get/set using_dma flag
15
-D Enable/disable drive defect management
16
-E Set cd/dvd drive speed
17
-f Flush buffer cache for device on exit
18
-F Flush drive write cache
19
-g Display drive geometry
20
-h Display terse usage information
21
-H Read temperature from drive (Hitachi only)
22
-i Display drive identification
23
-I Detailed/current information directly from drive
24
-J Get/set Western DIgital "Idle3" timeout for a WDC "Green" drive (DANGEROUS)
25
-k Get/set keep_settings_over_reset flag (0/1)
26
-K Set drive keep_features_over_reset flag (0/1)
27
-L Set drive doorlock (0/1) (removable harddisks only)
28
-m Get/set multiple sector count
29
-M Get/set acoustic management (0-254, 128: quiet, 254: fast)
30
-n Get/set ignore-write-errors flag (0/1)
31
-N Get/set max visible number of sectors (HPA) (VERY DANGEROUS)
32
-p Set PIO mode on IDE interface chipset (0,1,2,3,4,...)
33
-P Set drive prefetch count
34
-q Change next setting quietly
35
-Q Get/set DMA queue_depth (if supported)
36
-r Get/set device readonly flag (DANGEROUS to set)
37
-R Get/set device write-read-verify flag
38
-s Set power-up in standby flag (0/1) (DANGEROUS)
39
-S Set standby (spindown) timeout
40
-t Perform device read timings
41
-T Perform cache read timings
42
-u Get/set unmaskirq flag (0/1)
43
-U Obsolete
44
-v Use defaults; same as -acdgkmur for IDE drives
45
-V Display program version and exit immediately
46
-w Perform device reset (DANGEROUS)
47
-W Get/set drive write-caching flag (0/1)
48
-x Obsolete
49
-X Set IDE xfer mode (DANGEROUS)
50
-y Put drive in standby mode
51
-Y Put drive to sleep
52
-z Re-read partition table
53
-Z Disable Seagate auto-powersaving mode
54
--dco-freeze Freeze/lock current device configuration until next power cycle
55
--dco-identify Read/dump device configuration identify data
56
--dco-restore Reset device configuration back to factory defaults
57
--dco-setmax Use DCO to set maximum addressable sectors
58
--direct Use O_DIRECT to bypass page cache for timings
59
--drq-hsm-error Crash system with a "stuck DRQ" error (VERY DANGEROUS)
60
--fallocate Create a file without writing data to disk
61
--fibmap Show device extents (and fragmentation) for a file
62
--fwdownload Download firmware file to drive (EXTREMELY DANGEROUS)
63
--fwdownload-mode3 Download firmware using min-size segments (EXTREMELY DANGEROUS)
64
--fwdownload-mode3-max Download firmware using max-size segments (EXTREMELY DANGEROUS)
65
--fwdownload-mode7 Download firmware using a single segment (EXTREMELY DANGEROUS)
66
--fwdownload-modee Download firmware using mode E (min-size segments) (EXTREMELY DANGEROUS)
67
--fwdownload-modee-max Download firmware using mode E (max-size segments) (EXTREMELY DANGEROUS)
68
--idle-immediate Idle drive immediately
69
--idle-unload Idle immediately and unload heads
70
--Istdin Read identify data from stdin as ASCII hex
71
--Istdout Write identify data to stdout as ASCII hex
72
--make-bad-sector Deliberately corrupt a sector directly on the media (VERY DANGEROUS)
73
--offset use with -t, to begin timings at given offset (in GiB) from start of drive
74
--prefer-ata12 Use 12-byte (instead of 16-byte) SAT commands when possible
75
--read-sector Read and dump (in hex) a sector directly from the media
76
--repair-sector Alias for the --write-sector option (VERY DANGEROUS)
77
--security-help Display help for ATA security commands
78
--trim-sector-ranges Tell SSD firmware to discard unneeded data sectors: lba:count ..
79
--trim-sector-ranges-stdin Same as above, but reads lba:count pairs from stdin
80
--verbose Display extra diagnostics from some commands
81
--write-sector Repair/overwrite a (possibly bad) sector directly on the media (VERY DANGEROUS)
Copied!
hdparm -I /dev/sda to get information about Hard Disk :
1
[email protected]:~# hdparm -I /dev/sda
2
3
/dev/sda:
4
5
ATA device, with non-removable media
6
Model Number: Samsung SSD 850 EVO 500GB
7
Serial Number: S2R9NX0H503451W
8
Firmware Revision: EMT02B6Q
9
Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
10
Standards:
11
Used: unknown (minor revision code 0x0039)
12
Supported: 9 8 7 6 5
13
Likely used: 9
14
Configuration:
15
Logical max current
16
cylinders 16383 16383
17
heads 16 16
18
sectors/track 63 63
19
--
20
CHS current addressable sectors: 16514064
21
LBA user addressable sectors: 268435455
22
LBA48 user addressable sectors: 976773168
23
Logical Sector size: 512 bytes
24
Physical Sector size: 512 bytes
25
Logical Sector-0 offset: 0 bytes
26
device size with M = 1024*1024: 476940 MBytes
27
device size with M = 1000*1000: 500107 MBytes (500 GB)
28
cache/buffer size = unknown
29
Form Factor: 2.5 inch
30
Nominal Media Rotation Rate: Solid State Device
31
Capabilities:
32
LBA, IORDY(can be disabled)
33
Queue depth: 32
34
Standby timer values: spec'd by Standard, no device specific minimum
35
R/W multiple sector transfer: Max = 1 Current = 1
36
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
37
Cycle time: min=120ns recommended=120ns
38
PIO: pio0 pio1 pio2 pio3 pio4
39
Cycle time: no flow control=120ns IORDY flow control=120ns
40
Commands/features:
41
Enabled Supported:
42
* SMART feature set
43
Security Mode feature set
44
* Power Management feature set
45
* Write cache
46
* Look-ahead
47
* Host Protected Area feature set
48
* WRITE_BUFFER command
49
* READ_BUFFER command
50
* NOP cmd
51
* DOWNLOAD_MICROCODE
52
SET_MAX security extension
53
* 48-bit Address feature set
54
* Device Configuration Overlay feature set
55
* Mandatory FLUSH_CACHE
56
* FLUSH_CACHE_EXT
57
* SMART error logging
58
* SMART self-test
59
* General Purpose Logging feature set
60
* WRITE_{DMA|MULTIPLE}_FUA_EXT
61
* 64-bit World wide name
62
Write-Read-Verify feature set
63
* WRITE_UNCORRECTABLE_EXT command
64
* {READ,WRITE}_DMA_EXT_GPL commands
65
* Segmented DOWNLOAD_MICROCODE
66
* Gen1 signaling speed (1.5Gb/s)
67
* Gen2 signaling speed (3.0Gb/s)
68
* Gen3 signaling speed (6.0Gb/s)
69
* Native Command Queueing (NCQ)
70
* Phy event counters
71
* READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
72
* DMA Setup Auto-Activate optimization
73
Device-initiated interface power management
74
* Asynchronous notification (eg. media change)
75
* Software settings preservation
76
* Device Sleep (DEVSLP)
77
* SMART Command Transport (SCT) feature set
78
* SCT Write Same (AC2)
79
* SCT Error Recovery Control (AC3)
80
* SCT Features Control (AC4)
81
* SCT Data Tables (AC5)
82
* reserved 69[4]
83
* DOWNLOAD MICROCODE DMA command
84
* SET MAX SETPASSWORD/UNLOCK DMA commands
85
* WRITE BUFFER DMA command
86
* READ BUFFER DMA command
87
* Data Set Management TRIM supported (limit 8 blocks)
88
Security:
89
Master password revision code = 65534
90
supported
91
not enabled
92
not locked
93
frozen
94
not expired: security count
95
supported: enhanced erase
96
2min for SECURITY ERASE UNIT. 8min for ENHANCED SECURITY ERASE UNIT.
97
Logical Unit WWN Device Identifier: 5002538d40e515dc
98
NAA : 5
99
IEEE OUI : 002538
100
Unique ID : d40e515dc
101
Device Sleep:
102
DEVSLP Exit Timeout (DETO): 50 ms (drive)
103
Minimum DEVSLP Assertion Time (MDAT): 30 ms (drive)
104
Checksum: correct
Copied!
lets check how fast is my ssd drive with -t switch:
1
[email protected]:~# hdparm -t /dev/sda
2
3
/dev/sda:
4
Timing buffered disk reads: 1528 MB in 3.00 seconds = 509.06 MB/sec
Copied!
Wow 509.06 MB/ sec.Not bad :). Some other amazing parameters and features that can be manipulated with hdparm:
hdparam command switch
Description
hdparm -B 125 /dev/sda
Set the Advanced Power Management, valus <1-255>. While 1-127 permit spin-down, 128-254 no not allow spin-down and 255 disable feature completly
hdparm -S 240 /dev/sda
Set standby time.specifies how long to wait in idle (with no disk activity) before turning off the motor to save power. 0 can disbale feature,the values from 1 to 240 specify multiples of 5 seconds and values from 241 to 251 specify multiples of 30 minutes.
hdparm -d 1 /dev/sda
set DMA (Direct Memory Access)on or off,values 0 or 1
and many many other options.

sdparm

scsi version of hdparm. sdparm manupulate scsi specific attributes of hard drive.

NVMEe

Linux has NVMe driver which is natively included in the kernel since version 3.3. NVMe devices should show up under /dev/nvme*

tune2fs

tune2fs command is used by the system administrator to change/modify tunable parameters on ext2, ext3 and ext4 type filesystems. Incomparison to hdparm it manipulates disk drive from File System layer.
1
[email protected]:~# tune2fs
2
tune2fs 1.42.13 (17-May-2015)
3
Usage: tune2fs [-c max_mounts_count] [-e errors_behavior] [-g group]
4
[-i interval[d|m|w]] [-j] [-J journal_options] [-l]
5
[-m reserved_blocks_percent] [-o [^]mount_options[,...]] [-p mmp_update_interval]
6
[-r reserved_blocks_count] [-u user] [-C mount_count] [-L volume_label]
7
[-M last_mounted_dir] [-O [^]feature[,...]]
8
[-Q quota_options]
9
[-E extended-option[,...]] [-T last_check_time] [-U UUID]
10
[ -I new_inode_size ] device
Copied!
We have already seen tune2fs -l /dev/sda command and we have used -i and -c flags to change fsck check interval. Lets do something more interesting. By default every File System in Linux has some space reserved for root, so no regular user can fill file system up to 100%. As a standard, each File System reserve 5% of total space.Lets change it:
1
[email protected]:~# tune2fs -l /dev/sda8 | grep -i reserved
2
Reserved block count: 610355
3
Reserved GDT blocks: 1021
4
Reserved blocks uid: 0 (user root)
5
Reserved blocks gid: 0 (group root)
6
7
[email protected]:~# tune2fs -m 6 /dev/sda8
8
tune2fs 1.42.13 (17-May-2015)
9
Setting reserved blocks percentage to 6% (732426 blocks)
10
11
[email protected]:~# tune2fs -l /dev/sda8 | grep -i reserved
12
Reserved block count: 732426
13
Reserved GDT blocks: 1021
14
Reserved blocks uid: 0 (user root)
15
Reserved blocks gid: 0 (group root)
Copied!
When we have very huge partition, ext file system sparse lots of super blocks bakcups in different places of hard disk which consume noticeable space, we can Limit the number of backup superblocks to save space on large filesystems using tune2fs -s 0 /dev/sda10 command.
note that after any manipulation fsck should be run in order to changes take effect.

sysctl

We have previously seen sysctl as a tool to view and change kernel parameters at the run time. sysctl deals with /proc directory .
1
2
3
Usage:
4
sysctl [options] [variable[=value] ...]
5
6
Options:
7
-a, --all display all variables
8
-A alias of -a
9
-X alias of -a
10
--deprecated include deprecated parameters to listing
11
-b, --binary print value without new line
12
-e, --ignore ignore unknown variables errors
13
-N, --names print variable names without values
14
-n, --values print only values of a variables
15
-p, --load[=<file>] read values from file
16
-f alias of -p
17
--system read values from all system directories
18
-r, --pattern <expression>
19
select setting that match expression
20
-q, --quiet do not echo variable set
21
-w, --write enable writing a value to variable
22
-o does nothing
23
-x does nothing
24
-d alias of -h
25
26
-h, --help display this help and exit
27
-V, --version output version information and exit
28
29
For more details see sysctl(8).
Copied!
We used sysctl -a to list all parameters ,and as an example we know sysctl -w net.ipv4.ipforward=1 is the same as echo "1" > /proc/sys/net/ipv4/ip_forward. Then for marking changes persistent we learned that sysctl has a configuration file which is /etc/sysctl.conf and kernel parameters are defined there.

What is LUN?

a logical unit number, or LUN, is a number used to identify a logical unit, which is a device addressed by the SCSI protocol or Storage Area Network protocols which encapsulate SCSI, such as Fibre Channel or iSCSI.

iscsi "EYE-skuzzy"

Befor talking about iscsi, lets talk about scsi to understand it better, Whats scsi? SCSI "skuzzy" ( Small Computer System Interface), is a set of American National Standards Institute (ANSI) standard electronic interfaces that allow personal computers (PCs) to communicate with peripheral hardware such as disk drives, tape drives, CD-ROM drives, printers and scanners faster and more flexibly than previous parallel data transfer interfaces . But what if we could do it remotely?
iSCSI (Internet Small Computer System Interface), works on top of the TCP (Transport Control Protocol) and allows the SCSI command to be sent end-to-end over LANs and WANs or the Internet.
How dose it work? iSCSI works by transporting block-level data between an iSCSI initiator on a computer(as client) and an iSCSI target on a storage device(as server). The iSCSI protocol encapsulates SCSI commands and assembles the data in packets for the TCP/IP layer. Packets are sent over the network using a point-to-point connection. When packets arrived, the iSCSI protocol disassembles the packets, take out SCSI commands so the operating system will see the storage as a local SCSI device that can be formatted as usual.

iqn,EUI

Each iscsi target or iscsi initirator is called iscsi node.All iSCSI nodes are identified by an iSCSI name. An iSCSI name is not IP address or DNS name of that host. Names enable iSCSI storage resources to be managed regardless of other kind of addresses, Also it is used in authentication of targets to initiators and initiators to targets.
iSCSI addresses can be one of two types:
    1.
    iSCSI Qualified Name (iQN)
    2.
    IEEE Naming convention (EUI)
iQN format ‐ iqn.yyyy‐mm.com.xyz.aabbccddeeffgghh :
    iqn ‐ Naming convention identifier
    yyyy‐nn ‐ Point in time when the .com domain was registered
    com.xyz ‐ Domain of the node backwards
    aabbccddeeffgghh ‐ Device identifier (can be a WWN, the system name, or any other vendorimplemented
    standard)
EUI format ‐ eui.64‐bit WWN:
    eui ‐ Naming prefix
    64‐bit WWN ‐ FC WWN of the host
Okey enough introduction lets start.To keep it simple we use CentOS 7 server (target) with additional 10 gig Disk Drive and then get Our Ubuntu client (initiator) connected to it. First We need to add EPEL repository:
1
###Server Side , CentOS with 10 gig Disk, iscsitarget
2
[[email protected] ~]# yum install epel-release
Copied!
and now lets install iscsi target:
1
[[email protected] ~]# yum search scsi target
2
Loaded plugins: fastestmirror, langpacks
3
Loading mirror speeds from cached hostfile
4
* base: mirror.us.leaseweb.net
5
* epel: mirrors.lug.mtu.edu
6
* extras: mirror.us.leaseweb.net
7
* updates: mirror.us.leaseweb.net
8
============================== N/S matched: scsi, target ==============================
9
netbsd-iscsi.x86_64 : User-space implementation of iSCSI target from NetBSD project
10
scsi-target-utils.x86_64 : The SCSI target daemon and utility programs
11
scsi-target-utils-gluster.x86_64 : Support for the Gluster backstore to
12
: scsi-target-utils
13
python-rtslib.noarch : API for Linux kernel LIO SCSI target
14
15
Full name and summary matches only, use "search all" for everything.
16
17
[[email protected] ~]# yum install -y scsi-target-utils
18
Loaded plugins: fastestmirror, langpacks
19
Loading mirror speeds from cached hostfile
20
* base: mirror.steadfast.net
21
* epel: fedora-epel.mirror.iweb.com
22
* extras: mirror.steadfast.net
23
* updates: mirror.steadfast.net
24
Resolving Dependencies
25
--> Running transaction check
26
---> Package scsi-target-utils.x86_64 0:1.0.55-4.el7 will be installed
27
--> Processing Dependency: perl(Config::General) for package: scsi-target-utils-1.0.55-4.el7.x86_64
28
--> Processing Dependency: sg3_utils for package: scsi-target-utils-1.0.55-4.el7.x86_64
29
--> Running transaction check
30
---> Package perl-Config-General.noarch 0:2.61-1.el7 will be installed
31
---> Package sg3_utils.x86_64 0:1.37-12.el7 will be installed
32
--> Finished Dependency Resolution
33
34
Dependencies Resolved
35
36
=======================================================================================
37
Package Arch Version Repository Size
38
=======================================================================================
39
Installing:
40
scsi-target-utils x86_64 1.0.55-4.el7 epel 209 k
41
Installing for dependencies:
42
perl-Config-General noarch 2.61-1.el7 epel 75 k
43
sg3_utils x86_64 1.37-12.el7 base 644 k
44
45
Transaction Summary
46
=======================================================================================
47
Install 1 Package (+2 Dependent packages)
48
49
Total download size: 927 k
50
Installed size: 2.3 M
51
Downloading packages:
52
(1/3): perl-Config-General-2.61-1.el7.noarch.rpm | 75 kB 00:00:03
53
(2/3): scsi-target-utils-1.0.55-4.el7.x86_64.rpm | 209 kB 00:00:05
54
(3/3): sg3_utils-1.37-12.el7.x86_64.rpm | 644 kB 00:00:06
55
---------------------------------------------------------------------------------------
56
Total 131 kB/s | 927 kB 00:07
57
Running transaction check
58
Running transaction test
59
Transaction test succeeded
60
Running transaction
61
Installing : perl-Config-General-2.61-1.el7.noarch 1/3
62
Installing : sg3_utils-1.37-12.el7.x86_64 2/3
63
Installing : scsi-target-utils-1.0.55-4.el7.x86_64 3/3
64
Verifying : sg3_utils-1.37-12.el7.x86_64 1/3
65
Verifying : perl-Config-General-2.61-1.el7.noarch 2/3
66
Verifying : scsi-target-utils-1.0.55-4.el7.x86_64 3/3
67
68
Installed:
69
scsi-target-utils.x86_64 0:1.0.55-4.el7
70
71
Dependency Installed:
72
perl-Config-General.noarch 0:2.61-1.el7 sg3_utils.x86_64 0:1.37-12.el7
73
74
Complete!
75
76
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
77
sdb 8:16 0 10G 0 disk
78
sr0 11:0 1 1024M 0 rom
79
sda 8:0 0 50G 0 disk
80
├─sda2 8:2 0 49G 0 part
81
│ ├─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
82
│ └─centos-root 253:0 0 45.1G 0 lvm /
83
└─sda1 8:1 0 1G 0 part /boot
Copied!
Now lets configure target vi /etc/tgt/target.conf:
1
# This is a sample config file for tgt-admin.
2
#
3
# The "#" symbol disables the processing of a line.
4
5
# Set the driver. If not specified, defaults to "iscsi".
6
default-driver iscsi
7
8
# Set iSNS parameters, if needed
9
#iSNSServerIP 192.168.111.222
10
#iSNSServerPort 3205
11
#iSNSAccessControl On
12
#iSNS On
13
14
# Continue if tgtadm exits with non-zero code (equivalent of
15
# --ignore-errors command line option)
16
#ignore-errors yes
17
###############################
18
#Our added configuration:
19
<target 192.168.10.134:target00>
20
backing-store /dev/sdb
21
initiator-address 192.168.10.151
22
incominguser myuser mypassword
23
</target>
Copied!
lets restrat service:
1
[[email protected] ~]# systemctl restrat tgtd.service
2
Unknown operation 'restrat'.
3
[[email protected] ~]# systemctl restart tgtd.service
4
[[email protected] ~]# systemctl status tgtd.service
5
● tgtd.service - tgtd iSCSI target daemon
6
Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)
7
Active: active (running) since Tue 2018-01-09 03:56:23 EST; 7s ago
8
Process: 5601 ExecStop=/usr/sbin/tgtadm --op delete --mode system (code=exited, status=0/SUCCESS)
9
Process: 5592 ExecStop=/usr/sbin/tgt-admin --update ALL -c /dev/null (code=exited, status=0/SUCCESS)
10
Process: 5589 ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)
11
Process: 5644 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)
12
Process: 5614 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)
13
Process: 5612 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)
14
Process: 5610 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)
15
Main PID: 5609 (tgtd)
16
CGroup: /system.slice/tgtd.service
17
└─5609 /usr/sbin/tgtd -f
18
19
Jan 09 03:56:17 server1 systemd[1]: Starting tgtd iSCSI target daemon...
20
Jan 09 03:56:17 server1 tgtd[5609]: tgtd: iser_ib_init(3436) Failed to initialize...es?
21
Jan 09 03:56:17 server1 tgtd[5609]: tgtd: work_timer_start(146) use timer_fd base...ler
22
Jan 09 03:56:17 server1 tgtd[5609]: tgtd: bs_init_signalfd(267) could not open ba...ore
23
Jan 09 03:56:17 server1 tgtd[5609]: tgtd: bs_init(386) use signalfd notification
24
Jan 09 03:56:23 server1 tgtd[5609]: tgtd: device_mgmt(246) sz:14 params:path=/dev/sdb
25
Jan 09 03:56:23 server1 tgtd[5609]: tgtd: bs_thread_open(408) 16
26
Jan 09 03:56:23 server1 systemd[1]: Started tgtd iSCSI target daemon.
27
Hint: Some lines were ellipsized, use -l to show in full.
Copied!
use tgtadmin to scan what is result:
1
[[email protected] ~]# tgtadm --mode target --op show
2
Target 1: 192.168.10.134:target00
3
System information:
4
Driver: iscsi
5
State: ready
6
I_T nexus information:
7
LUN information:
8
LUN: 0
9
Type: controller
10
SCSI ID: IET 00010000
11
SCSI SN: beaf10
12
Size: 0 MB, Block size: 1
13
Online: Yes
14
Removable media: No
15
Prevent removal: No
16
Readonly: No
17
SWP: No
18
Thin-provisioning: No
19
Backing store type: null
20
Backing store path: None
21
Backing store flags:
22
LUN: 1
23
Type: disk
24
SCSI ID: IET 00010001
25
SCSI SN: beaf11
26
Size: 10737 MB, Block size: 512
27
Online: Yes
28
Removable media: No
29
Prevent removal: No
30
Readonly: No
31
SWP: No
32
Thin-provisioning: No
33
Backing store type: rdwr
34
Backing store path: /dev/sdb
35
Backing store flags:
36
Account information:
37
myuser
38
ACL information:
39
192.168.10.151
Copied!
Do not forget to sysemctl enable tgt.service to keep iscsi target service Active even after reboot. Before testing we have to make tcp port 3260 open, CentOS 7 use firewalld on top of iptables so we use firewall-cmd to manipulate ip table:
1
[[email protected] ~]# firewall-cmd --add-port=3260/tcp --zone=public --permanent
2
success
3
[[email protected] ~]# firewall-cmd --reload
4
success
5
[[email protected] ~]# firewall-cmd --zone=public --list-all
6
public
7
target: default
8
icmp-block-inversion: no
9
interfaces:
10
sources:
11
services: ssh dhcpv6-client
12
ports: 3260/tcp
13
protocols:
14
masquerade: no
15
forward-ports:
16
source-ports:
17
icmp-blocks:
18
rich rules:
Copied!
okey its time to configure our client as iscsi initiator , we use ubuntu 16 here:
1
###Client side, ubuntu 16.04, initiator
2
[email protected]:~# apt install open-iscsi
Copied!
lets configure iscsi initiator vi /etc/iscsi/iscsid.conf :
1
#
2
# Open-iSCSI default configuration.
3
# Could be located at /etc/iscsi/iscsid.conf or ~/.iscsid.conf
4
#
5
# Note: To set any of these values for a specific node/session run
6
# the iscsiadm --mode node --op command for the value. See the README
7
# and man page for iscsiadm for details on the --op command.
8
#
9
10
######################
11
# iscsid daemon config
12
######################
13
# If you want iscsid to start the first time a iscsi tool
14
# needs to access it, instead of starting it when the init
15
# scripts run, set the iscsid startup command here. This
16
# should normally only need to be done by distro package
17
# maintainers.
18
#
19
# Default for Fedora and RHEL. (uncomment to activate).
20
# iscsid.startup = /etc/rc.d/init.d/iscsid force-start
21
#
22
# Default for upstream open-iscsi scripts (uncomment to activate).
23
iscsid.startup = /usr/sbin/iscsid
24
25
26
#############################
27
# NIC/HBA and driver settings
28
#############################
29
# open-iscsi can create a session and bind it to a NIC/HBA.
30
# To set this up see the example iface config file.
31
32
#*****************
33
# Startup settings
34
#*****************
35
36
# To request that the iscsi initd scripts startup a session set to "automatic".
37
# node.startup = automatic
38
#
39
# To manually startup the session set to "manual". The default is manual.
40
node.startup = manual
41
42
# For "automatic" startup nodes, setting this to "Yes" will try logins on each
43
# available iface until one succeeds, and then stop. The default "No" will try
44
# logins on all availble ifaces simultaneously.
45
node.leading_login = No
46
47
# *************
48
# CHAP Settings
49
# *************
50
51
# To enable CHAP authentication set node.session.auth.authmethod
52
# to CHAP. The default is None.
53
#node.session.auth.authmethod = CHAP
54
55
# To set a CHAP username and password for initiator
56
# authentication by the target(s), uncomment the following lines:
57
#node.session.auth.username = username
58
#node.session.auth.password = password
59
60
# To set a CHAP username and password for target(s)
61
# authentication by the initiator, uncomment the following lines:
62
#node.session.auth.username_in = username_in
63
#node.session.auth.password_in = password_in
64
65
# To enable CHAP authentication for a discovery session to the target
66
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
67
#discovery.sendtargets.auth.authmethod = CHAP
68
69
# To set a discovery session CHAP username and password for the initiator
70
# authentication by the target(s), uncomment the following lines:
71
#discovery.sendtargets.auth.username = username
72
#discovery.sendtargets.auth.password = password
73
74
# To set a discovery session CHAP username and password for target(s)
75
# authentication by the initiator, uncomment the following lines:
76
#discovery.sendtargets.auth.username_in = username_in
77
#discovery.sendtargets.auth.password_in = password_in
78
79
# ********
80
# Timeouts
81
# ********
82
#
83
# See the iSCSI REAME's Advanced Configuration section for tips
84
# on setting timeouts when using multipath or doing root over iSCSI.
85
#
86
# To specify the length of time to wait for session re-establishment
87
# before failing SCSI commands back to the application when running
88
# the Linux SCSI Layer error handler, edit the line.
89
# The value is in seconds and the default is 120 seconds.
90
# Special values:
91
# - If the value is 0, IO will be failed immediately.
92
# - If the value is less than 0, IO will remain queued until the session
93
# is logged back in, or until the user runs the logout command.
94
node.session.timeo.replacement_timeout = 120
95
96
# To specify the time to wait for login to complete, edit the line.
97
# The value is in seconds and the default is 15 seconds.
98
node.conn[0].timeo.login_timeout = 15
99
100
# To specify the time to wait for logout to complete, edit the line.
101
# The value is in seconds and the default is 15 seconds.
102
node.conn[0].timeo.logout_timeout = 15
103
104
# Time interval to wait for on connection before sending a ping.
105
node.conn[0].timeo.noop_out_interval = 5
106
107
# To specify the time to wait for a Nop-out response before failing
108
# the connection, edit this line. Failing the connection will
109
# cause IO to be failed back to the SCSI layer. If using dm-multipath
110
# this will cause the IO to be failed to the multipath layer.
111
node.conn[0].timeo.noop_out_timeout = 5
112
113
# To specify the time to wait for abort response before
114
# failing the operation and trying a logical unit reset edit the line.
115
# The value is in seconds and the default is 15 seconds.
116
node.session.err_timeo.abort_timeout = 15
117
118
# To specify the time to wait for a logical unit response
119
# before failing the operation and trying session re-establishment
120
# edit the line.
121
# The value is in seconds and the default is 30 seconds.
122
node.session.err_timeo.lu_reset_timeout = 30
123
124
# To specify the time to wait for a target response
125
# before failing the operation and trying session re-establishment
126
# edit the line.
127
# The value is in seconds and the default is 30 seconds.
128
node.session.err_timeo.tgt_reset_timeout = 30
129
130
131
#******
132
# Retry
133
#******
134
135
# To specify the number of times iscsid should retry a login
136
# if the login attempt fails due to the node.conn[0].timeo.login_timeout
137
# expiring modify the following line. Note that if the login fails
138
# quickly (before node.conn[0].timeo.login_timeout fires) because the network
139
# layer or the target returns an error, iscsid may retry the login more than
140
# node.session.initial_login_retry_max times.
141
#
142
# This retry count along with node.conn[0].timeo.login_timeout
143
# determines the maximum amount of time iscsid will try to
144
# establish the initial login. node.session.initial_login_retry_max is
145
# multiplied by the node.conn[0].timeo.login_timeout to determine the
146
# maximum amount.
147
#
148
# The default node.session.initial_login_retry_max is 8 and
149
# node.conn[0].timeo.login_timeout is 15 so we have:
150
#
151
# node.conn[0].timeo.login_timeout * node.session.initial_login_retry_max =
152
# 120 seconds
153
#
154
# Valid values are any integer value. This only
155
# affects the initial login. Setting it to a high value can slow
156
# down the iscsi service startup. Setting it to a low value can
157
# cause a session to not get logged into, if there are distuptions
158
# during startup or if the network is not ready at that time.
159
node.session.initial_login_retry_max = 8
160
161
################################
162
# session and device queue depth
163
################################
164
165
# To control how many commands the session will queue set
166
# node.session.cmds_max to an integer between 2 and 2048 that is also
167
# a power of 2. The default is 128.
168
node.session.cmds_max = 128
169
170
# To control the device's queue depth set node.session.queue_depth
171
# to a value between 1 and 1024. The default is 32.
172
node.session.queue_depth = 32
173
174
##################################
175
# MISC SYSTEM PERFORMANCE SETTINGS
176
##################################
177
178
# For software iscsi (iscsi_tcp) and iser (ib_iser) each session
179
# has a thread used to transmit or queue data to the hardware. For
180
# cxgb3i you will get a thread per host.
181
#
182
# Setting the thread's priority to a lower value can lead to higher throughput
183
# and lower latencies. The lowest value is -20. Setting the priority to
184
# a higher value, can lead to reduced IO performance, but if you are seeing
185
# the iscsi or scsi threads dominate the use of the CPU then you may want
186
# to set this value higher.
187
#
188
# Note: For cxgb3i you must set all sessions to the same value, or the
189
# behavior is not defined.
190
#
191
# The default value is -20. The setting must be between -20 and 20.
192
node.session.xmit_thread_priority = -20
193
194
195
#***************
196
# iSCSI settings
197
#***************
198
199
# To enable R2T flow control (i.e., the initiator must wait for an R2T
200
# command before sending any data), uncomment the following line:
201
#
202
#node.session.iscsi.InitialR2T = Yes
203
#
204
# To disable R2T flow control (i.e., the initiator has an implied
205
# initial R2T of "FirstBurstLength" at offset 0), uncomment the following line:
206
#
207
# The defaults is No.
208
node.session.iscsi.InitialR2T = No
209
210
#
211
# To disable immediate data (i.e., the initiator does not send
212
# unsolicited data with the iSCSI command PDU), uncomment the following line:
213
#
214
#node.session.iscsi.ImmediateData = No
215
#
216
# To enable immediate data (i.e., the initiator sends unsolicited data
217
# with the iSCSI command packet), uncomment the following line:
218
#
219
# The default is Yes
220
node.session.iscsi.ImmediateData = Yes
221
222
# To specify the maximum number of unsolicited data bytes the initiator
223
# can send in an iSCSI PDU to a target, edit the following line.
224
#
225
# The value is the number of bytes in the range of 512 to (2^24-1) and
226
# the default is 262144
227
node.session.iscsi.FirstBurstLength = 262144
228
229
# To specify the maximum SCSI payload that the initiator will negotiate
230
# with the target for, edit the following line.
231
#
232
# The value is the number of bytes in the range of 512 to (2^24-1) and
233
# the defauls it 16776192
234
node.session.iscsi.MaxBurstLength = 16776192
235
236
# To specify the maximum number of data bytes the initiator can receive
237
# in an iSCSI PDU from a target, edit the following line.
238
#
239
# The value is the number of bytes in the range of 512 to (2^24-1) and
240
# the default is 262144
241
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
242
243
# To specify the maximum number of data bytes the initiator will send
244
# in an iSCSI PDU to the target, edit the following line.
245
#
246
# The value is the number of bytes in the range of 512 to (2^24-1).
247
# Zero is a special case. If set to zero, the initiator will use
248
# the target's MaxRecvDataSegmentLength for the MaxXmitDataSegmentLength.
249
# The default is 0.
250
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
251
252
# To specify the maximum number of data bytes the initiator can receive
253
# in an iSCSI PDU from a target during a discovery session, edit the
254
# following line.
255
#
256
# The value is the number of bytes in the range of 512 to (2^24-1) and
257
# the default is 32768
258
#
259
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
260
261
# To allow the targets to control the setting of the digest checking,
262
# with the initiator requesting a preference of enabling the checking, uncomment# one or both of the following lines:
263
#node.conn[0].iscsi.HeaderDigest = CRC32C,None
264
#node.conn[0].iscsi.DataDigest = CRC32C,None
265
#
266
# To allow the targets to control the setting of the digest checking,
267
# with the initiator requesting a preference of disabling the checking,
268
# uncomment one or both of the following lines:
269
#node.conn[0].iscsi.HeaderDigest = None,CRC32C
270
#node.conn[0].iscsi.DataDigest = None,CRC32C
271
#
272
# To enable CRC32C digest checking for the header and/or data part of
273
# iSCSI PDUs, uncomment one or both of the following lines:
274
#node.conn[0].iscsi.HeaderDigest = CRC32C
275
#node.conn[0].iscsi.DataDigest = CRC32C
276
#
277
# To disable digest checking for the header and/or data part of
278
# iSCSI PDUs, uncomment one or both of the following lines:
279
#node.conn[0].iscsi.HeaderDigest = None
280
#node.conn[0].iscsi.DataDigest = None
281
#
282
# The default is to never use DataDigests or HeaderDigests.
283
#
284
285
# For multipath configurations, you may want more than one session to be
286
# created on each iface record. If node.session.nr_sessions is greater
287
# than 1, performing a 'login' for that node will ensure that the
288
# appropriate number of sessions is created.
289
node.session.nr_sessions = 1
290
291
#************
292
# Workarounds
293
#************
294
295
# Some targets like IET prefer after an initiator has sent a task
296
# management function like an ABORT TASK or LOGICAL UNIT RESET, that
297
# it does not respond to PDUs like R2Ts. To enable this behavior uncomment
298
# the following line (The default behavior is Yes):
299
node.session.iscsi.FastAbort = Yes
300
301
# Some targets like Equalogic prefer that after an initiator has sent
302
# a task management function like an ABORT TASK or LOGICAL UNIT RESET, that
303
# it continue to respond to R2Ts. To enable this uncomment this line
304
# node.session.iscsi.FastAbort = No
305
######################
306
### Copy and uncoment and modify:
307
308
309
node.session.auth.username = myuser
310
node.session.auth.password = mypassword
Copied!
and lets start the service:
1
[email protected]:~# systemctl start iscsid.service
Copied!
and check with iscsiadm tool:
1
[email protected]:~# iscsiadm --mode discovery -t sendtargets --portal 192.168.10.134
2
192.168.10.134:3260,1 192.168.10.134:target00
Copied!
see current hard disk of our client and lets add new hard disk using iscsiadm:
1
[email protected]:~# ls /dev/sd*
2
/dev/sda /dev/sda1 /dev/sda2 /dev/sda5
3
4
[email protected]:~# iscsiadm --mode node --targetname 192.168.10.134:target00 --portal 192.168.10.134 --login
5
Logging in to [iface: default, target: 192.168.10.134:target00, portal: 192.168.10.134,3260] (multiple)
6
Login to [iface: default, target: 192.168.10.134:target00, portal: 192.168.10.134,3260] successful.
7
8
[email protected]:~# ls /dev/sd*
9
/dev/sda /dev/sda1 /dev/sda2 /dev/sda5 /dev/sdb
Copied!
okey /dev/sdb is added, if you have noticed. to confirm:
1
[email protected]:~# iscsiadm -m session -o show
2
tcp: [1] 192.168.10.134:3260,1 192.168.10.134:target00 (non-flash)
3
[email protected]:~# cat /proc/partitions
4
major minor #blocks name
5
8 0 52428800 sda
6
8 1 51380224 sda1
7
8 2 1 sda2
8
8 5 1045504 sda5
9
11 0 1048575 sr0
10
8 32 10485760 sdb
Copied!
and if we reboot the system BoOoMM! every thing is gone! to make it persistent change node.startup = manual to automatic inside /etc/iscsi/iscsi.conf file. and copy iscsiadm script to /etc/rc.local and enable it if required:
1
[email protected]:~# echo "iscsiadm --mode node --targetname 192.168.10.134:target00 --portal 192.168.10.134 --login" >> /etc/rc.local
2
[email protected]:~# systemctl enable rc-local.service
3
[email protected]:~# chmod 750 /etc/rc.local
Copied!
and we are done. Now we can format mount the partition or we can add it to fstab using _netdev option.
Last modified 2yr ago