206.2. Backup operations

206.2 Backup operations

Weight: 3
Description: Candidates should be able to use system tools to back up important system data.
Key Knowledge Areas:
    Knowledge about directories that have to be include in backups
    Awareness of network backup solutions such as Amanda, Bacula, Bareos and BackupPC
    Knowledge of the benefits and drawbacks of tapes, CDR, disk or other backup media
    Perform partial and manual backups.
    Verify the integrity of backup files.
    Partially or fully restore backups.
Terms and Utilities:
    /bin/sh
    dd
    tar
    /dev/st* and /dev/nst*
    mt
    rsync

Why do we need to Backup?

We have already talk about RAID and LVM, although making RAID group or creating LVM, make sort of reliability and safetiness but they are not consider as backup solutions. We might lose data and experience failure because of different reasons:
    electricity goes off
    Hardware failure (mobo, cpu, ram, hard disks, ...)
    human kind miss configuration
    ...
To tell the truth the last reason is the most common and the most dangerous one. To avoid them we need to backup, we have to backup, we are forced to backup.

What to backup?

Not all directories and file are required to be backed up especially when there backup space issues. In linux File System Hierarchy Standard (FHS) there are directories and files with different priority for backing up:
Directory
Priority
Description
/etc/
high
system wide configuration files required for all programs
/home/
high
Home directories for all users to store their personal files
/usr/local/
high
contains users programs that you install from source
/var/lib/
medium
contains lots of data, getting full backup leave us on the safe side.
/var/mail/
medium
local emails
/var/www/
medium
deafult web root
/var/spool/
medium
printer queues but might used by some applications
/var/log/
low
system log files
/opt/
medium
Contains add-on applications from individual vendors
/usr/
low
Contains binaries, libraries, documentation, and source-code for second level program

How to backup?

In any platform there are always some native tools and third party programs for backing up:
Package
Licence
Language
Graphical user (GUI) interface
Command Line(CMD)interface
BSD
C , Perl
No (except with Amanda Enterprise)
Yes
GPLv2.0
Perl
Yes
Yes
AGPLv3.0
C , C++
Yes
Yes
All three Packages has Linux, windows, MacOS versions. Now lets spend time on some traditional native tools for backing up.

tape

Using tapes for backing up is some how out modded but tapes are still used because they are cheap and huge but they are very slow. If we have had chance to administrate a system with tape device plugged, we would see these directories.
    /dev/st0
    /dev/nst0
The /dev/nst0 device is a non rewinding tape device, where as the /dev/st0 device is a rewinding tape device. The device you choose to use depends on your goal. Both devices are for the same piece of hardware but they behave differently. we can rewind /dev/st0 by using software but we can not use software for /dev/nst0 so we have to rewind it physically.

Understanding tape file marks and block size

Each tape device can store multiple tape backup files. Tape backup files are created using cpio, tar, dd, and so on. However, tape device can be opened, written data to, and closed by the various program. We can store several backups (tapes) on physical tape. Between each tape file is a “tape file mark”. This is used to indicate where one tape file ends and another begins on physical tape. You need to use mt command to positions the tape (winds forward and rewinds and marks).

mt command

mt command is used to control operations of the tape drive, such as finding status or seeking through files on a tape or writing tape control marks to the tape.
some mt command examples
Description
mt -f /dev/st0 rewind
rewind tape drive
mt -f /dev/st0 status
Display status information about tape unit
mt -f /dev/st0 erase
erase the tape
mt -f /dev/st0 eject
eject tape drive
mt -f /dev/st0 eof
Writes n EOF marks in the current position of tape
Here is the list of tape position commands:
1
fsf Forward space count files. The tape is positioned on the first block of the next file.
2
3
fsfm Forward space count files. The tape is positioned on the last block of the previous file.
4
5
bsf Backward space count files. The tape is positioned on the last block of the previous file.
6
7
bsfm Backward space count files. The tape is positioned on the first block of the next file.
8
9
asf The tape is positioned at the beginning of the count file.
10
Positioning is done by first rewinding the tape and then spacing forward over count filemarks.
11
12
fsr Forward space count records.
13
14
bsr Backward space count records.
15
16
fss (SCSI tapes) Forward space count setmarks.
17
18
bss (SCSI tapes) Backward space count setmarks.
Copied!
and many many other options.

How is data stored on a tape drive ?

All data is stored subsequently in sequential tape archive format using tar. The first tape archive will start on the physical beginning of the tape (tar #0). The next will be tar #1 and so on.

tar

We have got to use tar (tape archive(r)) to create tar files, but infact its designed to archive files on tape device.
1
[email protected]:~# tree mydirectory/
2
mydirectory/
3
├── dir1
4
│ └── file1.txt
5
├── dir2
6
│ └── file2.txt
7
├── dir3
8
│ └── file3.txt
9
└── myfile
10
11
3 directories, 4 files
12
[email protected]:~# tar -cvf backup-mydirectory.tar mydirectory/
13
mydirectory/
14
mydirectory/dir3/
15
mydirectory/dir3/file3.txt
16
mydirectory/myfile
17
mydirectory/dir1/
18
mydirectory/dir1/file1.txt
19
mydirectory/dir2/
20
mydirectory/dir2/file2.txt
21
23
backup-mydirectory.tar mydirectory pooler-cpuminer-2.5.0.tar.gz
24
cpuminer mynfs
Copied!
Opps, delete some file inorder to restore it from our backup:
1
[email protected]:~# rm -rf mydirectory/myfile , mydirectory/dir3
2
[email protected]:~# tree mydirectory/
3
mydirectory/
4
├── dir1
5
│ └── file1.txt
6
└── dir2
7
└── file2.txt
8
9
2 directories, 2 files
11
backup-mydirectory.tar mydirectory pooler-cpuminer-2.5.0.tar.gz
12
cpuminer mynfs
Copied!
List the files inside tar file with -tvf switches:
1
[email protected]:~# tar -tvf backup-mydirectory.tar
2
drwxr-xr-x root/root 0 2018-02-05 03:54 mydirectory/
3
drwxr-xr-x root/root 0 2018-02-05 03:54 mydirectory/dir3/
4
-rw-r--r-- root/root 0 2018-02-05 03:54 mydirectory/dir3/file3.txt
5
-rw-r--r-- root/root 0 2018-02-05 03:53 mydirectory/myfile
6
drwxr-xr-x root/root 0 2018-02-05 03:54 mydirectory/dir1/
7
-rw-r--r-- root/root 0 2018-02-05 03:54 mydirectory/dir1/file1.txt
8
drwxr-xr-x root/root 0 2018-02-05 03:54 mydirectory/dir2/
9
-rw-r--r-- root/root 0 2018-02-05 03:54 mydirectory/dir2/file2.txt
Copied!
We need to restore myfile and dir3 from our backup:
1
[email protected]:~# tar -xvf backup-mydirectory.tar mydirectory/myfile
2
mydirectory/myfile
3
[email protected]:~# tree mydirectory/
4
mydirectory/
5
├── dir1
6
│ └── file1.txt
7
├── dir2
8
│ └── file2.txt
9
└── myfile
10
11
2 directories, 3 files
12
[email protected]:~# tar -xvf backup-mydirectory.tar mydirectory/dir3/file3.txt
13
mydirectory/dir3/file3.txt
14
[email protected]:~# tree mydirectory/
15
mydirectory/
16
├── dir1
17
│ └── file1.txt
18
├── dir2
19
│ └── file2.txt
20
├── dir3
21
│ └── file3.txt
22
└── myfile
23
24
3 directories, 4 files
Copied!
Lets combine usefull tar switches as a quick review:
tar command
Description
tar -cvf mybackup.tar myfiles/
Create tar backup file
tar -cvzf mybackup.tar.gz myfiles/
Create tar.gzip backup file
tar -cvjf mybackup.tar.bz2 myfiles/
Create tar.bzip2 backup file
tar -xvf mybackup.tar
Uncompress tar or gzip or bzip2 files
tar -tvf mybackup.tar
List content of tar or gzip or bzip2 files
tar -xvf mybackup.tar myfile
Extract a single file from tar, tar.gz and tar.bz2 file
tar -xvf mybackup.tar "file1.txt" "file2.txt"
Untar Multiple files from tar, tar.gz and tar.bz2 file
tar -xvf mybackup.tar --wildcards '*.conf'
Extract group of files using wildcard
tar -rvf mybackup.tar xyz.txt
Add files or directories to a tar, tar.gz and tar.bz2 file
tar -xvfW mybackup.tar
Verify tar, tar.gz and tar.bz2 Archive File
tar -czf mybackup.tar
Check the Size of the tar, tar.gz and tar.bz2 Archive File

rsync

Rsync (Remote Sync) is a most commonly used command for copying and synchronizing files and directories remotely as well as locally in Linux systems. With the help of rsync command we can copy and synchronize our data remotely and locally across directories, across disks and networks, perform data backups and mirroring between two Linux machines.

Some advantages and features of Rsync:

    It efficiently copies and sync files to or from a remote system.
    Supports copying links, devices, owners, groups and permissions.
    It’s faster than scp (Secure Copy) Why ? because rsync uses remote-update protocol which allows to transfer just the differences between two sets of files. First time, it copies the whole content of a file or a directory from source to destination but from next time, it copies only the changed blocks and bytes to the destination.
    Rsync consumes less bandwidth as it uses compression and decompression method while sending and receiving data both ends.
We might need to install rsync using yum install rsync command or apt install rsync in Debian.
Basic syntax of rsync is like rsync options source destination , and what are options:
rsync common options
Description
-v
copies data recursively (but don’t preserve timestamps and permission while transferring data)
-r
copies data recursively (but don’t preserve timestamps and permission while transferring data
-a
archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
-z
compress file data
-h
human-readable, output numbers in a human-readable format
Enough introduction lets see rsync in action:
2
backup-mydirectory.tar mydirectory pooler-cpuminer-2.5.0.tar.gz
3
cpuminer mynfs
4
5
### copy / sync a file on a local computer
6
[email protected]:~# rsync -zvh backup-mydirectory.tar /tmp/backups/
7
backup-mydirectory.tar
8
9
sent 316 bytes received 35 bytes 702.00 bytes/sec
10
total size is 10.24K speedup is 29.17
11
12
### copy/sync a directory on a local computer
13
[email protected]:~# tree mydirectory/
14
mydirectory/
15
├── dir1
16
│ └── file1.txt
17
├── dir2
18
│ └── file2.txt
19
├── dir3
20
│ └── file3.txt
21
└── myfile
22
23
3 directories, 4 files
24
25
[email protected]:~# rsync -avzh mydirectory/ /tmp/backups/
26
sending incremental file list
27
./
28
myfile
29
dir1/
30
dir1/file1.txt
31
dir2/
32
dir2/file2.txt
33
dir3/
34
dir3/file3.txt
35
36
sent 379 bytes received 115 bytes 988.00 bytes/sec
37
total size is 0 speedup is 0.00
Copied!
and to Copy a Directory from Local Server to a Remote Server:
1
[email protected]:~# rsync -azv mydirectory [email protected]:/home/
2
[email protected]'s password:
3
sending incremental file list
4
mydirectory/
5
mydirectory/myfile
6
mydirectory/dir1/
7
mydirectory/dir1/file1.txt
8
mydirectory/dir2/
9
mydirectory/dir2/file2.txt
10
mydirectory/dir3/
11
mydirectory/dir3/file3.txt
12
13
sent 389 bytes received 112 bytes 143.14 bytes/sec
14
total size is 0 speedup is 0.00
Copied!
and result:
2
mydirectory payam
Copied!
And visa versa, Copy/Sync a Remote Directory to a Local Machine:
1
[email protected]'s password:
2
receiving incremental file list
3
mydirectory/
4
mydirectory/myfile
5
mydirectory/dir1/
6
mydirectory/dir1/file1.txt
7
mydirectory/dir2/
8
mydirectory/dir2/file2.txt
9
mydirectory/dir3/
10
mydirectory/dir3/file3.txt
11
12
sent 124 bytes received 389 bytes 146.57 bytes/sec
13
total size is 0 speedup is 0.00
14
15
[email protected]:~# ls /tmp/
16
backups
17
_cafenv-appconfig_
18
config-err-aoEHBZ
19
mydirectory
20
systemd-private-5c9b83ef1a904073864354a680e17c01-colord.service-JNMAsj
21
systemd-private-5c9b83ef1a904073864354a680e17c01-rtkit-daemon.service-wp0M3z
22
systemd-private-5c9b83ef1a904073864354a680e17c01-systemd-timesyncd.service-fh7T2M
23
unity_support_test.0
24
VMwareDnD
25
vmware-payam
26
vmware-root
Copied!

rsync over ssh

Most of the time, rsync is run on top of ssh. In the rare case where someone has bothered to set up an rsync daemon, that uses port 873:
1
[email protected]:~# cat /etc/services | grep rsync
2
rsync 873/tcp
3
rsync 873/udp
Copied!
using SSH protocol while transferring our data you can be ensured that your data is being transferred in a secured connection with encryption so that nobody can read your data while it is being transferred over the wire on the internet.
when we use rsync we need to provide the user/root password to accomplish that particular task, so using SSH option will send our logins in an encrypted manner so that our password will be safe. use -e option to make sure we are using rsync over ssh:
1
### Copy a Directory from a Local Server to a Remote Server with SSH
2
[email protected]:~# rsync -azve ssh mydirectory [email protected]:/home/
3
[email protected]'s password:
4
sending incremental file list
5
6
sent 233 bytes received 20 bytes 72.29 bytes/sec
7
total size is 0 speedup is 0.00
8
9
### Copy a directory from a Remote Server to a Local Server with SSH
10
[email protected]:~# rsync -azve ssh [email protected]:/home/mydirectory /tmp/
11
[email protected]'s password:
12
receiving incremental file list
13
14
sent 28 bytes received 225 bytes 33.73 bytes/sec
15
total size is 0 speedup is 0.00
Copied!
Other rsync usefull commands:
    Show Progress While Transferring Data with rsync:
rsync -azve ssh --progress mydirectory [email protected]:/home/
    Include and exclude:
rsync -azve ssh --include 'D*' --exclude '*' mydirectory [email protected]:/home/ : include those files and directory only which starts with ‘D’ and exclude all other files and directory
    delete option
If a file or directory not exist at the source, but already exists at the destination, you might want to delete that existing file/directory at the target while syncing .
We can use ‘–delete‘ option to delete files that are not there in source directory:
rsync -azv --delete [email protected]:/home/mydirectory
rsync other usefull options (might seen in exam)
Description
--max-size='200K'
Set the Max Size of Files to be Transferred
--remove-source-files
Automatically Delete source Files after successful Transfer
--bwlimit=100
Set Bandwidth Limit and Transfer File
--dry-run
Do dry run, does not doing any thing, just show what would be done
by default rsync syncs changed blocks and bytes only, if you want explicitly want to sync whole file then you use ‘-W‘ option with it:
1
rsync -zvhW backup.tar /tmp/backups/backup.tar
Copied!

dd

The dd command stands for “data duplicator” and used for copying and converting data. It is very powerful low level utility of Linux. We should be very careful while working with this utility, data loss can convert the dd utility as a “data destroyer” for us. That’s why it is recommended that do not use dd command on a production machine until you get familiarity on this.
It can be used for making clones of volumes, filesystems, writing images to disks, and even erasing drives. The syntax of dd command is dd if=<source file name> of=<target file name> [Options]
dd command
Description
dd if=/dev/sda of=/dev/sdb
Clone one hard disk to another hard disk
dd if =/dev/sda2 of=~/hddpar1.img
Backing up a partition to a file
dd if=hddpar1.img of=/dev/sdb1
Restoring image to another disk
dd if =/dev/sda2 | bzip2 hddpar1.img.bz2
using bzip2 to compress when creating image
dd if=/home/myuser/abc.txt of=/mnt/abc.txt
dd command can be used as file copier as well
dd command can be pretty dangerous, watch out when using it.
Last modified 2yr ago