103.4. Use streams, pipes and redirects

Weight: 4
Description: Candidates should be able to redirect streams and connect them in order to efficiently process textual data. Tasks include redirecting standard input, standard output and standard error, piping the output of one command to the input of another command, using the output of one command as arguments to another command and sending output to both stdout and a file.
Key Knowledge Areas:
Redirecting standard input, standard output and standard error
Pipe the output of one command to the input of another command
Use the output of one command as arguments to another command
Send output to both stdout and a file
Terms and Utilities:
tee
xargs
We have talked about basics of piping and redirecting in previous sections and in this tutorial we take a closer look at Linux techniques for redirecting standard IO streams.
Input, output, and streams
In computing, a stream is something that can transfer data. Data streams, like water streams, have two ends. They have a source and an outflow. 
A Linux shell, such as Bash, receives input and sends output as sequences or streams of characters. Stream of characters comes from or goes to a file, a keyboard, a window on a display, or some other IO device. 
stdin, stdout, and stederr are three standard streams that are established when a Linux command is executed. 
Linux shells use these three standard I/O streams , each of which is associated with file descriptor::
stdout is the standard output stream, which displays output from commands(file descriptor 1)
stderr is the standard error stream, which displays error output from commands(file descriptor 2)
stdin is the standard input stream, which provides input to commands(file descriptor 0)
 So  we can see that there are two output streams, stdout and stderr, and one input stream, stdin
What are file descriptors?
Streams Are Handled Like Files. We can read text from a file, and we can write text into a file. Both of these actions involve a stream of data. So the concept of handling a stream of data as a file isn’t that much of a stretch.
Each file associated with a process is allocated a unique number to identify it.
1
STDIN - /proc/<processID>/fd/0
2
STDOUT - /proc/<processID>/fd/1
3
STDERR - /proc/<processID>/fd/2
Copied!
 This is known as the file descriptor. Whenever an action is required to be performed on a file, the file descriptor is used to identify the file.  These values are always used for stdin, stdout, and stderr:
0: stdin 
1: stdout 
2: stderr
To make life more convenient the system creates some shortcuts for us:
1
 [email protected]:~/test-space/myfiles# ls -al /dev/std*
2
lrwxrwxrwx 1 root root 15 Dec  1  2018 /dev/stderr -> /proc/self/fd/2
3
lrwxrwxrwx 1 root root 15 Dec  1  2018 /dev/stdin -> /proc/self/fd/0
4
lrwxrwxrwx 1 root root 15 Dec  1  2018 /dev/stdout -> /proc/self/fd/1
Copied!
Redirecting standard IO
 Although the model for standard input and output is a serial stream of characters to and from a terminal, we might want to prepare input data in a file, or save output or error information in a file. That’s where .comes in.
Redirecting output
There are two ways to redirect output to a file:
n>  redirects output from file descriptor n to a file. You must have write authority to the file. If the file does not exist, it is created. If it does exist, the existing contents are usually lost without any warning.
n>>  also redirects output from file descriptor n to a file. Again, we must have write authority to the file. If the file does not exist, it is created. If it does exist, the output is appended to the existing file.
 The n in n> or n>> refers to the file descriptor. If it omitted, then standard output (file descriptor 1) is assumed.
The > and >> redirection symbols works with stdout by default. We can use one of the numeric file descriptors to indicate which standard output stream you wish to redirect.
> - standard output
2> - standard error
">" redirects stdout
1
[email protected]:~/test-space/myfiles# ls -l
2
total 1144
3
drwxr-xr-x 2 root root    4096 Sep 30 05:42 dir1
4
drwxr-xr-x 2 root root    4096 Oct  5 05:25 dir2
5
-rw-r--r-- 1 root root    2084 Sep 30 05:44 myconf.txt
6
-rw-r----- 1 root root 1156369 Sep 30 05:41 mylog.txt
7
[email protected]:~/test-space/myfiles# ls -l > list
8
[email protected]:~/test-space/myfiles# cat list 
9
total 1144
10
drwxr-xr-x 2 root root    4096 Sep 30 05:42 dir1
11
drwxr-xr-x 2 root root    4096 Oct  5 05:25 dir2
12
-rw-r--r-- 1 root root       0 Oct  9 01:42 list
13
-rw-r--r-- 1 root root    2084 Sep 30 05:44 myconf.txt
14
-rw-r----- 1 root root 1156369 Sep 30 05:41 mylog.txt
Copied!
"2>" redirects stderr
1
[email protected]:~/test-space/myfiles# ls -l BlahBlah
2
ls: cannot access 'BlahBlah': No such file or directory
3
[email protected]:~/test-space/myfiles# ls -l BlahBlah 2> error
4
[email protected]:~/test-space/myfiles# ls -l
5
total 1152
6
drwxr-xr-x 2 root root    4096 Sep 30 05:42 dir1
7
drwxr-xr-x 2 root root    4096 Oct  5 05:25 dir2
8
-rw-r--r-- 1 root root      56 Oct  9 01:43 error
9
-rw-r--r-- 1 root root     267 Oct  9 01:42 list
10
-rw-r--r-- 1 root root    2084 Sep 30 05:44 myconf.txt
11
-rw-r----- 1 root root 1156369 Sep 30 05:41 mylog.txt
12
[email protected]:~/test-space/myfiles# cat error 
13
ls: cannot access 'BlahBlah': No such file or directory
Copied!
Append
 Commands with a double bracket do not overwrite the destination’s existing contents, they append.
>> - standard output
2>> - standard error
1
[email protected]:~/test-space/myfiles# date
2
Sat Oct 12 02:52:04 PDT 2019
3
[email protected]:~/test-space/myfiles# date > mydate
4
[email protected]:~/test-space/myfiles# cat mydate
5
Sat Oct 12 02:52:10 PDT 2019
6
[email protected]:~/test-space/myfiles# date > mydate 
7
[email protected]:~/test-space/myfiles# cat mydate 
8
Sat Oct 12 02:53:00 PDT 2019
9
[email protected]:~/test-space/myfiles# date >> mydate 
10
[email protected]:~/test-space/myfiles# cat mydate 
11
Sat Oct 12 02:53:00 PDT 2019
12
Sat Oct 12 02:53:23 PDT 2019
Copied!
 To clobber or to noclobber?
We said that output redirection using n> usually overwrites existing files. You can control this with the noclobber option of the set builtin.  use set -C for enabling noclobber:
1
[email protected]:~$ set -o | grep noclobber
2
noclobber      	off
3
[email protected]:~$ ls
4
Desktop    Downloads         Music     Public     Videos
5
Documents  examples.desktop  Pictures  Templates
6
[email protected]:~$ ls > list1
7
[email protected]:~$ set -C
8
[email protected]:~$ set -o | grep noclobber
9
noclobber      	on
10
[email protected]:~$ ls > list1
11
bash: list1: cannot overwrite existing file
Copied!
If it has been set, we can override it using n>|
1
[email protected]:~$ ls >| list1
Copied!
use set +C for turning globbing off.
redirect both standard output and standard error
Sometimes, we might want to redirect both standard output and standard error into a file. This is often done for automated processes or background jobs so that we can review the output later.
Use &> or &>> to redirect both standard output and standard error to the same place.
1
[email protected]:~/test-space/myfiles# ls blah &>result
2
[email protected]:~/test-space/myfiles# cat result 
3
ls: cannot access 'blah': No such file or directory
4
[email protected]:~/test-space/myfiles# ls blah2 &>>result
5
[email protected]:~/test-space/myfiles# cat result 
6
ls: cannot access 'blah': No such file or directory
7
ls: cannot access 'blah2': No such file or directory
Copied!
We can do it in another way,  redirect file descriptor n and then redirect file descriptor m to the same place using the construct m>&n or m>>&n.
&1 and &2 and &0  reffer to current place of stdout , stderror and stdin.
1
[email protected]:~/test-space/myfiles# ls blah3 > result 2>&1
2
[email protected]:~/test-space/myfiles# cat result 
3
ls: cannot access 'blah3': No such file or directory
Copied!
 The order in which outputs are redirected is important. For example,
command 2>&1 >output.txt
 is not the same as
command >output.txt 2>&1
In the first case, stderr is redirected to the current location of stdout and then stdout is redirected to output.txt, but this second redirection affects only stdout, not stderr. In the second case, stderr is redirected to the current location of stdout and that is output.txt.
 redirecting to /dev/null
At other times, we might want to ignore either standard output or standard error entirely. To do this, redirect the appropriate stream to the empty file, /dev/null.
/dev/null is a special file called the null device. it is also called the bit-bucket or the black-hole because it immediately discards anything written to it and only returns an end-of-file EOF when read.
1
[email protected]:~/test-space/myfiles# ls -l
2
total 1156
3
drwxr-xr-x 2 root root    4096 Sep 30 05:42 dir1
4
drwxr-xr-x 2 root root    4096 Oct  5 05:25 dir2
5
-rw-r--r-- 1 root root     170 Oct  9 02:25 error
6
-rw-r--r-- 1 root root      97 Oct  9 03:17 list
7
-rw-r--r-- 1 root root    2084 Sep 30 05:44 myconf.txt
8
-rw-r--r-- 1 root root      58 Oct 12 02:53 mydate
9
-rw-r----- 1 root root 1156369 Sep 30 05:41 mylog.txt
10
[email protected]:~/test-space/myfiles# ls -l Blah 2>/dev/null
Copied!
redirecting input
"<" redirects stdin 
1
[email protected]:~/test-space/myfiles# cat list 
2
total 1144
3
drwxr-xr-x 2 root root    4096 Sep 30 05:42 dir1
4
drwxr-xr-x 2 root root    4096 Oct  5 05:25 dir2
5
-rw-r--r-- 1 root root       0 Oct  9 01:42 list
6
-rw-r--r-- 1 root root    2084 Sep 30 05:44 myconf.txt
7
-rw-r----- 1 root root 1156369 Sep 30 05:41 mylog.txt
8
[email protected]:~/test-space/myfiles# wc < list 
9
  6  47 267
Copied!
or simply wc 0< list do the same thing.
here documents
A here document (or heredoc) is a way of getting text input into a script or command without having to feed it in from a separate file.
1
Command << HeredocDelimiter
2
. . .
3
. . .
4
. . .
5
HeredocDelimiter
Copied!
<<  redirects standard input and the command will take input until "HeredocDelimiter" string .
 Note: There should not be any space between the << symbol and the limit string. If there are any characters between the limit string and the << symbol then the here document is unlikely to work.
1
[email protected]:~/test-space/myfiles# cat << EOF > /tmp/myfile.txt
2
> Theis line will be written to the file.
3
>     this line is 2nd line
4
> this is last line.
5
> EOF
6
[email protected]:~/test-space/myfiles# cat /tmp/myfile.txt 
7
Theis line will be written to the file.
8
    this line is 2nd line
9
this is last line.
Copied!
Reminder: "-" A hyphen (used alone) generally signifies that input will be taken from stdin as opposed to a named file. try: cat - << EOF > interesting.txt
here string
 <<< Redirect a single line of text to the stdin of cmd. This is called a here-string.
1
cmd <<< "string"
Copied!
Pipes
We use the | (pipe) operator between two commands to direct the stdout of the first to the stdin of the second. command | command
1
[email protected]:~/test-space/myfiles# ls -l
2
total 1156
3
drwxr-xr-x 2 root root    4096 Sep 30 05:42 dir1
4
drwxr-xr-x 2 root root    4096 Oct  5 05:25 dir2
5
-rw-r--r-- 1 root root     170 Oct  9 02:25 error
6
-rw-r--r-- 1 root root      97 Oct  9 03:17 list
7
-rw-r--r-- 1 root root    2084 Sep 30 05:44 myconf.txt
8
-rw-r--r-- 1 root root      58 Oct 12 02:53 mydate
9
-rw-r----- 1 root root 1156369 Sep 30 05:41 mylog.txt
10
[email protected]:~/test-space/myfiles# ls -l | cut -f1 -d' '
11
total
12
drwxr-xr-x
13
drwxr-xr-x
14
-rw-r--r--
15
-rw-r--r--
16
-rw-r--r--
17
-rw-r--r--
18
-rw-r-----
19
[email protected]:~/test-space/myfiles# ls -l | cut -f1 -d' ' | wc -l
20
8
Copied!
Some commands like grep can accept input as parameters, but some commands accepts arguments
Parameter vs Argument: 
Parameter is variable in the declaration of function.
Argument is the actual value of this variable that gets passed to function.
1
                 Parameters
2
                   |     |
3
function test ( param1 , param2 ){
4
    return ( param1 + param2 );
5
    }
6
test (5, 6)
7
      |  |
8
     Arguments
Copied!
Using output as arguments
In the privious of pipelines, we learned how to take the output of one command and use it as input to another command. Instead , what if  we want to use the output of a command or the contents of a file as arguments to a command rather than as input. Pipelines don’t work for that.
 Three common methods are:
1.
The xargs command
2.
The find command with the -exec option (previous section)
3.
Command substitution ( will be discussed later)
xargs
xargs is a Unix command which can be used to build and execute commands from standard input.
1
xargs [options] [command]
Copied!
If no command is specified, xargs executes echo by default.
1
[email protected]:~/test-space/myfiles# ls | xargs 
2
dir1 dir2 interesting.txt list myconf.txt mydate mylog.txt
Copied!
 There are several ways in which xargs is useful in daily usage of the command line.
We can replace occurrences of arguments via -I switch:
1
[email protected]:~/test-space/myfiles# find . -type f | xargs -I FILE  echo this is my file FILE yes
2
this is my file ./dir1/myvideo.mp4 yes
3
this is my file ./interesting.txt yes
4
this is my file ./list yes
5
this is my file ./dir2/zabbix-cli-rpm-2.1.1-1.tar.gz yes
6
this is my file ./dir2/zabbix-cli-manual-1.7.0.pdf yes
7
this is my file ./mylog.txt yes
8
this is my file ./myconf.txt yes
9
this is my file ./mydate yes
Copied!
it is also possible to run multiple commands with xargs 
cat a.txt | xargs -I % sh -c {command1; command2; ... }
1
[email protected]:~/test-space# cat foo 
2
one
3
two
4
three
5
​
6
[email protected]:~/test-space# cat foo | xargs -I % sh -c 'mkdir  %  ;echo  "directory % has been made" '
7
directory one has been made
8
directory two has been made
9
directory three has been made
10
​
11
[email protected]:~/test-space# ls
12
foo  one  three  two
Copied!
With -L option, the input will break by line and not by blanks. Other options:
1
xargs options :
2
-0 : input items are terminated by null character instead of white spaces
3
-a file : read items from file instead of standard input
4
-d,–delimiter = delim : input items are terminated by a special character
5
-E eof-str : set the end of file string to eof-str
6
-I replace-str : replace occurrences of replace-str in the initial arguments with names read from standard input
7
-L max-lines : use at-most max-lines non-blank input lines per command line.
8
-p : prompt the user about whether to run each command line and read a line from terminal.
9
-r : If the standard input does not contain any nonblanks, do not run the command
10
-x : exit if the size is exceeded.
11
–help : print the summary of options to xargs and exit
12
–version : print the version no. of xargs and exit
Copied!
xarg command example
Description
xargs --version
Prints the version number of xargs command
xargs -a test.txt
It will show contents of file
xargs -p -a test.txt
-p option prompts for confirmation before running each command line.
xargs -r -a test.txt
-r option ensures if standard input is empty, then command is not executed
tee
 Sometimes we might want to see output on the screen while saving a copy for later. While we could do this by redirecting the command output to a file in one window and then using tail -f to follow the output in another screen, using the tee command is easier.
1
tee [OPTION]... [FILE]...
Copied!
 tee reads the standard input and writes it to both the standard output and one or more files.
tee always work after the '|' and it does both the tasks simultaneously.example:
1
[email protected]:~/test-space/myfiles# ls  | tee list.txt
2
dir1
3
dir2
4
interesting.txt
5
list
6
list.txt
7
myconf.txt
8
mydate
9
mylog.txt
10
[email protected]:~/test-space/myfiles# cat list.txt 
11
dir1
12
dir2
13
interesting.txt
14
list
15
list.txt
16
myconf.txt
17
mydate
18
mylog.txt
Copied!
with  -a  option It basically do not overwrite the file but append to the given file.
​
.
.
.
​https://developer.ibm.com/tutorials/l-lpic1-103-4/​
​https://www.gnu.org/software/libc/manual/html_node/Streams-and-File-Descriptors.html​
​https://geek-university.com/linux/streams/​
​https://ryanstutorials.net/bash-scripting-tutorial/bash-input.php​
​https://www.howtogeek.com/435903/what-are-stdin-stdout-and-stderr-on-linux/​
​https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-i-o-redirection​
​https://www.serverwatch.com/columns/article.php/3860446/Shell-Scripts-and-Here-Documents.htm​
​https://linuxhint.com/bash-heredoc-tutorial/​
​https://stackoverflow.com/questions/40230008/whats-the-usages-of-hyphen-in-linux-shell​
​https://www.geeksforgeeks.org/xargs-command-unix/​
​https://www.tecmint.com/xargs-command-examples/​
.

103.3. Perform basic file management

103.5. Create, monitor and kill processes

Last modified 1yr ago

Copy link

Contents

Input, output, and streams

Redirecting standard IO

Using output as arguments

xargs

tee