Pipes and Redirection in Linux
2. Standard Input and Output
All applications have three unique streams that connect them to the outside world. These are referred to as Standard Input, or stdin; Standard Output, or stdout; and Standard Error, or stderr.
Standard input is the default mechanism for getting input into an interactive program. This is typically a direct link to the keyboard when running directly in a terminal, and isn’t connected to anything otherwise.
Standard output is the default mechanism for writing output from a program. This is typically a link to the output terminal but is often buffered for performance reasons. This can be especially important when the program is running over a slow network link.
The standard error is an alternative mechanism for writing output from a program. This is typically a different link to the same output terminal but is unbuffered so that we can write errors immediately while the normal output can be written one line at a time.
3. Redirecting Into Files
One common need when we run applications is to direct the output into a file instead of the terminal. Specifically, what we’re doing here is replacing the standard output stream with one that goes where we want: In this case, a file.
This is typically done with the > operator between the application to run and the file to write the output into. For example, we can send the output of the ls command into a file called files as follows:
$ ls > files
We can do this after any command, including any needed arguments:
$ ls -1 *.txt > text-files
3.1. Appending To Files
$ ls -1 *.txt > files $ ls -1 *.text >> files $ ls -1 *.log >> files
We can use this to build up files using all manner of output if we need to, including just using echo to exact output lines:
$ echo Text Files > files $ ls -1 *.txt >> files $ echo Log Files >> files $ ls -1 *.log >> files
3.2. Redirecting Standard Error
All three of the standard streams have ID values, as defined in the POSIX specification and used in the C language. Standard input is 0, standard output is 1, and the standard error is 2.
When we use the redirect operators, by default, this applies to standard output. We can explicitly specify the stream to redirect, though, by prefixing it with the stream ID.
For example, to redirect standard error, from the cat command we would use 2>:
$ cat does-not-exist 2> log
We can be consistent and use 1> to redirect standard output if we wish, though this is identical to what we saw earlier.
We can also use &> to redirect both standard output and standard error at the same time:
$ ls -1 &> log
This sends all output from the command, regardless of which stream it is on, into the same file. This only works in certain shells, though – for example, we can use this in bash or zsh, but not in sh.
3.3. Redirecting into Streams
Sometimes we want to redirect into a stream instead of a file. We can achieve this by using the stream ID, prefixed by &, in place of the filename. For example, we can redirect into standard error by using *>&2*:
$ echo Standard Output >&1 $ echo Standard Error >&2
We can combine this with the above to combine streams, by redirecting from one stream into another. A common construct is to combine standard error into standard output, so that both can be used together. We achieve this using 2>&1 – literally redirecting stream 2 into stream 1:
# ls -1 2>&1
We can sometimes use this to create new streams, simply by using new IDs. This stream must already have been used elsewhere first though, otherwise, it is an error. Most often this is used as a stream source first.
For example, we can swap standard output and standard error by going via a third stream, using 3>&2 2>&1 1>&3:
$ ls -1 3>&2 2>&1 1>&3
This construct doesn’t work correctly in all shells. In bash, the end result is that standard output and standard error are directly swapped. In zsh the result is that both standard output and standard error have ended up on standard output instead.
3.4. Redirecting Multiple Streams
We can easily combine the above to redirect standard output and standard error at the same time. This mostly works exactly as we would expect – we simply combine the two different redirects on the same command:
$ ls -1 > stdout.log 2> stderr.log
This won’t work as desired if we try to redirect both streams into the same file. What happens here is that both streams are redirected individually, and whichever comes second wins, rather than combining both into the same file.
If we want to redirect both into the same file, then we can use &> as we saw above, or else we can use stream combination operators. If we wish to use the stream combination operators, then we must do this after we have redirected into the file, or else only the standard output gets redirected:
$ ls -1 > log 2>&1
4. Reading From Files
Sometimes we also want to achieve the opposite – redirecting a file into an application. We do this using the < operator, and the contents of the file will replace the standard input for the application:
$ wc < /usr/share/dict/words 235886 235886 2493109
When we do this, the only input that the application can receive comes from this source, and it will all happen immediately. It’s effectively the same as when the user types out the entire contents of the file at the very start of the application.
However, the end of the file is signaled to the application as well, which many applications can use to stop processing.
5. Piping Between Applications
$ ls | wc 11 11 138
This directly connects the standard output of our first application into the standard input of the second one and then lets the data directly flow between them.
5.1. Handling Standard Error
The standard error isn’t connected by default, so we’ll still get anything written to that stream appearing in the console. This is by design since the standard error is designed for error reporting and not for normal application output.
If we want to redirect standard error as well then we can use the technique from above to first redirect it into standard output and then pipe into the next application:
$ ls i-dont-exist | wc ls: i-dont-exist: No such file or directory 0 0 0 $ ls i-dont-exist 2>&1 | wc 1 7 44
If we want only to pipe standard error then we need to swap the standard output and standard error streams around, as we saw earlier.
$ ls 3>&2 2>&1 1>&3 | wc -l some-file 0 $ ls i-dont-exist 3>&2 2>&1 1>&3 | wc -l 1
5.2. Combining Pipes
$ docker images | cut -d' ' -f1 | tail -n +2 | sort -u | wc -l 17
This command looks scary, but we can break this down to the individual parts:
So we have a command here to get the number of unique docker images, ignoring the version of the image.
Many console applications are designed for exactly this use, which is why they can often consume input from standard input and write to standard output.
Certain applications also have special modes that allow for this – git, for example, has what is termed porcelain and plumbing commands, where the plumbing commands are specially designed to be combined in this manner while the porcelain commands are designed for human consumption.