Wizard Boot Camp, Part Three: Using Standard I/O from Shells

Last month's column, the second in this series of obscure Linux features that wizards should know, introduced Standard I/O. This month we'll see how to take advantage of Standard I/O from a shell-- including an example using named pipes (FIFOs).

Last month’s column, the second in this series of obscure Linux features that wizards should know, introduced Standard I/O. This month we’ll see how to take advantage of Standard I/O from a shell– including an example using named pipes (FIFOs).

Duplicating a file descriptor

The Bourne shells’ operator m>&n copies a file descriptor. The first digit, m, is the file descriptor you want to change. The second digit, n, is the file descriptor to copy.

For instance, 2>&1 copies f.d. 2 from f.d. 1. This makes f.d. 2 point to the same open file as f.d. 1 does. Because f.d. 2 is the standard error and f.d. 1 is standard output, after using that operator, stderr will point to the same place as stdout.

The operator 2>&1 isn’t any use in the default situation where both stderr and stdout point to the terminal. One place it is handy is when you’re piping the output of one command to another– for instance, when you’re viewing the output of a program with a pager program like less (1). Without the 2>&1 operator, like this:

$ grep xyz * | less

Only the stdout of grep is piped to less. grep‘ s stderr goes to the terminal– along with the stdout of less. So, if grep outputs an error, it’s likely to be mixed into the output text from less; you may not notice the error at all. Using 2>&1 fixes that:

$ grep xyz * 2>&1 | less

With that operator, grep‘ s stderr goes the same place as grep‘ s stdout: down the pipe, to be read by (and paged by) less. bash sets up the pipe first– connecting grep‘ s stdout to the stdin of less– then it redirects grep‘ s stderr. The 2>&1 must be written to the left side of the pipe operator; it applies to the command on the left-hand side.

The order of redirections is important: the shell reads a command line and processes redirections from left to right.

Let’s see another example: using 2>&1 together with the file-redirection operator >. Here’s the correct way to use it:

$ someprog > $outfile 2>&1

Why is that correct?

Reading that command line from left to right, you’re first telling the shell to redirect the stdout of someprog to the file named in $outfile. Next, you’re telling the shell to make someprog‘ s stderr (f.d. 2) go the same place as its stdout: that is, to $outfile.

Let’s compare the previous correct example with the next one, which doesn’t do what we want:

$ someprog 2>&1 > $outfile

What’s wrong?

By default, as the shell starts to handle redirections, both f.d. 1 and f.d. 2 are going the same place: to the terminal. As the shell reads left to right, it sees 2>&1 and copies f.d. 1 to f.d. 2. But, at this point, both f.d. 1 and f.d. 2 are still both going to the terminal. So, the operator has no effect. Then > $outfile redirects stdout to $outfile, without affecting stderr.

In the same way, you can force command substitution to collect both stdout and stderr– instead of just stdout. For instance, to capture all the output of someprog into a shell variable progout:

progout=$(someprog 2>&1)

Redirecting a file descriptor for all commands: exec

Let’s start with a little bit of background info. If you pass a command name (and optional arguments) to the shell’s built-in exec command, that command will replace the shell and continue running within the same process.

For example, if you’re running the bash shell and you want to run zsh instead, type:

$ exec zsh

The original bash program is gone, replaced (in the same process) by an instance of zsh.

Using exec this way was important in the early days of Unix because it saved precious system resources.

(The old shell didn’t stay around, waiting for a child process to finish.)

In Bourne-type shells, the exec command can also redirect open files permanently– that is, until the current shell process exits. It’s usually used in shell scripts, though it also works from a shell prompt.

For instance, to redirect the standard output of all following commands to a file named output, use this command:

exec > output

To redirect both stdout (file descriptor 1) and stderr (f.d. 2) to a file, use the 2>&1 operator:

exec > output 2>&1

From there until the shell terminates, both stdout and stderr will go to the file output and the script should now run silently.

Using higher file descriptors

So far we’ve been using the default file descriptors 0, 1, and 2. File descriptors 3 through 9 generally aren’t used in shell scripts. (File descriptors 10 and above may be used internally by the shell, so you shouldn’t change them.)

What can you use them for– and how?

One way is to gather the output of certain commands within a loop. As we saw last month, a loop is a shell statement, so the inputs and outputs of commands within the loop can be redirected before or after the loop. For example, let’s say you’re writing a long-running loop that logs the time each iteration starts into a file start-times and the directories visited by the loop in a file named dirnames. The inefficient way to do this, which opens and closes those log files on each pass through the loop, would be:

get-dirnames |
while read dir
   cd "$dir"

   date >> start-times
   echo "$dir" >> dirnames

It’s more efficient to open the two log files once, then write directly to those open files. Here’s the improved loop:

get-dirnames |
while read dir
   cd "$dir"
   date 1>&3
   echo "$dir" 1>&4

done 3>start-times 4>dirnames

As before, date 1>&3 means” make f.d. 1 go the same place as f.d. 3″: to the open file start-times, which was opened before the shell started to run the loop.

This is especially useful when more than one command within the loop needs to write to a particular file; it avoids constant re-opening of that file.

Higher file descriptors can also be used to” remember” where another file descriptor is pointing. That’s handy if, say, you want to preserve the location of a file descriptor while you temporarily change it. It’s also handy for swapping file descriptors.

For instance, you might want to use command substitution to capture the standard error of a process instead of the default, the standard output. Let’s look at an example.

In this situation, the variable text gets the stdout of the program prog. prog‘ s stderr goes to the default location (typically, the terminal):


Here’s how to route prog‘ s stderr to text and its stdout to wherever the stderr had been going (typically, the terminal):

text=$(prog 3>&2 2>&1 1>&3)

Reading that left to right, we’re using f.d. 3 to remember where f.d. 2 (stderr) was pointing.

Next we’re making f.d. 2 point to the place that f.d. 1 is currently pointing– which is to the shell variable $text, via command substitution. Finally, we make f.d. 1 (the stdout of prog) point to the place that f.d. 3 has been pointing– which is the previous location of f.d. 2.

If that makes your head hurt, try drawing a table, or a series of diagrams, showing where each file descriptor points after each m>&n operator.

To close an open output file, use m>&-, where m is the file descriptor.

Finally: to redirect an open input file, use m<&n. To close an open input file, use m<&-.

Filenames? Who needs filenames?

We mentioned last month that, once the shell has opened a file, the filename isn’t needed anymore.

You use the file descriptor instead. This fact leads to an obscure, and potentially useful, trick that shows

a lot about how open files work. To demonstrate this, here’s a shell script that creates a small file named afile with ten lines from the system password file.

Next, the script opens the file for reading on the standard input, uses head-1 to read and output a line from the file (via head‘ s standard input), removes the file, uses ls to confirm that the file is gone, then reads and outputs another line from the file:

head /etc/passed > afile
exec < afile
head -1
rm afile
ls afile
head -1

Running that script gives output like this:

ls: afile: No such file or directory

The important thing to understand here is that rm (1) doesn’t actually remove the physical file blocks from the disk. Instead, it removes the link to the file– that is, the filename that allows the system to locate the corresponding disk blocks. The disk blocks themselves aren’t removed (or added to a list of free disk blocks) until all processes with that file open have terminated.

You can see the open file by adding calls to lsof-p $$ before the exec command and after the rm command.

(lsof-p $$ lists all files open by process $$, the current shell process.) After rm, the lsof output will show /.../afile(deleted). You’ll also see the file descriptor numbers for stdin, stdout and stderr in the FD column– and the terminal’s filename, like /dev/pts/35, in the NAME column for the standard I/O files.

You might think of using this trick for extra security with temporary files in /tmp: if the temporary file doesn’t have a name, it’s hard for an attacker to replace or corrupt the file. Before you use this trick, though, consider the problems it can cause for a system administrator (maybe that’s you!) if a filesystem fills up. The open file is occupying disk space, but there’s no link to the file– that is, no filename– that the sysadmin can give to rm

to free disk space. If the sysadmin knows what’s happening, she can use lsof to find the process that has the file open and kill it.

This trick can also cause problems on some versions of NFS, where NFS has to make a temporary filename to stand in for the missing filename.

Named pipes

Typing proga|progb tells a shell to create a pipe that connects the standard output of proga to the standard input of progb. The system also manages the two processes so that proga will be stopped when it’s written some data for the progb to read– until progb has actually read that data. This kind of pipe is unnamed or anonymous; it connects two specific processes.

A named pipe or FIFO(” first in, first out”) is similar. But it’s a reusable pipe in the filesytem, with a name like any other file. A FIFO is handy in cases when you want two arbitrary processes to communicate. The first process writes to the FIFO with the shell’s > operator– or by writing directly to that filename. The second process reads from the FIFO by opening it as it would open a plain text file– or by using the shell’s < operator.

To make a FIFO, use mkfifo (1). When you list a FIFO with ls-l, the file type is p. Using ls-F, a FIFO is marked with a trailing | character:

$ mkfifo readme
$ ls -l readme
prw-r--r--  1 jpeek users 0 ... readme
$ ls -F readme

One place you can use a FIFO is when you want to watch the output of a process running on one terminal from a window on a different terminal. Here’s a simple shell script named writer that runs the date command every three seconds, sending its output both to the terminal (via stdout) and to our FIFO named readme (via f.d. 3):

while sleep 3
   echo "$dateout"
   echo "$dateout" 1>&3
done 3> readme

In one terminal, run writer. Not much happens because the process is being blocked, waiting for the write to the FIFO to succeed.

In the other terminal, run cat readme and wait a moment. Now you should start seeing output in both terminals every three seconds: via stdout on the terminal running writer, and via the FIFO on the terminal running cat. If you kill either process (either writer or cat) with CTRL-C, both processes exit. But the FIFO is still in the filesystem, so you can use it again. To remove the FIFO, use rm.

Jerry Peek is a freelance writer and instructor who has used Unix and Linux for more than 25 years. He’s happy to hear from readers; see http://www.jpeek.com/contact.html.

Comments on "Wizard Boot Camp, Part Three: Using Standard I/O from Shells"


Great article, very useful information not only for shell scripts, but also to deeply understand the standard I/O. Thanks.


Highly informative. A worthy effort. Thank you for sharing your knowledge.


Nice composition. Some things things to add: The above is (as far as I can see) based on bourne-shell and deriviants like ksh, bash and such. The c-shell (and deriviants like tcsh) have a different aproach on how stdio is handled. My experience tells that I prefer the bourne-shell in scripting and the c-shell on the commandline.

There is a page on ‘useless use of cat’ (just google for that). As far as I remember is it using stdio.


A ‘definitely not to be missed’ series – Thank you
- David


Hey Jerry

the “exec


Hey Jerry

the exec with afile as stdin trick doesn’t work for me
it seems that exec it trying to execute the
lines of afile and fails (bash 3.2.25)…

just FYI :)

anyways, great article…thanks.



Thanks for a really great article. These are things I really need, and have not been able to learn anywhere else.

I have the same problem as ‘bros’ with GNU bash, version 3.00.16(1): the ‘exec



Thanks for a really great article. These are things I really need, and have not been able to learn anywhere else.

I have the same problem as ‘bros’ with GNU bash, version 3.00.16(1): the ‘exec [from] afile’ reads afile as a list of commands (and closes the terminal when finished).

[and again same problem as 'bros' the editor is not happy about the left-arrow "[from]” sign]



I can confirm that the exec redirection script that you both have problems with works exactly as advertised under bash 3.2.25 and 2.05b (well, once you’ve corrected /etc/passed to /etc/passwd, of course).

On the off-chance that it’s being run under a non-Bourne shell, as Corne pointed out, try prefixing your script with the line:


If that doesn’t work, post the actual error message you see.


Thank you, Jerry.
When we review the command mode, we always understand something new, like in Literature when we read a good book once more.


Whoever complained about the so called lack of wizardry in the first installment of the series : keep quiet now !

Good work.


Great article! Thanks!

@bros/Soren: I experience the same problem with my bash 2.05b.0.

The following does work and demonstrates the principle being described:

cat afile | (ls afile; rm afile; ls afile; head -1)

However, the head command on my system causes STDIN to stop flowing; i.e.:

cat afile | (head -1; head -1)

only returns 1 line:

daemon:x:1:1:Owner of many system processes:/usr/sbin:/bin/sh

A workaround is to replace each 'head -1' with ‘read line; echo “$line”‘



We put our daemons right inside the inittab (poor practice, I know) and I always wondered about that mysterious 2>$1 on all of them. This really clears it up.


Oops, I mean 2>&1.

Hey There. I found your blog using msn. This is a very well written article. I’ll make sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll certainly comeback.

This is very fascinating, You’re a very professional blogger. I’ve joined your rss feed and sit up for in search of extra of your fantastic post. Additionally, I have shared your web site in my social networks!

This design is steller! You certainly know how to keep a reader entertained. Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Great job. I really loved what you had to say, and more than that, how you presented it. Too cool!

Below you will come across the link to some web pages that we assume you need to visit.

I like this web site so much, bookmarked. “Nostalgia isn’t what it used to be.” by Peter De Vries.

Usually posts some extremely intriguing stuff like this. If you?re new to this site.

u4PpUL pdjsdvqfquyl, [url=http://txdurnwpeepa.com/]txdurnwpeepa[/url], [link=http://zjdtylkkgmfy.com/]zjdtylkkgmfy[/link], http://bjqtwgcwpobu.com/

The subsequent time I learn a blog, I hope that it doesnt disappoint me as much as this one. I imply, I do know it was my choice to learn, however I really thought youd have something attention-grabbing to say. All I hear is a bunch of whining about one thing that you could repair if you werent too busy looking for attention.

I really do consider every one of the ideas you’ve presented in your post.
These are really convincing and might certainly work.
Still, the posts are too quick for newbies. May just you please
extend them a little from next time? Thanks a lot for the post.

Also visit my website AmeeRBiggs

Very great post. I just came across your blog and wanted to bring up that I’ve really loved browsing your weblog posts.
In any event I am going to be subscribing for your feed and I’m hoping you write again very soon!

Also visit my page: WyattHGrotts

Hello it’s me, I am also visiting this web site daily, this website is in fact
pleasant and the people are really sharing fastidious thoughts.

Review my webpage – BartHSellers

Leave a Reply