Wizard Boot Camp, Part One: Linux, Shells & Commands

Here's the start of a series on little-known topics that wizards should know. Knowledge of the Harry Potter series not required.

Over the past five years, this column has covered a lot of not-so-obvious features of Linux — mostly techniques that you’d use from the shell or another non-graphical environment. The start of your columnist’s sixth year seemed like a good time to try something a little different: a series of columns on features of Linux and shells that many books about Linux basically ignore, but that wizards should know. Studying these obscure features isn’t just for trivia buffs. Understanding how each feature works can help you grok “the big picture” of some pieces that make your favorite operating system the power platform that it is.

This month, let’s start with a quick overview of Linux and shell concepts that you should know, then dig into some shell features that might be new to you.

Linux Overview in Ten Paragraphs

Let’s review some of the fundamentals to start from the same place. (Please note: this isn’t an exact, complete description. Some details are simplified or omitted.)

A Linux system manages a set of processes, letting them share system resources (the processors, disks, networks, and so on). Basically, a process is a running program, a program waiting to run, or a suspended program. A process has a name (often the name of the program file that it’s executing) and a list of arguments (for instance, some filenames). Each process has a unique process ID, or PID, that’s assigned when the process starts. The ps utility lists processes.

A process runs under the user ID, or UID, of one of the accounts on the system. In most cases, a process runs with the UID of the account it was started under. So, if user joe starts a text editor, that process inherits the abilities that joe has: for instance, it can read and write his files. The superuser, root, has UID 0. A process running under UID 0 is privileged: it can change its UID (actually, its effective UID) to any other UID on the system. The su and sudo programs, which run under UID 0, allow any user to run processes as another user.

A parent process can start other processes called child processes. A child process inherits many of its parent’s attributes — as you’ll see next. A child process can also start its own child processes. The process named init, with PID 1, is the parent (or “grandparent”, or great-grandparent, and so on) of all processes. After a parent process starts a child process, it typically waits for the child to finish. However, if the parent doesn’t wait, it can “disown” the child, and that “orphaned” process becomes a child of init.

A process has attributes including: a current directory (the starting place for relative pathnames); a list of environment variables (for example, EDITOR=emacs or EDITOR=vi); and a list of open files for input/output.

The open files include three as standard input and output: the standard input, abbreviated stdin, file descriptor number 0; standard output, stdout, with descriptor 1; and standard error, stderr, with file descriptor 2.

A child process inherits attributes from its parent. For instance, the child starts in the same current directory as its parent, and has the same open files and environment variables. Once the child begins running, it can manage these attributes, and any changes made in the child process don’t affect its parent.

When a process starts, its standard I/O files are often connected to a terminal device, such as an X Window System terminal or the system console. So, for instance, if a process prints a message to its stdout, the text appears on the terminal. Because a child process inherits its parent’s open files, it too writes to and reads from the same terminal (by default) as its parent.

When a process exits, it returns a numeric exit status to its parent. By convention, a zero status signals success and a non-zero status means there was some kind of error.

If a process was started under an X Window System, it can also open windows there. Then, although that process also has standard I/O available, it generally won’t use them.

Shells in Five Easy Steps

One particularly interesting kind of program is the shell. A shells is a program designed especially to run other programs (usually, non-shell programs– though a shell can start a child shell). Here’s how a shell works:

1.The shell outputs a prompt and waits for input (unless the shell is reading a shell script file).

2.It reads the command-line, parses the command-line to find the command names, handles operators like | and > (which tell the shell to redirect the child’s standard I/O), and interprets any arguments (and special characters in them, such as wildcards).

3.The shell then starts one or more child processes to run the specified program (s), unless the command you specified is built-in to the shell. (For example, the built-in command cd changes the shell’s current directory. It doesn’t start a child process because the command needs to affect the current process.)

4.It waits for the program to finish (unless you suspend the program after it’s started running by typing Control-Z, or if you started the program in the background by typing & at the end of the command-line).

5.The shell then repeats step 1, unless the script file ends or you used the built-in command exit, in which case the shell process terminates.

There are several different shells. This series of columns cover bash.

How Shells Get Commands

A shell is a command interpreter (as you saw in the previous section) that reads commands and runs them. Commands can come from several places:

*A shell can run a command that you pass as a single argument when the shell is invoked. You might do this from a non-shell program that needs to run a command-line entered by a user. Most shells have a –c option for this purpose. For example, to cd to the user’s subdirectory named lib, list its contents, and rename a file, another program could execute:

$ sh -c "cd ~/lib && ls; mv afile afile.old"

The shell expands ~ into the absolute pathname of the user’s home directory; the result is passed to cd. The && operator executes ls only if cd succeeded. The ; operator executes mv after ls finishes.

*If you pass an argument that isn’t associated with an option, the shell opens that argument as a file and reads command-lines from it. This is how shell scripts are read.

This method and the previous one make the shell non-interactive: the shell doesn’t prompt a user for input, it simply reads one or more commands and executes them. After the last child process exits, the shell process itself exits.

*A shell invoked without any arguments or only with certain options runs in interactive mode. After reading initialization files like .bashrc, the shell outputs a prompt string and waits to read input from stdin.

The latter is the most complex (and interesting) way that commands are input. Let’s look into it some more.

Linux newbies think of a shell reading everything they type on the keyboard until the Return (Enter) key is pressed. That’s the simplest case. Shells like bash with built-in command line editing can edit multiple-line commands like while and for loops. You might be wondering how multi-line statements like these are entered from a prompt.

As you’re entering a command-line, the shell reads the line you’ve typed (actually, the text from its standard input) when you press Enter. The shell checks the line to be sure it’s a complete statement. If it’s not complete, bash prints its secondary prompt, >, and keeps reading input until it finds the end of the statement.

What’s an incomplete statement? Here are some examples:

*A line with unmatched single quotes (), double quotes (), and backquotes (`); unquoted opening parentheses ( and curly braces {; and so on.

For instance, to make a shell script output a multi-line message, you don’t need to use one echo command for each line. Instead, pass a multi-line argument (containing newline characters) to echo, surrounded by quote marks.

$ echo 'This is the first line
and this is the second line.'

Here’s how that example looks when it’s typed at the primary ($) and secondary (>) shell prompts:

$ echo 'This is the first line
> and this is the second line.'
This is the first line
and this is the second line.

*A loop before you type done. Here’s a for loop that finds all TIFF images (all filenames ending with .tif) in the current directory), strips off the .tif, and uses the ImageMagick convert utility to create a JPEG image:

$ for file in *.tif
> do
>   echo "doing $file..."
>   base=${file%.tif}
>   convert "$file" "$base.jpg"
> done

*An unfinished pipeline. Typing a pipe symbol at the end of a line tells the shell that there’s another command to be read. Writing pipelines this way can make a shell script easier to read. You can intersperse comments, too:

# List the 20 largest files:
ls -S | head -20 |
# open and search for errors:
xargs grep ERROR |
# write result to logging host:
ssh $loghost "cat > errors"

Problematic Ways that Shells Get Commands

The previous section discussed shells reading commands interactively from a terminal — actually, from standard input. A shell can also read commands from its standard input non-interactively, by redirection from another process. Here are two examples:

$ bash < script-file
$ command-generator | bash

script-file is a shell script file, and command-generator is a program that outputs shell command lines for bash to read and execute.

The problem here is that the standard input of bash is redirected. So, if any of the commands in script-file, or in the output of command-generator, try to read from the standard input, they’ll take input from script-file or command-generator, which is almost certainly not the right place! (Because child processes inherit the parent’s standard I/O channels, the programs that bash runs get standard input from the same place as bash does.)

This is a big “gotcha” in redirecting a shell’s standard input. You may be able to work around this with the shell operator m<&n (which we’ll cover in November), but it’s likely to be painful at best.

The Shell Expands Wildcards

A wildcard is a character like * or ? that stands for other characters in a filename or pathname. Some operating systems require each program to interpret wildcards. Linux doesn’t.

Linux shells interpret wildcards. They replace arguments containing wildcards with a list of one or more matching names. (However, shells don’t replace quoted wildcards. For example, in the command find*foo/.-name”bar*”, the shell replaces *foo/. with a list of all subdirectory names ending with foo, like myfoo/., hisfoo/., and so on. But the shell passes bar* to find literally, so that find can interpret that argument in each of the subdirectories it visits.)

If an unquoted wildcarded argument doesn’t match any pathnames, what happens depends on the shell you’re using and its settings. By default, bash simply passes the unexpanded wildcard to the command you’re running and lets the command complain if it wants to. Most shells can also be set to print an error and not run the command; that’s the default for tcsh.

Shell Quoting

It’s common knowledge that quoting special characters — that is, preceding a special character with a backslash (like afile\*) or surrounding its argument with quotes (like “afile*” or ‘afile*’) — disables those characters’ special meanings. The shell strips off the quoting character (s) and passes the special character, as-is, as part of the argument. Hence, cp afile*somedir copies all filenames starting with afile into somedir, and cp afile\*somedir copies the file named afile* to somedir.

A pair of single quotes () is “stronger” than a pair of double quotes (). Within double quotes, a few special characters keep their meanings, including $ for variables (like cp”$var”somedir), and backquotes, as in message=”The files are: `ls`.”. Knowing this helps you decide what kind of quotes to use.

Comments on "Wizard Boot Camp, Part One: Linux, Shells & Commands"


None of this was very wizardly…james


Very interesting – I’ll be with you the rest of the way – thanks


good starting point, i agree with Oden…. where’s the wizard part? But I guess everyone must know the basics first. I think I know where you’re going with this though, I will also be along for the ride.


OK, so Part I is not very wizardly. Is patience impossible?


“This month, let’s start with a quick overview of Linux and shell concepts that you should know, then dig into some shell features that might be new to you.”

real wizards don’t fault the master for following his posted lesson plan…


I learned some new things, thank you.


Thanks for this lesson! very helpful.


Indeed Jerry followed his lesson plan. I’m sure that this is laying the appropriate groundwork (as evidenced by some of the preceding comments) such that the masses can follow his future columns. Given that he’s educated folks for a while, I’m sure that there is a method to his madness! I’ll be following along. Thanks Jerry!


I learned a lot from this and gives a good overview. looking forward to the next Part of this “Boot Camp”.


This was a very eloquent and concise walk through of “what I already knew”. I’m looking forward to the next installments. I really think you can make a wizard out of me :-)


we know that good programmer is one who is strong in basics. Even though the above lesson is not in brief they give some important key points on which we can think or explore more about it…..


I need to brush up my Shell understanding. This Boot Camp Wizard has arrived in time for me.


Good stuff. Thanks for posting. I’m looking forward to the next installment.


Its a good refresher article.


I find it very basic but promising, the only thing I would object is the following paragraph under “Problematic Ways that Shells Gets Commands”

This is a big “gotcha” in redirecting a shell’s standard input. You may be able to work around this with the shell operator m>&n (which we’ll cover in November), but it’s likely to be painful at best.

I would rephrase as when we play with redirection

But that is just me….


grrrr… I didn’t reverse the redirection operator…sorry …


Doesn’t this say “PART ONE” and “basics” Give the Guy a chance. ~Reina~


From one who is “self educating”, thanks. This article is clearer than many of the books I got at Amazon/Borders/Barnes& noble…tem


yeah, gotta admit, as temartim01 said, it is indeed clearer than most books/tutorials/wiki’s i have seen.
thanx jerry

P.S.: i know it is a little too late, but i wanted to start the bootcamp from the beginning.


I’ve been scripting all day and read this article for a break. I had a couple of those “Oh, yeah. I forgot about that…” moments. Just because it is basic doesn’t mean it shouldn’t be repeated once in awhile.


hea, hea, am i a peon? guess so, gotta go and for those of us who play in multiple fields, I wish this series was monthly, picked up from 5th series and overall I’d say excellent, not too short and unintelligible and not too long and confusing…


Good review of basic stuff, thanks! ;-) bb


Thank you :)

I was recommended this blog by my cousin. I’m not sure whether this post is written by him as no one else know such detailed about my trouble. You’re wonderful! Thanks!

I was curious if you ever thought of changing the layout of your blog? Its very well written; I love what youve got to say. But maybe you could a little more in the way of content so people could connect with it better. Youve got an awful lot of text for only having 1 or two pictures. Maybe you could space it out better?

This is one awesome blog article.Thanks Again. Awesome.

wtiZcu If so, Alcuin as origins may lie in the fact that the Jags are

That will be the finish of this report. Right here you?ll obtain some internet sites that we believe you?ll value, just click the hyperlinks.

Just beneath, are many absolutely not associated websites to ours, nonetheless, they may be surely worth going over.

“Great, thanks for sharing this blog post.Much thanks again. Fantastic.”

Appreciate you sharing, great post.Really looking forward to read more. Great.

I like the valuable information you provide in your articles.
I’ll bookmark your blog and check again here frequently.
I am quite sure I’ll learn plenty of new stuff right here!
Best of luck for the next!

This is very interesting, You are a very skilled blogger.
I have joined your rss feed and look forward to seeking more of your great post.
Also, I’ve shared your web site in my social networks!

my blog post: MaliaSGehrke

cjuuQr gwgphmanupnb, [url=http://jkmdzkhywuax.com/]jkmdzkhywuax[/url], [link=http://ryubjeqaqqbl.com/]ryubjeqaqqbl[/link], http://szczdeeoctxs.com/

My relatives every time point out that I am just killing my time here at net,
except I know I am getting familiarity daily by reading such pleasant articles.

Also visit my weblog … IselaMFanner

Leave a Reply