How To Expect

In last month's column, I used a small Expect script to communicate with a highly accurate clock that was attached to my system's serial port. In this month's column, we will take a more extended look at this very useful tool.

In last month’s column, I used a small Expect script to communicate with a highly accurate clock that was attached to my system’s serial port. In this month’s column, we will take a more extended look at this very useful tool.

Expect is a freely available software facility/programming language that allows you to automate pretty much any interactive task. Don Libes, who began writing Expect in 1990, puts it more tersely: “Expect [is a] software suite for automating interactive tools.” It allows a system administrator to create scripts that provide input to commands and programs that would otherwise “expect” — or demand — their input from the “terminal” (traditionally /dev/tty). Such input would otherwise need to be supplied by a human user or system administrator. Expect can send the proper input at the appropriate time to such programs without any user intervention; it can even make decisions about how to respond to successive prompts based upon the previous responses and/or whatever other factors and logic you have chosen to supply to the script.

Expect is an understated package. When I first heard about it, I thought that it might be somewhat useful in certain circumstances. However, as time has gone on, I have continued to find Expect ever more useful and have grown to depend on it as an essential item in my administrative tool suite.

Expect is built on top of the Tcl programming language, which makes having Tcl installed on your system a prerequisite for building and running Expect. In addition, it can also use the graphical component of Tcl — Tk — that enables you to create graphical tools via Expect scripts.

How Expect Works

At its simplest level, Expect works like a generalized chat script facility. Chat scripts first came into wide usage via the UUCP facility in order to specify the login conversation that needed to take place whenever two computers wanted to create a connection.

Chat scripts consist of a series of expect-send pairs: expect to see something in the output you get (often a prompt), and when you do, send some specific text. For example, the following chat script says to look for the characters Login:, send somebody when it is found, look for Password:, and finally send sillyme:

Login: somebody Password: sillyme

This chat script defines a conversation between two computers and sends the appropriate username and password information when the remote system asks for them.

Expect’s simplest operating mode works in essentially the same way. We’ll look at a script that responds to the chsh command in a moment. First, let’s review the interactive format of that command. Figure One changes the login shell for user chavez.

Figure One: Using the chsh Command

# chsh chavez
Changing the login shell for chavez
Enter the new value, or press return for the default
Login Shell [/bin/bash]: /bin/tcsh

The command prints a couple of lines of output and then prompts for the desired shell for the user specified on the command line. We must enter the desired shell at that prompt, and entering a carriage return will select the default shell that is indicated in square brackets.

Here is an Expect script that can automate the previous command:

# Change a login shell to tcsh

set user [lindex $argv 0]
spawn chsh $user
expect “]:”
send “/bin/tcsh\n”
expect eof

This simple script illustrates many features of an Expect program. As with most other kinds of scripts, the initial line specifies the executable image that should be used to run the script (in this case, #!/usr/bin/ expect).

Within the script proper, the first line sets the variable user to the script’s first argument: the first element of the array $argv (note that element numbering begins at 0).

The second command uses Expect’s spawn command to initiate a conversation between the script and a command — chsh in this case (the actual command is run via a spawned subprocess, the source of this Expect command’s name).

The expect and send commands that follow define and control the resulting conversation. The script first looks for the string “]:” (right bracket-colon) in the output from chsh, via the first expect command. In order for the command to find a match, the specified characters must simply be the final ones that it sees. By default, the strings specified to the expect command do not require an exact and complete match; on the contrary, expect waits for whatever it is looking for, ignoring or optionally processing whatever precedes it as directed (as we shall see).

When the script finds those characters it then will send /bin/tcsh plus a carriage return (\n) to the command. Finally, the script waits for the command (spawned process) to exit by looking for an eof (end-of-file) via the second expect command. Once an eof has been received, the script also exits.

Deciding How to Reply

You will often want to reply to a command in different ways, depending on what has happened so far in the process. We can illustrate Expect’s capabilities for making complex response decisions by a simple modification of the preceding script.

Here is a more complex form of the preceding expect-send sequence:

expect -re “\\\[(.*)]:”
if {$expect_out(1,string)!=”/bin/tcsh”} {
send “/bin/tcsh” }
send “\n”
expect eof

In this case, the first expect command now uses the -re option. This option indicates that the specified string is a regular expression (and not a literal character string). In this case, we are looking for a literal left bracket character (which must be triple escaped — hence the three backslashes — since it is a special character both within regular expressions and to Expect), followed by zero or more characters, followed by right bracket-colon (as before). We enclose the “.*” regular expression characters in parentheses so that we can retrieve whatever matches that part later.

When we find a match, we check the substring enclosed in square brackets and see if it is /bin/tcsh. If not, we send the full path to the tcsh shell, followed by a carriage return in the second send command; otherwise, we just send the carriage return. This is a trivial example of alternate response options but it does illustrate the potential that Expect offers in this area.

Within a regular expression, you can enclose several components in parentheses and retrieve them via the expect_out array. Components are numbered from left to right within the expression, beginning with 1 (element 0 holds the entire matched output). Parentheses may also be nested, in which case numbering moves from innermost to outermost within each item.

Using Timeouts

Our next Expect example illustrates a prompt function with a built-in timeout. This script will prompt a user for input, and if he or she does not respond within a given period of time, it times out and returns a default response. The script accepts three arguments: the prompt string, default response, and timeout period in seconds. Figure Two shows the first part of the script. This first section of the script assigns the script’s arguments to internal variables.

Figure Two: Prompt with timeout – Part I

# Prompt function with timeout and default.
set prompt [lindex $argv 0]
set def [lindex $argv 1]
set response $def
set tout [lindex $argv 2]

Figure Three shows the remainder of the script. As you can see the send_tty command displays the prompt string on the terminal screen, appending a final colon and space. The set timeout command sets the timeout period for any subsequent Expect commands to the value of the tout variable (an argument of -1 may be used to disable all timeouts).

Figure Three: Prompt with timeout – Part II

send_tty “$prompt: ”
set timeout $tout
expect “\n” {
set raw $expect_out(buffer)
# remove final carriage return
set response [string trimright "$raw" "\n"]
if {“$response” == “} {set response $def}
send “$response\n”
# Prompt function with timeout and default.
set prompt [lindex $argv 0]
set def [lindex $argv 1]
set response $def
set tout [lindex $argv 2]

This expect command is looking for a carriage return character. If one is encountered before the timeout period expires, then the commands that are within the curly braces are executed. In this case, the set command will assign to the variable raw whatever the user types in response to the prompt. The subsequent command removes the final carriage return (if any) from the user’s input and assigns the resulting character string to the variable response.

Next, the if command sets the value of response to the default answer if that value is null (which will be the case if no user input was given, before the timeout period expired, or if the user entered only a carriage return). The final set command sends the final response plus a carriage return to the script’s standard output.

An interesting thing to note about this script is that is does not use any spawn command; the Expect conversation occurs with whatever process calls this script.

If the Expect script is named prompt, then it could be used in this way (from a C-style shell):

% set a=’prompt “Enter an answer” silence 10′
Enter an answer: test

% echo Answer was “$a”
Answer was test

This prompt uses a timeout period of 10 seconds. If the prompt had timed out or the user had entered only a carriage return, the output of the echo command would have been this instead:

Answer was “silence”

A More Complex Conversation

In this section, we will look at a significantly more complex Expect script than the ones we have considered so far. We will see more complex control structures as well as more complicated conversations.

The example we will consider is a script that allows you to send a write command to an arbitrary list of users, taking the message to be sent either from a prepared file or from the keyboard interactively.

Figure Four shows the beginning of the script, which ensures that there are enough arguments to the script by checking the value of the built-in variable argc.

Figure Four: Sending a write Command – Part I

# Write to multiple users from a prepared file
# or a message input interactively

if {$argc<2} {
send_user “usage: $argv0 file user1 user2 …\n”

The send_user command is used to display output to the standard output of the calling process (usually the user’s shell).

The next section of the script (Figure Five) processes the initial argument, which must be either a filename for the message file or an “i” indicating interactive mode.

Figure Five: Sending a write Command – Part II

set nofile 0
# get filename via the Tcl lindex function
set file [lindex $argv 0]
if {$file==”i”} {
set nofile 1
} else {
# make sure message file exists
if {[file isfile $file]!=1} {
send_user “$argv0: file $file not found.\n”
exit }}

The variable file is set to the value of the first argument to the script in the third line via a Tcl function call to lindex, which returns a specified element from a list/array. Enclosing the function call in square brackets causes its return value to be used as an argument with the set command.

If the script’s first argument is a lowercase i, then the variable nofile is set to 1 (after having been previously initialized to 0). If the argument is anything else, then the script makes sure that the specified file exists using the Tcl file function.

This code section also illustrates the syntax of the if command. The command takes the condition to be tested as its argument (enclosed in braces) and executes the curly braces-enclosed command block that follows if the condition is true. In this case, an else clause is also included to specify another command block to be executed when the condition is false.

The next portion of the script starts the write processes using the Expect spawn command:

set procs {}
# start write processes
for {set i 1} {$i<$argc}
{incr i} {
spawn -noecho write
[lindex $argv $i]
lappend procs $spawn_id

This part of the script uses a for loop, which takes the initial value of the loop variable, the test condition for ending the loop, and the command to be performed after each loop iteration as its arguments. The body of the loop is enclosed in curly braces (the construction is very C-like).

In our case, we use the second and later script arguments (which are taken as usual from the array $argv) to spawn a write command, using each argument as the username given to write. The lappend command constructs a list of process IDs in the variable procs using the internal variable $spawn_id, which is automatically set to the PID of each spawned process.

The next section of the script opens the message file or directs the user to type in the message, depending on the value of the nofile variable:

if {$nofile==0} {
setmesg [open "$file" "r"]
} else {
send_user “enter message,
ending with ^D:\n” }

Figure Six demonstrates how the actual message text is sent via an infinite while loop (with the loop body statements again being enclosed in curly braces).

Figure Six: Sending the Message Text With a while Loop

set timeout -1
while 1 {
if {$nofile==0} {
if {[gets $mesg chars] == -1} break
set line “$chars\n”
} else {
expect_user {
-re “\n” {}
eof break }
set line $expect_out(buffer) }

foreach spawn_id $procs {
send $line }
sleep 1}

The if statement within the while loop determines how message lines are obtained in the two cases. In the non-interactive case, the next line is read from the message file, and the while loop ends when the end of that file is reached (the break command terminates the loop).

In the interactive case, an expect_ user command accepts the next line of input from the user; when he or she types a Control-D the while loop is terminated as well (again, via break).

In both cases, the variable $line is used to hold the next message line; a carriage return is added when text is taken from a message file.

The foreach loop runs over all of the process IDs stored in the list $procs, making each of them the current communication channel in turn. The send command, which makes up the body of the foreach loop, sends the line to the current write process. The final command in the while loop is a sleep command, which is needed in non-interactive mode, so that text does not get transmitted to each process too quickly (and it is imperceptible in interactive mode). Once the while loop ends, the Expect script exits.

Added Bonuses

The Expect package also includes some example scripts that are useful not only for learning and understanding Expect, but also as tools in their own right. They are all available under /usr/ doc/packages/expect/example, and some of them are installed in /usr/bin in some Linux distributions as well. See the A Couple of Useful Expect Scripts sidebar for more information on some of these sample scripts. Between those examples and the scripts we’ve looked at here, you should have plenty of “expecting” to do between now and next month.

Books of Interest

Don Libes, Exploring Expect, O’Reilly & Associates, 1995.

John Ousterhout, Tcl and the Tk Toolkit, Addison-Wesley, 1994.

A Couple of Useful Expect Scripts

Figure One: A system administrator instructs an inexperienced user in a sample kibitz session.

autoexpect: This script will create an Expect script corresponding to the actions that you take while it is running. It functions in a similar fashion to the keyboard macro facility found in the Emacs editor and some word processing programs. This automatically generated script is often a good way to start creating a more general script of your own.

kibitz: This is a very useful tool. It allows two (or more) users to be attached to the same shell process. It can be used for technical support and training purposes (see Figure One) and also for a variety of other collaborative activities. As an example of the latter, when I wear my marketing hat, I am often asked to look over some mail messages before they are sent out. Using kibitz, the sender and I can share a shell running an editor and we can both look at and edit the text at the same time.

tkpasswd: This script provides a GUI interface for changing user passwords, including the capability of testing whether a proposed password is found in a dictionary or not. It is somewhat useful as a tool and also serves as a good starting point for learning about Expect and Tk.

Æleen Frisch is the author of Essential System Administration. She can be reached at aefrisch@lorentzian.com.

Comments are closed.