How to Use Expect Scripts to Automate Tasks

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. Expect is also useful for testing these same applications. And by adding Tk, you can also wrap interactive applications in X11 GUIs.

Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily.

To install expect:

$ sudo yum install expect

$ sudo apt-get install expect

See also:

autoexpect - generate an Expect script from watching a session

Example Uses Of Expect

The Expect Interpreter

expect ?-d? -f scriptfile

The Expect interpreter is called expect. It's a plain Tcl interpreter with the Expect language extensions added.

A Typical Expect Script

A typical Expect script controls another process via a dialogue. Expect alternates sending some text to the process and then reacting to the text produced by the process in response. Since the responses will vary, Expect uses pattern matching to distinguish between possible responses. Commonly, after an initial dialogue under Expect's control, the rest of the interaction is turned over to the user. So the typical Expect script looks something like:

Spawn a process
Repeatedly:
1. Send a message to the process
2. Expect a set of possible responses, and react
Turn the interaction over to the user

The `spawn` Command

spawn program ?args?

The spawn command is similar to the Tcl exec command, in that it fires up another process under Tcl's control. However,spawn's process is running asynchronously, so spawn returns control to Tcl immediately. Also, special Expect commands are used to communicate with the spawned process (rather than file descriptors as withexec). spawn returns the process ID of the spawned process.

The `expect` Command

expect ?-opts? pat1 body1 ... ?-opts? patn ?bodyn?

The expect command is the heart of the Expect language. expect is a control structure that selects abody-i to execute based on matching the patterns pat-i against the output of the spawned process. For example:

expect {*Account:*}

The above command has a pattern but no body; it will simply wait for the spawned process to output the stringAccount:. Note that (by default) globbing is used for the pattern matching, so the leading* causes any text coming before Account: to be ignored. The trailing* likewise causes any characters after Account: (like blanks) to be ignored. In writing Expect patterns its generally important not to be too specific, or your patterns will fail to match.

A more complex example:

expect "Wait for response" {} "SERVICE UNAVAILABLE" {error}

In the above, we are expecting the typical response to be the string "Wait for response", but we want to generate an error in the case of the "SERVICE UNAVAILABLE" message.

Here's a part of an Expect script that handles typical initial responses from telnet:

expect {
    {Connected*Escape character is*.} {}
    "unknown*" {
 error $expect_out(buffer)
    }
    "unreachable*" {
 error $expect_out(buffer)
    }
    eof {
        error "Connection closed."
    }
    timeout {
        error "Connection timed out."
    }
}

Note that the pattern/body pairs can be grouped together in braces to allow them to span multiple lines (much the way the Tclswitch command works).

The first pattern is the one we're hoping for; if we see it, we execute the null body and proceed to the next command in the script.

The next two patterns match two commontelnet error messages, "unknown host" and "host unreachable"; the bodies turn them into Tcl errors.

Thetimeout pattern is actually a special expect pattern that matches when the command times out (the global variabletimeout specifies the timeout in seconds). The eof pattern is similar: it matches iff Expect gets end of file from the spawned process.

This segment of code illustrates another feature: the array variable expect_out has several fields that contain information about the matching process.expect_out(buffer) contains the matching text (and any previously unmatched text).

Each pattern can be preceded by a number of options; the -re option specifies that regular expression pattern matching be used for that pattern instead of globbing.

The timeout pattern can be used in other ways. This short, complete Expect script runs an arbitrary program with a timeout:

#!/local/bin/expect -f
# run a program for a given amount of time
# i.e. time 20 long_running_program
set timeout [lindex $argv 0]
eval spawn [lrange $argv 1 end]
expect

The `log_user` Command

log_user 0
log_user 1

Normally, Expect echoes the input and output of the spawned process so the user can see it. You can turn this echoing on and off with thelog_user command, which takes a Boolean argument. With 0, it turns echoing off.

A Complete Expect Script

This pointless Expect script uses telnet to connect to the daytime port of the host named on the command line and printsjust the time in the form 99:99:99.

#!/local/bin/expect -f
log_user 0
if {[llength $argv] != 1} {
    puts stderr "Usage: daytime host"
    exit 1
}
spawn telnet [lindex $argv 0] daytime
expect "scape character*\en"
expect "*\en"
regexp {([0-9][0-9]:[0-9][0-9]:[0-9][0-9])} $expect_out(buffer) _ time
puts $time

The `send` Command

send string

The send command sends string to the spawned process. The process receives it as if the user had typed it on the terminal.send has fancy options to send the string slowly (to avoid overflowing buffers on a slow remote machine) and to simulate a human typing.

The `interact` Command

interact string1 body1 ... stringn ?bodyn?

The interact command returns control of the spawned process to the user. Wheninteract is executed, input to the process comes from the user at the terminal and output from the process goes to the user at the terminal (regardless of the setting oflog_user). However, Expect watches the user's keystrokes during the interaction and can arrange to execute Tcl commands when the user types certain strings.

Each string of interest is paired with a body. A null body gives the user access to the Expect interpreter. Here is an example:

set CTRLZ \e032
interact {
  -reset $CTRLZ {exec kill -STOP 0}
  \e001    {puts "you typed a control-A\en";
    send "\e001"
   }
  $       {puts "The date is [exec date]."}
  \e003    exit
  foo     {puts "bar"}
  ~~
}

The -reset option preceding the control-Z pattern tells expect to reset the terminal modes; this allows thekill command to temporarily suspend the Expect interpreter. When the user resumes Expect, the terminal modes will be reset again.

Note that the \e001 pattern not only prints a message to the user but also passes the control-A on to the spawned process.

The final ~~ pattern, has no body, so if the user types ~~ they will be given access to the Expect interpreter's command loop.

Expect example uses

as system administrators we may have need for automating interactive activities
- testing remote access works (telnet, ftp)
- down loading ftp files
- changing password
- running security checks

Expect and examples

automatically creating passwords
- can be done via C
  - takes a lot of time to write such programs
  - will it work with NIS, shadow passwords or Kerboros?
with expect we run the user program passwd and send keyboard input

Password Trial

advice first run the program that you want to connect to expect by hand
- note the output and build expect around it
- so let us change a password for bob

passwd bob
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

passwords were entered, but obviously the passwd program did not echo back our input!

Trial results

we note that the program prompted us for a password twice
- both times it ended its sentence with password:
- as we are lazy we can wait until we see password: and ignore anything that was before
  - why is this good practice?
  - what must we watch out for?

#!/usr/bin/expect


spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password\r"
expect "password:"
send "$password\r"
expect eof

Testing the example

./exp1.exp bob 123
spawn passwd bob
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

here we change user bob password to 123

Script explanation

#!/usr/bin/expect


spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password\r"
expect "password:"
send "$password\r"
expect eof

#!/usr/bin/expect script is interpreted via expect located in directory /usr/bin
spawn passwd [lindex $argv 0]
- run the program passwd bob and connect expect to this program
- note that [lindex $argv 0] resolves to bob
  - actually argument one (hmm..)

Script explanation

set password [lindex $argv 1]
- defines a variable password and sets it to 123
expect "password:"
- waits for the program passwd to issue password: before continuing
send "$password\r"
- sends the users password to passwd followed by a carriage return
note the script repeats the last two commands, why?
finally expect eof wait for passwd to finish

Anchoring

you might want to match text at the beginning or end of a line, this is via
- ^ for the beginning of a line
- $ for the end of a line
also note that * means any number of characters

Pattern action pairs

#!/usr/bin/expect

set timeout 15


expect "hi"  { send "You said hi\n" } \
     "hello"  { send "Hello to you\n" } \
     "bye"    { send "Goodbye\n" } \
     timeout  { send "I’m fed up\nbye\n" }

if we run the script as shown below and type nothing we get:
```
I’m fed up
bye
```
note that different actions can be associated with different input
- note also that the default timeout time is set at 10 seconds to disable the timeout facility
- set timeout -1

Autoftp and expect

so far we have built the front end to autoftp
- scans the input file for URLs
- handles arguments
we will use expect to control ftp, we will build this up this utility
firstly we will ftp manually

Ftp session

fred@merlin:$ ftp guenevere
Connected to guenevere.
220 guenevere FTP server (Version wu-2.6.0(1))
Name (guenevere:fred): anonymous
331 Guest login ok, send your complete e-mail
331 address as password.
Password:nobody@nowhere.com


230-Welcome, archive user anonymous@merlin !
230-
230-The local time is: Mon Feb 12 15:18:14 2001
230-
230 Guest login ok, access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.

ftp> dir
200 PORT command successful.
150 Opening ASCII mode data connection for /bin/ls.
total 20
d--x--x--x    2 0 0 4096 Aug 10  2000 bin
d--x--x--x    2 0 0 4096 Aug 10  2000 etc
d--x--x--x    2 0 0 4096 Aug 10  2000 lib
dr-xr-xr-x    4 0 0 4096 Sep 28 22:17 pub
-rw-r--r--    1 0 0  346 Aug 10  2000 welcome.msg
226 Transfer complete.
ftp> quit
221-You have transferred 0 bytes in 0 files.
221-Total traffic for this session was 1182
221-bytes in 1 transfers.
221-Thank you for using the FTP service on
221-guenevere. Goodbye.

Simple semi automated ftp script

this script will log us into a site and then interact with the user

#!/usr/bin/expect


set site [lindex $argv 0]
spawn ftp $site
expect "Name"
send "anonymous\r"
expect "Password:"
send "nobody@nowhere.com\r"
interact

note the interact statement
- connects the keyboard to the ftp program
most ftp servers do not check for a valid email address!

Expect control constructs

expect uses extends the Tcl language
expect provides: if then else, while, set, interact, expect, for, switch, incr, return, proc
```
if {$count < 0} {
   set total 1
}
```

if {$count < 5} {
   puts "count is less than five"
} else {
   puts "count is not less than five"
}

you must place the statement begin brace on the same line as the if or else
- if in doubt follow the templates in these notes

More complex if

if {$count < 0} {
   puts "count is less than zero"
} elseif {$count > 0 {
   puts "count is greater than zero"
} else {
   puts "count is equal to zero"
}

While statement

#!/usr/bin/expect


set count 10
while {$count > 0} {
   puts "the value of count is $count"
   set count [expr $count-1]
}

note the effect of the braces in {$count > 0}
- they defer evaluation of $count
- if you remove the braces then $count > 0 is internally replaced via: 10 > 0

for command

has the syntax
for start expression next {
}

#!/usr/bin/expect


for {set count 10} {$count > 0} {incr count -1} {
   puts "the value of count is $count"
}

Expressions

0 is false, 1 is true
boolean operators || (or), && (and), ! (not)
comparison operators <=, ==, != etc
the brackets [ ] give this expression a higher precedence
so in the example
```
set count [expr $count-1]
```
the [expr $count-1] is evaluated before the set!

proc and return

#!/usr/bin/expect

proc mycompare {a b} {
    if {$a < $b} {
        puts "$a is less than $b"
        return -1
    } elseif {$a > $b} {
        puts "$a is greater than $b"
        return 1
    } else {
        puts "$a is equal to $b"
        return 0
    }
}


set value [mycompare 1 4]
puts "comparison returned $value"

./exp6.exp
1 is less than 4
comparison returned -1

Autoftp Tutorial

#!/usr/bin/expect
# this program is called exp7.exp
proc connect {} {
   expect {
       "Name*:" {
           send "anonymous\r"
           expect {
               "Password:" {
                   send "nobody@nowhere.com\r"
                   expect "login ok*ftp>"
                   return 0
               }
           }
       }
   }
   # timed out
   return 1
}

set site [lindex $argv 0]
spawn ftp $site
while {[connect]} {
    send "quit\r"
    expect eof
    spawn ftp $site
}
send "binary\r"
send "cd  [lindex $argv 1]\r"
send "get [lindex $argv 2]\r"
send "quit\r"
expect eof

Search This Blog

Remote Technology Jobs

Digital Marketing