How to Use Expect Scripts to Automate Tasks

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. Expect is also useful for testing these same applications. And by adding Tk, you can also wrap interactive applications in X11 GUIs.

Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily.

To install expect:

$ sudo yum install expect

or

$ sudo apt-get install expect

See also:

autoexpect - generate an Expect script from watching a session

Example Uses Of Expect

The Expect Interpreter

expect ?-d? -f scriptfile

The Expect interpreter is called expect. It's a plain Tcl interpreter with the Expect language extensions added.

A Typical Expect Script

A typical Expect script controls another process via a dialogue. Expect alternates sending some text to the process and then reacting to the text produced by the process in response. Since the responses will vary, Expect uses pattern matching to distinguish between possible responses. Commonly, after an initial dialogue under Expect's control, the rest of the interaction is turned over to the user. So the typical Expect script looks something like:

1. Spawn a process
2. Repeatedly:
   1. Send a message to the process
   2. Expect a set of possible responses, and react
3. Turn the interaction over to the user

The spawn Command

spawn program ?args?

The spawn command is similar to the Tcl exec command, in that it fires up another process under Tcl's control. However,spawn's process is running asynchronously, so spawn returns control to Tcl immediately. Also, special Expect commands are used to communicate with the spawned process (rather than file descriptors as withexec). spawn returns the process ID of the spawned process.

The expect Command

expect ?-opts? pat1 body1 ... ?-opts? patn ?bodyn?

The expect command is the heart of the Expect language. expect is a control structure that selects abody-i to execute based on matching the patterns pat-i against the output of the spawned process. For example:

expect {*Account:*}

The above command has a pattern but no body; it will simply wait for the spawned process to output the stringAccount:. Note that (by default) globbing is used for the pattern matching, so the leading* causes any text coming before Account: to be ignored. The trailing* likewise causes any characters after Account: (like blanks) to be ignored. In writing Expect patterns its generally important not to be too specific, or your patterns will fail to match.

A more complex example:

expect "Wait for response" {} "SERVICE UNAVAILABLE" {error}

In the above, we are expecting the typical response to be the string "Wait for response", but we want to generate an error in the case of the "SERVICE UNAVAILABLE" message.

Here's a part of an Expect script that handles typical initial responses from telnet:

expect {
    {Connected*Escape character is*.} {}
    "unknown*" {
 error $expect_out(buffer)
    }
    "unreachable*" {
 error $expect_out(buffer)
    }
    eof {
        error "Connection closed."
    }
    timeout {
        error "Connection timed out."
    }
}

Note that the pattern/body pairs can be grouped together in braces to allow them to span multiple lines (much the way the Tclswitch command works).

The first pattern is the one we're hoping for; if we see it, we execute the null body and proceed to the next command in the script.

The next two patterns match two commontelnet error messages, "unknown host" and "host unreachable"; the bodies turn them into Tcl errors.

Thetimeout pattern is actually a special expect pattern that matches when the command times out (the global variabletimeout specifies the timeout in seconds). The eof pattern is similar: it matches iff Expect gets end of file from the spawned process.

This segment of code illustrates another feature: the array variable expect_out has several fields that contain information about the matching process.expect_out(buffer) contains the matching text (and any previously unmatched text).

Each pattern can be preceded by a number of options; the -re option specifies that regular expression pattern matching be used for that pattern instead of globbing.

The timeout pattern can be used in other ways. This short, complete Expect script runs an arbitrary program with a timeout:

#!/local/bin/expect -f
# run a program for a given amount of time
# i.e. time 20 long_running_program
set timeout [lindex $argv 0]
eval spawn [lrange $argv 1 end]
expect

The log_user Command

log_user 0
log_user 1

Normally, Expect echoes the input and output of the spawned process so the user can see it. You can turn this echoing on and off with thelog_user command, which takes a Boolean argument. With 0, it turns echoing off.

A Complete Expect Script

This pointless Expect script uses telnet to connect to the daytime port of the host named on the command line and printsjust the time in the form 99:99:99.

#!/local/bin/expect -f
log_user 0
if {[llength $argv] != 1} {
    puts stderr "Usage: daytime host"
    exit 1
}
spawn telnet [lindex $argv 0] daytime
expect "scape character*\en"
expect "*\en"
regexp {([0-9][0-9]:[0-9][0-9]:[0-9][0-9])} $expect_out(buffer) _ time
puts $time

The send Command

send string

The send command sends string to the spawned process. The process receives it as if the user had typed it on the terminal.send has fancy options to send the string slowly (to avoid overflowing buffers on a slow remote machine) and to simulate a human typing.

The interact Command

interact string1 body1 ... stringn ?bodyn? 

The interact command returns control of the spawned process to the user. Wheninteract is executed, input to the process comes from the user at the terminal and output from the process goes to the user at the terminal (regardless of the setting oflog_user). However, Expect watches the user's keystrokes during the interaction and can arrange to execute Tcl commands when the user types certain strings.

Each string of interest is paired with a body. A null body gives the user access to the Expect interpreter. Here is an example:

set CTRLZ \e032
interact {
  -reset $CTRLZ {exec kill -STOP 0}
  \e001    {puts "you typed a control-A\en";
    send "\e001"
   }
  $       {puts "The date is [exec date]."}
  \e003    exit
  foo     {puts "bar"}
  ~~
}

The -reset option preceding the control-Z pattern tells expect to reset the terminal modes; this allows thekill command to temporarily suspend the Expect interpreter. When the user resumes Expect, the terminal modes will be reset again.

Note that the \e001 pattern not only prints a message to the user but also passes the control-A on to the spawned process.

The final ~~ pattern, has no body, so if the user types ~~ they will be given access to the Expect interpreter's command loop.

Expect example uses

• as system administrators we may have need for automating interactive activities
  • testing remote access works (telnet, ftp)
  • down loading ftp files
  • changing password
  • running security checks

Expect and examples

• automatically creating passwords
  • can be done via C
    • takes a lot of time to write such programs
    • will it work with NIS, shadow passwords or Kerboros?
• with expect we run the user program passwd and send keyboard input

---

Password Trial

• advice first run the program that you want to connect to expect by hand
  • note the output and build expect around it
  • so let us change a password for bob
• passwd bob
  Enter new UNIX password:
  Retype new UNIX password:
  passwd: password updated successfully

  • passwords were entered, but obviously the passwd program did not echo back our input! 

---

Trial results

• we note that the program prompted us for a password twice
  • both times it ended its sentence with password:
  • as we are lazy we can wait until we see password: and ignore anything that was before
    • why is this good practice?
    • what must we watch out for?
• #!/usr/bin/expect
  
  
  spawn passwd [lindex $argv 0]
  set password [lindex $argv 1]
  expect "password:"
  send "$password\r"
  expect "password:"
  send "$password\r"
  expect eof

---

Testing the example

• ./exp1.exp bob 123
  spawn passwd bob
  Enter new UNIX password:
  Retype new UNIX password:
  passwd: password updated successfully
• here we change user bob password to 123 

---

Script explanation

• #!/usr/bin/expect
  
  
  spawn passwd [lindex $argv 0]
  set password [lindex $argv 1]
  expect "password:"
  send "$password\r"
  expect "password:"
  send "$password\r"
  expect eof
• #!/usr/bin/expect script is interpreted via expect located in directory /usr/bin
• spawn passwd [lindex $argv 0]
  • run the program passwd bob and connect expect to this program
  • note that [lindex $argv 0] resolves to bob
    • actually argument one (hmm..) 

---

Script explanation

• set password [lindex $argv 1]
  • defines a variable password and sets it to 123
• expect "password:"
  • waits for the program passwd to issue password: before continuing
• send "$password\r"
  • sends the users password to passwd followed by a carriage return
• note the script repeats the last two commands, why?
• finally expect eof wait for passwd to finish

---

Anchoring

• you might want to match text at the beginning or end of a line, this is via
  • ^ for the beginning of a line
  • $ for the end of a line
• also note that * means any number of characters

---

Pattern action pairs

• #!/usr/bin/expect
  
  set timeout 15
  
  
  expect "hi"  { send "You said hi\n" } \
       "hello"  { send "Hello to you\n" } \
       "bye"    { send "Goodbye\n" } \
       timeout  { send "I’m fed up\nbye\n" }
• if we run the script as shown below and type nothing we get:
• I’m fed up
  bye
• note that different actions can be associated with different input
  • note also that the default timeout time is set at 10 seconds to disable the timeout facility
  • set timeout -1 

---

Autoftp and expect

• so far we have built the front end to autoftp
  • scans the input file for URLs
  • handles arguments
• we will use expect to control ftp, we will build this up this utility
• firstly we will ftp manually

---

Ftp session

• fred@merlin:$ ftp guenevere
  Connected to guenevere.
  220 guenevere FTP server (Version wu-2.6.0(1))
  Name (guenevere:fred): anonymous
  331 Guest login ok, send your complete e-mail
  331 address as password.
  Password:nobody@nowhere.com
  
  
  230-Welcome, archive user anonymous@merlin !
  230-
  230-The local time is: Mon Feb 12 15:18:14 2001
  230-
  230 Guest login ok, access restrictions apply.
  Remote system type is UNIX.
  Using binary mode to transfer files.
• ftp> dir
  200 PORT command successful.
  150 Opening ASCII mode data connection for /bin/ls.
  total 20
  d--x--x--x    2 0 0 4096 Aug 10  2000 bin
  d--x--x--x    2 0 0 4096 Aug 10  2000 etc
  d--x--x--x    2 0 0 4096 Aug 10  2000 lib
  dr-xr-xr-x    4 0 0 4096 Sep 28 22:17 pub
  -rw-r--r--    1 0 0  346 Aug 10  2000 welcome.msg
  226 Transfer complete.
  ftp> quit
  221-You have transferred 0 bytes in 0 files.
  221-Total traffic for this session was 1182
  221-bytes in 1 transfers.
  221-Thank you for using the FTP service on
  221-guenevere. Goodbye.

---

Simple semi automated ftp script

• this script will log us into a site and then interact with the user
• #!/usr/bin/expect
  
  
  set site [lindex $argv 0]
  spawn ftp $site
  expect "Name"
  send "anonymous\r"
  expect "Password:"
  send "nobody@nowhere.com\r"
  interact
• note the interact statement
  • connects the keyboard to the ftp program
• most ftp servers do not check for a valid email address! 

---

Expect control constructs

• expect uses extends the Tcl language
• expect provides: if then else, while, set, interact, expect, for, switch, incr, return, proc
• if {$count < 0} {
     set total 1
  }
• if {$count < 5} {
     puts "count is less than five"
  } else {
     puts "count is not less than five"
  }
• you must place the statement begin brace on the same line as the if or else
  • if in doubt follow the templates in these notes 

---

More complex if

• if {$count < 0} {
     puts "count is less than zero"
  } elseif {$count > 0 {
     puts "count is greater than zero"
  } else {
     puts "count is equal to zero"
  }

---

While statement

• #!/usr/bin/expect
  
  
  set count 10
  while {$count > 0} {
     puts "the value of count is $count"
     set count [expr $count-1]
  }
• note the effect of the braces in {$count > 0}
  • they defer evaluation of $count
  • if you remove the braces then $count > 0 is internally replaced via: 10 > 0 

---

for command

• has the syntax
  for start expression next {
  }
• #!/usr/bin/expect
  
  
  for {set count 10} {$count > 0} {incr count -1} {
     puts "the value of count is $count"
  }

---

Expressions

• 0 is false, 1 is true
• boolean operators || (or), && (and), ! (not)
• comparison operators <=, ==, != etc
• the brackets [ ] give this expression a higher precedence
• so in the example
• set count [expr $count-1]
• the [expr $count-1] is evaluated before the set! 

---

proc and return

• #!/usr/bin/expect
  
  proc mycompare {a b} {
      if {$a < $b} {
          puts "$a is less than $b"
          return -1
      } elseif {$a > $b} {
          puts "$a is greater than $b"
          return 1
      } else {
          puts "$a is equal to $b"
          return 0
      }
  }
  
  
  set value [mycompare 1 4]
  puts "comparison returned $value"
• ./exp6.exp
  1 is less than 4
  comparison returned -1

---

Autoftp Tutorial

• #!/usr/bin/expect
  # this program is called exp7.exp
  proc connect {} {
     expect {
         "Name*:" {
             send "anonymous\r"
             expect {
                 "Password:" {
                     send "nobody@nowhere.com\r"
                     expect "login ok*ftp>"
                     return 0
                 }
             }
         }
     }
     # timed out
     return 1
  }
• set site [lindex $argv 0]
  spawn ftp $site
  while {[connect]} {
      send "quit\r"
      expect eof
      spawn ftp $site
  }
  send "binary\r"
  send "cd  [lindex $argv 1]\r"
  send "get [lindex $argv 2]\r"
  send "quit\r"
  expect eof