Pexpect version VERSION
a Pure Python Expect-like module

Pexpect makes Python a better tool for controlling other applications.

Pexpect is a pure Python module for spawning child applications; controlling them; and responding to expected patterns in their output. Pexpect works like Don Libes' Expect. Pexpect allows your script to spawn a child application and control it as if a human were typing commands.

Pexpect can be used for automating interactive applications such as ssh, ftp, passwd, telnet, etc. It can be used to a automate setup scripts for duplicating software package installations on different servers. It can be used for automated software testing. Pexpect is in the spirit of Don Libes' Expect, but Pexpect is pure Python. Unlike other Expect-like modules for Python, Pexpect does not require TCL or Expect nor does it require C extensions to be compiled. It should work on any platform that supports the standard Python pty module. The Pexpect interface was designed to be easy to use.

Send questions to:

---

License: MIT style

Free, open source, and all that good stuff.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Pexpect Copyright (c) 2008 Noah Spurrier
http://pexpect.sourceforge.net/

---

Download

Download the current version here from the SourceForge site. Grab the current Pexpect tarball.

Installing Pexpect

The Pexpect tarball is a standard Python Distutil distribution.

1. download pexpect-VERSION.tar.gz
2. tar zxf pexpect-VERSION.tar.gz
3. cd pexpect-VERSION
4. python setup.py install do this as root

Examples

Under the pexpect-VERSION directory you should find the examples directory. This is the best way to learn to use Pexpect. See the descriptions of Pexpect Examples.

API Documentation

pexpect This is the main module that you want.
pxssh Pexpect SSH is an extension of 'pexpect.spawn' that specializes in SSH.

the following are experimental extensions to Pexpect

fdpexpect fdpexpect extension of 'pexpect.spawn' that uses an open file descriptor.
SCREEN This represents a virtual 'screen'.
ANSI This parses ANSI/VT-100 terminal escape codes.
FSM This is a finite state machine used by ANSI.

---

Project Status

Automated pyunit tests reach over 80% code coverage on pexpect.py. I regularly test on Linux and BSD platforms. I try to test on Solaris and Irix.

---

Requirements for use of Pexpect

Python

Pexpect was written and tested with Python 2.4. It should work on earlier versions that have the pty module. I sometimes even manually test it with Python 1.5.2, but I can't easily run the PyUnit test framework against Python 1.5.2, so I have less confidence in Pexpect on Python 1.5.2.

pty module

Any POSIX system (UNIX) with a working pty module should be able to run Pexpect. The pty module is part of the Standard Python Library, so if you are running on a POSIX system you should have it. The pty module does not run the same on all platforms. It should be solid on Linux and BSD systems. I have taken effort to try to smooth the wrinkles out of the different platforms. To learn more about the wrinkles see Bugs and Testing.

Pexpect does not currently work on the standard Windows Python (see the pty requirement); however, it seems to work fine using Cygwin. It is possible to build something like a pty for Windows, but it would have to use a different technique that I am still investigating. I know it's possible because Libes' Expect was ported to Windows. If you have any ideas or skills to contribute in this area then I would really appreciate some tips on how to approach this problem.

---

Overview

Pexpect can be used for automating interactive applications such as ssh, ftp, mencoder, passwd, etc. The Pexpect interface was designed to be easy to use. Here is an example of Pexpect in action:

# This connects to the openbsd ftp site and
# downloads the recursive directory listing.
import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('cd pub')
child.expect('ftp> ')
child.sendline ('get ls-lR.gz')
child.expect('ftp> ')
child.sendline ('bye')

Obviously you could write an ftp client using Python's own ftplib module, but this is just a demonstration. You can use this technique with any application. This is especially handy if you are writing automated test tools.

There are two important methods in Pexpect -- expect() and send() (or sendline() which is like send() with a linefeed). The expect() method waits for the child application to return a given string. The string you specify is a regular expression, so you can match complicated patterns. The send() method writes a string to the child application. From the child's point of view it looks just like someone typed the text from a terminal. After each call to expect() the before and after properties will be set to the text printed by child application. The before property will contain all text up to the expected string pattern. The after string will contain the text that was matched by the expected pattern. The match property is set to the re MatchObject.

An example of Pexpect in action may make things more clear. This example uses ftp to login to the OpenBSD site; list files in a directory; and then pass interactive control of the ftp session to the human user.

import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('ls /pub/OpenBSD/')
child.expect ('ftp> ')
print child.before   # Print the result of the ls command.
child.interact()     # Give control of the child to the user.

Special EOF and TIMEOUT patterns

There are two special patterns to match the End Of File or a Timeout condition. You you can pass these patterns to expect(). These patterns are not regular expressions. Use them like predefined constants.

If the child has died and you have read all the child's output then ordinarily expect() will raise an EOF exception. You can read everything up to the EOF without generating an exception by using the EOF pattern expect(pexpect.EOF). In this case everything the child has output will be available in the before property.

The pattern given to expect() may be a regular expression or it may also be a list of regular expressions. This allows you to match multiple optional responses. The expect() method returns the index of the pattern that was matched. For example, say you wanted to login to a server. After entering a password you could get various responses from the server -- your password could be rejected; or you could be allowed in and asked for your terminal type; or you could be let right in and given a command prompt. The following code fragment gives an example of this:

child.expect('password:')
child.sendline (my_secret_password)
# We expect any of these three patterns...
i = child.expect (['Permission denied', 'Terminal type', '[#\$] '])
if i==0:
    print 'Permission denied on host. Can't login'
    child.kill(0)
elif i==2:
    print 'Login OK... need to send terminal type.'
    child.sendline('vt100')
    child.expect ('[#\$] ')
elif i==3:
    print 'Login OK.'
    print 'Shell command prompt', child.after

If nothing matches an expected pattern then expect will eventually raise a TIMEOUT exception. The default time is 30 seconds, but you can change this by passing a timeout argument to expect():

# Wait no more than 2 minutes (120 seconds) for password prompt.
child.expect('password:', timeout=120)

Find the end of line -- CR/LF conventions
Matching at the end of a line can be tricky
$ regex pattern is useless.

Pexpect matches regular expressions a little differently than what you might be used to.

The $ pattern for end of line match is useless. The $ matches the end of string, but Pexpect reads from the child one character at a time, so each character looks like the end of a line. Pexpect can't do a look-ahead into the child's output stream. In general you would have this situation when using regular expressions with any stream.
Note, pexpect does have an internal buffer, so reads are faster than one character at a time, but from the user's perspective the regex patterns test happens one character at a time.

The best way to match the end of a line is to look for the newline: "\r\n" (CR/LF). Yes, that does appear to be DOS-style. It may surprise some UNIX people to learn that terminal TTY device drivers (dumb, vt100, ANSI, xterm, etc.) all use the CR/LF combination to signify the end of line. Pexpect uses a Pseudo-TTY device to talk to the child application, so when the child app prints "\n" you actually see "\r\n".

UNIX uses just linefeeds to end lines of text, but not when it comes to TTY devices! TTY devices are more like the Windows world. Each line of text end with a CR/LF combination. When you intercept data from a UNIX command from a TTY device you will find that the TTY device outputs a CR/LF combination. A UNIX command may only write a linefeed (\n), but the TTY device driver converts it to CR/LF. This means that your terminal will see lines end with CR/LF (hex 0D 0A). Since Pexpect emulates a terminal, to match ends of lines you have to expect the CR/LF combination.

child.expect ('\r\n')

If you just need to skip past a new line then expect ('\n') by itself will work, but if you are expecting a specific pattern before the end of line then you need to explicitly look for the \r. For example the following expects a word at the end of a line:

child.expect ('\w+\r\n')

But the following would both fail:

child.expect ('\w+\n')

And as explained before, trying to use '$' to match the end of line would not work either:

child.expect ('\w+$')

So if you need to explicitly look for the END OF LINE, you want to look for the CR/LF combination -- not just the LF and not the $ pattern.

This problem is not limited to Pexpect. This problem happens any time you try to perform a regular expression match on a stream. Regular expressions need to look ahead. With a stream it is hard to look ahead because the process generating the stream may not be finished. There is no way to know if the process has paused momentarily or is finished and waiting for you. Pexpect must implicitly always do a NON greedy match (minimal) at the end of a input {### already said this}.

Pexpect compiles all regular expressions with the DOTALL flag. With the DOTALL flag a "." will match a newline. See the Python documentation

Beware of + and * at the end of input.

Remember that any time you try to match a pattern that needs look-ahead that you will always get a minimal match (non greedy). For example, the following will always return just one character:

child.expect ('.+')

This example will match successfully, but will always return no characters:

child.expect ('.*')

Generally any star * expression will match as little as possible

One thing you can do is to try to force a non-ambiguous character at the end of your \d+ pattern. Expect that character to delimit the string. For example, you might try making the end of your pattrn be \D+ instead of \D*. That means number digits alone would not satisfy the (\d+) pattern. You would need some number(s) and at least one \D at the end.

Matching groups

You can group regular expression using parenthesis. After a match, the match parameter of the spawn object will contain the Python Match object.

Examples

Using "match" and groups...

Debugging

If you get the string value of a pexpect.spawn object you will get lots of useful debugging information. For debugging it's very useful to use the following pattern:

try:
    i = child.expect ([pattern1, pattern2, pattern3, etc])
except:
    print "Exception was thrown"
    print "debug information:"
    print str(child)

It is also useful to log the child's input and out to a file or the screen. The following will turn on logging and send output to stdout (the screen).

    child = pexpect.spawn (foo)
    child.logfile = sys.stdout

---

Exceptions

EOF

Note that two flavors of EOF Exception may be thrown. They are virtually identical except for the message string. For practical purposes you should have no need to distinguish between them, but they do give a little extra information about what type of platform you are running. The two messages are:

End Of File (EOF) in read(). Exception style platform.

End Of File (EOF) in read(). Empty string style platform.

Some UNIX platforms will throw an exception when you try to read from a file descriptor in the EOF state. Other UNIX platforms instead quietly return an empty string to indicate that the EOF state has been reached.

Expecting EOF

If you wish to read up to the end of the child's output without generating an EOF exception then use the expect(pexpect.EOF) method.

TIMEOUT

The expect() and read() methods will also timeout if the child does not generate any output for a given amount of time. If this happens they will raise a TIMEOUT exception. You can have these method ignore a timeout and block indefinitely by passing None for the timeout parameter.

child.expect(pexpect.EOF, timeout=None)

---

FAQ

Q: Why don't shell pipe and redirect (| and >) work when I spawn a command?

A: Remember that Pexpect does NOT interpret shell meta characters such as redirect, pipe, or wild cards (>, |, or *). That's done by a shell not the command you are spawning. This is a common mistake. If you want to run a command and pipe it through another command then you must also start a shell. For example:

    child = pexpect.spawn('/bin/bash -c "ls -l | grep LOG > log_list.txt"')
    child.expect(pexpect.EOF)

The second form of spawn (where you pass a list of arguments) is useful in situations where you wish to spawn a command and pass it its own argument list. This can make syntax more clear. For example, the following is equivalent to the previous example:

    shell_cmd = 'ls -l | grep LOG > log_list.txt'
    child = pexpect.spawn ('/bin/bash', ['-c', shell_cmd])
    child.expect (pexpect.EOF)

Q: Isn't there already a Python Expect?

A: Yes, there are several of them. They usually require you to compile C. I wanted something that was pure Python and preferably a single module that was simple to install. I also wanted something that was easy to use. This pure Python expect only recently became possible with the introduction of the pty module in the standard Python library. Previously C extensions were required.

Q: The before and after properties sound weird.

Originally I was going to model Pexpect more after Expect, but then I found that I could never remember how to get the context of the stuff I was trying to parse. I hate having to read my own documentation. I decided that it was easier for me to remember what before and after was. It just so happens that this is how the -B and -A options in grep works, so that made it even easier for me to remember. Whatever makes my life easier is what's best.

Q: Why not just use Expect?

A: I love it. It's great. I has bailed me out of some real jams, but I wanted something that would do 90% of what I need from Expect; be 10% of the size; and allow me to write my code in Python instead of TCL. Pexpect is not nearly as big as Expect, but Pexpect does everything I have ever used Expect for.

Q: Why not just use a pipe (popen())?

A: A pipe works fine for getting the output to non-interactive programs. If you just want to get the output from ls, uname, or ping then this works. Pipes do not work very well for interactive programs and pipes will almost certainly fail for most applications that ask for passwords such as telnet, ftp, or ssh.

There are two reasons for this.

First an application may bypass stdout and print directly to its controlling TTY. Something like SSH will do this when it asks you for a password. This is why you cannot redirect the password prompt because it does not go through stdout or stderr.

The second reason is because most applications are built using the C Standard IO Library (anything that uses #include <stdio.h>). One of the features of the stdio library is that it buffers all input and output. Normally output is line buffered when a program is printing to a TTY (your terminal screen). Every time the program prints a line-feed the currently buffered data will get printed to your screen. The problem comes when you connect a pipe. The stdio library is smart and can tell that it is printing to a pipe instead of a TTY. In that case it switches from line buffer mode to block buffered. In this mode the currently buffered data is flushed when the buffer is full. This causes most interactive programs to deadlock. Block buffering is more efficient when writing to disks and pipes. Take the situation where a program prints a message "Enter your user name:\n" and then waits for you type type something. In block buffered mode, the stdio library will not put the message into the pipe even though a linefeed is printed. The result is that you never receive the message, yet the child application will sit and wait for you to type a response. Don't confuse the stdio lib's buffer with the pipe's buffer. The pipe buffer is another area that can cause problems. You could flush the input side of a pipe, whereas you have no control over the stdio library buffer.

More information: the Standard IO library has three states for a FILE *. These are: _IOFBF for block buffered; _IOLBF for line buffered; and _IONBF for unbuffered. The STDIO lib will use block buffering when talking to a block file descriptor such as a pipe. This is usually not helpful for interactive programs. Short of recompiling your program to include fflush() everywhere or recompiling a custom stdio library there is not much a controlling application can do about this if talking over a pipe.

The program may have put data in its output that remains unflushed because the output buffer is not full; then the program will go and deadlock while waiting for input -- because you never send it any because you are still waiting for its output (still stuck in the STDIO's output buffer).

The answer is to use a pseudo-tty. A TTY device will force line buffering (as opposed to block buffering). Line buffering means that you will get each line when the child program sends a line feed. This corresponds to the way most interactive programs operate -- send a line of output then wait for a line of input.

I put "answer" in quotes because it's ugly solution and because there is no POSIX standard for pseudo-TTY devices (even though they have a TTY standard...). What would make more sense to me would be to have some way to set a mode on a file descriptor so that it will tell the STDIO to be line-buffered. I have investigated, and I don't think there is a way to set the buffered state of a child process. The STDIO Library does not maintain any external state in the kernel or whatnot, so I don't think there is any way for you to alter it. I'm not quite sure how this line-buffered/block-buffered state change happens internally in the STDIO library. I think the STDIO lib looks at the file descriptor and decides to change behavior based on whether it's a TTY or a block file (see isatty()).

I hope that this qualifies as helpful.

Don't use a pipe to control another application...

Pexpect may seem similar to os.popen() or commands module. The main difference is that Pexpect (like Expect) uses a pseudo-TTY to talk to the child application. Most applications do no work well through the system() call or through pipes. And probably all applications that ask a user to type in a password will fail. These applications bypass the stdin and read directly from the TTY device. Many applications do not explicitly flush their output buffers. This causes deadlocks if you try to control an interactive application using a pipe. What happens is that most UNIX applications use the stdio (#include <stdio.h>) for input and output. The stdio library behaves differently depending on where the output is going. There is no way to control this behavior from the client end.

Q: Can I do screen scraping with this thing?

A: That depends. If your application just does line-oriented output then this is easy. If it does screen-oriented output then it may work, but it could be hard. For example, trying to scrape data from the 'top' command would be hard. The top command repaints the text window.

I am working on an ANSI / VT100 terminal emulator that will have methods to get characters from an arbitrary X,Y coordinate of the virtual screen. It works and you can play with it, but I have no working examples at this time.

---

Bugs

Threads

On Linux (RH 8) you cannot spawn a child from a different thread and pass the handle back to a worker thread. The child is successfully spawned but you can't interact with it. The only way to make it work is to spawn and interact with the child all in the same thread. [Adam Kerrison]

Timing issue with send() and sendline()

This problem has been addressed and should not effect most users.

It is sometimes possible to read an echo of the string sent with send() and sendline(). If you call sendline() and then immediately call readline() you may get part of your output echoed back. You may read back what you just wrote even if the child application does not explicitly echo it. Timing is critical. This could be a security issue when talking to an application that asks for a password; otherwise, this does not seem like a big deal. But why do TTYs do this?

People usually report this when they are trying to control SSH or some other login. For example, if your code looks something like this:

child.expect ('[pP]assword:')
child.sendline (my_password)

1. SSH prints "password:" prompt to the user.
2. SSH turns off echo on the TTY device.
3. SSH waits for user to enter a password.

When scripting with Pexpect what can happen is that Pexpect will response to the "password:" prompt before SSH has had time to turn off TTY echo. In other words, Pexpect sends the password between steps 1. and 2., so the password gets echoed back to the TTY. I would call this an SSH bug.

Pexpect now automatically adds a short delay before sending data to a child process. This more closely mimics what happens in the usual human-to-app interaction. The delay can be tuned with the 'delaybeforesend' attribute of the spawn class. In general, this fixes the problem for everyone and so this should not be an issue for most users. For some applications you might with to turn it off. child = pexpect.spawn ("ssh user@example.com") child.delaybeforesend = 0

Try changing it to look like the following. I know that this fix does not look correct, but it works. I have not figured out exactly what is happening. You would think that the sleep should be after the sendline(). The fact that the sleep helps when it's between the expect() and the sendline() must be a clue.

child.expect ('[pP]assword:')
child.sendline (my_password)

Timing issue with isalive()

Reading the state of isalive() immediately after a child exits may sometimes return 1. This is a race condition. The child has closed its file descriptor, but has not yet fully exited before Pexpect's isalive() executes. Addings a slight delay before the isalive() will help. In the following example isalive() sometimes returns 1:

child = pexpect.spawn('ls')
child.expect(pexpect.EOF)
print child.isalive()

But if there is any delay before the call to isalive() then it will always return 0 as expected.

child = pexpect.spawn('ls')
child.expect(pexpect.EOF)
time.sleep(0.1)
print child.isalive()

Truncated output just before child exits

So far I have seen this only on older versions of Apple's MacOS X. If the child application quits it may not flush its output buffer. This means that your Pexpect application will receive an EOF even though it should have received a little more data before the child died. This is not generally a problem when talking to interactive child applications. One example where it is a problem is when trying to read output from a program like 'ls'. You may receive most of the directory listing, but the last few lines will get lost before you receive an EOF. The reason for this is that 'ls' runs; completes its task; and then exits. The buffer is not flushed before exit so the last few lines are lost. The following example demonstrates the problem:

child = pexpect.spawn ('ls -l')
child.expect (pexpect.EOF)
print child.before 
  

Controlling SSH on Solaris

Pexpect does not yet work perfectly on Solaris. One common problem is that SSH sometimes will not allow TTY password authentication. For example, you may expect SSH to ask you for a password using code like this:

child = pexpect.spawn ('ssh user@example.com')
child.expect ('assword')
child.sendline ('mypassword')

You may see the following error come back from a spawned child SSH:

Permission denied (publickey,keyboard-interactive).

This means that SSH thinks it can't access the TTY to ask you for your password. The only solution I have found is to use public key authentication with SSH. This bypasses the need for a password. I'm not happy with this solution. The problem is due to poor support for Solaris Pseudo TTYs in the Python Standard Library.

---

CHANGES

Current Release

Fixed OSError exception when a pexpect object is cleaned up. Previously you might have seen this exception:

Exception exceptions.OSError: (10, 'No child processes') 
in <bound method spawn.__del__ of
<pexpect.spawn instance at 0xd248c>> ignored

You should not see that anymore. Thanks to Michael Surette.

Added support for buffering reads. This greatly improves speed when trying to match long output from a child process. When you create an instance of the spawn object you can then set a buffer size. For now you MUST do the following to turn on buffering -- it may be on by default in future version.

child = pexpect.spawn ('my_command')
child.maxread=1000 # Sets buffer to 1000 characters.

---

INDEX

---

License
Download
Documentation
Project Status
Requirements
Overview
FAQ
Known Bugs
Recent Changes
Testing
To do
Browse SVN