On Debian/Ubuntu you can use apt-get:

sudo apt-get install dejagnu tcllib

This should also install the necessary expect and tcl packages.

On Fedora and RHEL/CentOS (with EPEL) you can use yum:

sudo yum install dejagnu tcllib

This should also install the necessary expect and tcl packages.

The tests are grouped into different areas, called tool in DejaGnu:

Each tool has a slightly different way of loading the test fixtures, see Test context below.

Completion tests are spread over two directories: completion/\*.exp calls completions in lib/completions/\*.exp. This two-file system stems from bash-completion-lib (http://code.google.com/p/bash-completion-lib/, containing dynamic loading of completions) where tests are run twice per completion; once before dynamic loading and a second time after to confirm that all dynamic loading has gone well.

For example:

set test "Completion via comp_load() should be installed"
set cmd "complete -p awk"
send "$cmd\r"
expect {
    -re "^$cmd\r\ncomplete -o filenames -F comp_load awk\r\n/@$" { pass "$test" }
    -re /@ { fail "$test at prompt" }
}


source "lib/completions/awk.exp"


set test "Completion via _longopt() should be installed"
set cmd "complete -p awk"
send "$cmd\r"
expect {
    -re "^$cmd\r\ncomplete -o filenames -F _longopt awk\r\n/@$" { pass "$test" }
    -re /@ { fail "$test at prompt" }
}


source "lib/completions/awk.exp"

Looking to the completion tests from a broader perspective, every test for a command has two stages which are now reflected in the two files:

The test suite requires a connected terminal (tty). When invoked via cron, no tty is connected and the test suite may respond with this error:

can't read "multipass_name": no such variable

To run the tests successfully via cron, connect a terminal by redirecting stdin from a tty, e.g. /dev/tty40. (In Linux, you can press alt-Fx or ctrl-alt-Fx to switch the console from /dev/tty1 to tty7. There are many more /dev/tty* which are not accessed via function keys. To be safe, use a tty greater than tty7)

./runUnit < /dev/tty40

If the process doesn’t run as root (recommended), root will have to change the owner and permissions of the tty:

sudo chmod o+r /dev/tty40

To make this permission permanent (at least on Debian) - and not revert back on reboot - create the file /etc/udev/rules.d/10-mydejagnu.rules, containing:

KERNEL=="tty40", MODE="0666"

To start the test at 01:00, set the crontab to this:

* 1 * * * cd bash-completion/test && ./cron.sh < /dev/tty40

Here’s an example batch file cron.sh, to be put in the bash-completion test directory. This batch file only e-mails the output of each test-run if the test-run fails.

#!/bin/sh

set -e  # Exit if simple command fails
set -u  # Error if variable is undefined

CRON=running
LOG=/tmp/bash-completion.log~

    # Retrieve latest sources
git pull

    # Run tests on bash-4

./runUnit --outdir log/bash-4 --tool_exec /opt/bash-4.0/bin/bash > $LOG || cat $LOG
./runCompletion --outdir log/bash-4 --tool_exec /opt/bash-4.0/bin/bash > $LOG || cat $LOG

    # Clean up log file
[ -f $LOG ] && rm $LOG

The test suite standard uses bash as found in the tcl path (/bin/bash). Using --tool_exec you can specify which bash binary you want to run the test suite against, e.g.:

./runUnit --tool_exec /opt/bash-4.0/bin/bash

You can run cd test && ./generate cmd to add a test for the cmd command. This will add two files with a very basic tests:

test/completion/cmd.exp
test/lib/completions/cmd.exp

Place any additional tests into test/lib/completions/cmd.exp.

Let’s consider this real-life example where an ssh completion bug is fixed. First you’re triggered by unsuccessful tests:

$ ./runCompletion
...
        === completion Summary ===

# of expected passes        283
# of unexpected failures    8
# of unresolved testcases   2
# of unsupported tests      47

Take a look in log/completion.log to find out which specific command is failing.

$ vi log/completion.log

Search for UNRESOLVED or FAIL. From there scroll up to see which .exp test is failing:

/@Running ./completion/ssh.exp ...
...
UNRESOLVED: Tab should complete ssh known-hosts at prompt

In this case it appears ssh.exp is causing the problem. Isolate the ssh tests by specifying just ssh.exp to run. Furthermore add the --debug flag, so output gets logged in dbg.log:

$ ./runCompletion ssh.exp --debug
...
        === completion Summary ===

# of expected passes        1
# of unresolved testcases   1

Now we can have a detailed look in dbg.log to find out what’s going wrong. Open dbg.log and search for UNRESOLVED (or FAIL if that’s what you’re looking for):

UNRESOLVED: Tab should complete ssh known-hosts at prompt

From there, search up for the first line saying:

expect: does "..." match regular expression "..."

This tells you where the actual output differs from the expected output. In this case it looks like the test "ssh -F fixtures/ssh/config <TAB>" is expecting just hostnames, whereas the actual completion is containing commands - but no hostnames. So what should be expected after "ssh -F fixtures/ssh/config <TAB>" are both commands and hostnames. This means both the test and the completion need fixing. Let’s start with the test.

$ vi lib/completions/ssh.exp

Search for the test "Tab should complete ssh known-hosts". Here you could’ve seen that what was expected were hostnames ($hosts):

set expected "^$cmd\r\n$hosts\r\n/@$cmd$"

Adding all commands (which could well be over 2000) to expected, seems a bit overdone so we’re gonna change things here. Lets expect the unit test for _known_hosts assures all hosts are returned. Then all we need to do here is expect one host and one command, just to be kind of sure that both hosts and commands are completed.

Looking in the fixture for ssh:

$ vi fixtures/ssh/known_hosts

it looks like we can add an additional host ls_known_host. Now if we would perform the test "ssh -F fixtures/ssh/config ls<TAB>" both the command ls and the host ls_known_host should come up. Let’s modify the test so:

$ vi lib/completions/ssh.exp
...
set expected "^$cmd\r\n.*ls.*ls_known_host.*\r\n/@$cmd$"

Running the test reveals we still have an unresolved test:

$ ./runCompletion ssh.exp --debug
...
        === completion Summary ===

# of expected passes        1
# of unresolved testcases   1

But if now look into the log file ‘dbg.log` we can see the completion only returns commands starting with ls but fails to match our regular expression which also expects the hostname `ls_known_host’:

$ vi dbg.log
...
expect: does "ssh -F fixtures/ssh/config ls\r\nls           lsattr       lsb_release  lshal        lshw         lsmod        lsof         lspci        lspcmcia     lspgpot      lss16toppm\r\nlsusb\r\n/@ssh -F fixtures/ssh/config ls" (spawn_id exp9) match regular expression "^ssh -F fixtures/ssh/config ls\r\n.*ls.*ls_known_host.*\r\n/@ssh -F fixtures/ssh/config ls$"? no

Now let’s fix ssh completion:

$ vi ../contrib/ssh
...

until the test shows:

$ ./runCompletion ssh.exp
...
        === completion Summary ===

# of expected passes        2

Now let’s consider a unit test failure. First you’re triggered by unsuccessful tests:

$ ./runUnit
...
        === unit Summary ===

# of expected passes        1
# of unexpected failures    1

Take a look in log/unit.log to find out which specific command is failing.

$ vi log/unit.log

Search for UNRESOLVED or FAIL. From there scroll up to see which .exp test is failing:

/@Running ./unit/_known_hosts_real.exp ...
...
FAIL: Environment should stay clean

In this case it appears _known_hosts_real.exp is causing the problem. Isolate the _known_hosts_real test by specifying just _known_hosts_real.exp to run. Furthermore add the --debug flag, so output gets logged in dbg.log:

$ ./runUnit _known_hosts_real.exp --debug
...
        === completion Summary ===

# of expected passes        1
# of unexpected failures    1

Now, if we haven’t already figured out the problem, we can have a detailed look in dbg.log to find out what’s going wrong. Open dbg.log and search for UNRESOLVED (or FAIL if that’s what you’re looking for):

FAIL: Environment should stay clean

From there, search up for the first line saying:

expect: does "..." match regular expression "..."

This tells you where the actual output differs from the expected output. In this case it looks like the the function _known_hosts_real is unexpectedly modifying global variables cur and flag. In case you need to modify the test:

$ vi lib/unit/_known_hosts_real.exp

This is the bash configuration file (bashrc) used for testing:

# bashrc file for DejaGnu testsuite

        # Use emacs key bindings
set -o emacs
        # Use bash strict mode
set -o posix
        # Unset `command_not_found_handle' as defined on Debian/Ubuntu, because this
        # troubles and slows down testing
unset -f command_not_found_handle
        # Set fixed prompt `/@'
TESTDIR=$(pwd)
export PS1='/@'
export PS2='> '
        # Configure readline
export INPUTRC=$SRCDIR/config/inputrc
    # When not running via cron, avoid escape junk at beginning of line from
    # readline, see e.g.  http://bugs.gentoo.org/246091
[ "$CRON" ] || export TERM=dummy
        # Ensure enough columns so expect doesn't have to care about line breaks
stty columns 150
    # Also test completions of system administrator commands, which are
    # installed via the same PATH expansion in `bash_completion.have()'
export PATH=$PATH:/sbin:/usr/sbin:/usr/local/sbin

        # Make sure default settings are in effect
unset -v \
    COMP_CONFIGURE_HINTS \
    COMP_CVS_REMOTE \
    COMP_KNOWN_HOSTS_WITH_HOSTFILE \
    COMP_TAR_INTERNAL_PATHS

        # Load bash testsuite helper functions
. $SRCDIR/lib/library.sh

# Local variables:
# mode: shell-script
# sh-basic-offset: 4
# sh-indent-comment: t
# indent-tabs-mode: nil
# End:
# ex: ts=4 sw=4 et filetype=sh

This is the readline configuration file (inputrc) used for testing:

# Readline init file for DejaGnu testsuite
# See: info readline

        # Press TAB once (instead of twice) to auto-complete
set show-all-if-ambiguous on
        # No bell.  No ^G in output
set bell-style none
        # Don't query user about viewing the number of possible completions
set completion-query-items -1
        # Display completions sorted horizontally, not vertically
set print-completions-horizontally on
        # Don't use pager when showing completions
set page-completions off

# Local variables:
# mode: shell-script
# sh-basic-offset: 4
# sh-indent-comment: t
# indent-tabs-mode: nil
# End:
# ex: ts=4 sw=4 et filetype=sh