summaryrefslogtreecommitdiff
path: root/doc/testing.txt
blob: 505d0848759be1d82179374e6864821953c75133 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
Automated testing
=================

Introduction
------------
The bash-completion package contains an automated test suite.  Running the
tests should help verifying that bash-completion works as expected.  The tests
are also very helpful in uncovering software regressions at an early stage.

The bash-completion test suite is written on top of the
http://www.gnu.org/software/dejagnu/[DejaGnu] testing framework.  DejaGnu is
written in http://expect.nist.gov[Expect], which in turn uses
http://tcl.sourceforge.net[Tcl] -- Tool command language.


Coding Style Guide
------------------

The bash-completion test suite tries to adhere to this
http://wiki.tcl.tk/708[Tcl Style Guide].


Installing dependencies
-----------------------

Installing dependencies should be easy using your local package manager.


Debian/Ubuntu
~~~~~~~~~~~~~

On Debian/Ubuntu you can use `apt-get`:
-------------
sudo apt-get install dejagnu tcllib
-------------
This should also install the necessary `expect` and `tcl` packages.

Fedora/RHEL/CentOS
~~~~~~~~~~~~~~~~~~

On Fedora and RHEL/CentOS (with EPEL) you can use `yum`:
-------------
sudo yum install dejagnu tcllib
-------------
This should also install the necessary `expect` and `tcl` packages.



Structure
---------


Main areas (DejaGnu tools)
~~~~~~~~~~~~~~~~~~~~~~~~~~

The tests are grouped into different areas, called _tool_ in DejaGnu:

*completion*::
  Functional tests per completion.
*install*:: 
  Functional tests for installation and caching of the main bash-completion
  package.
*unit*:: 
  Unit tests for bash-completion helper functions.

Each tool has a slightly different way of loading the test fixtures, see
<<Test_context,Test context>> below.


Completion
~~~~~~~~~~

Completion tests are spread over two directories: `completion/\*.exp` calls
completions in `lib/completions/\*.exp`.  This two-file system stems from
bash-completion-lib (http://code.google.com/p/bash-completion-lib/, containing
dynamic loading of completions) where tests are run twice per completion; once
before dynamic loading and a second time after to confirm that all dynamic
loading has gone well.

For example:

----
set test "Completion via comp_load() should be installed"
set cmd "complete -p awk"
send "$cmd\r"
expect {
    -re "^$cmd\r\ncomplete -o filenames -F comp_load awk\r\n/@$" { pass "$test" }
    -re /@ { fail "$test at prompt" }
}


source "lib/completions/awk.exp"


set test "Completion via _longopt() should be installed"
set cmd "complete -p awk"
send "$cmd\r"
expect {
    -re "^$cmd\r\ncomplete -o filenames -F _longopt awk\r\n/@$" { pass "$test" }
    -re /@ { fail "$test at prompt" }
}


source "lib/completions/awk.exp"
----

Looking to the completion tests from a broader perspective, every test for a
command has two stages which are now reflected in the two files:

. Tests concerning the command completions' environment (typically in
`test/completion/foo`)
. Tests invoking actual command completion (typically in
`test/lib/completions/foo`)


Running the tests
-----------------

The tests are run by calling `runtest` in the test directory:
-----------------------
runtest --outdir log --tool completion
runtest --outdir log --tool install
runtest --outdir log --tool unit
-----------------------
The commands above are already wrapped up in shell scripts within the `test`
directory:
-----------------------
./runCompletion
./runInstall
./runUnit
-----------------------


Running tests via cron
~~~~~~~~~~~~~~~~~~~~~~

The test suite requires a connected terminal (tty).  When invoked via cron, no
tty is connected and the test suite may respond with this error:
---------------------------------------------
can't read "multipass_name": no such variable
---------------------------------------------

To run the tests successfully via cron, connect a terminal by redirecting
stdin from a tty, e.g. /dev/tty40.  (In Linux, you can press alt-Fx or
ctrl-alt-Fx to switch the console from /dev/tty1 to tty7.  There are many more
/dev/tty* which are not accessed via function keys.  To be safe, use a tty
greater than tty7)

----------------------
./runUnit < /dev/tty40
----------------------

If the process doesn't run as root (recommended), root will have to change the
owner and permissions of the tty:
-------------------------
sudo chmod o+r /dev/tty40
-------------------------

To make this permission permanent (at least on Debian) - and not revert back on
reboot - create the file `/etc/udev/rules.d/10-mydejagnu.rules`, containing:
----------------------------
KERNEL=="tty40", MODE="0666"
----------------------------

To start the test at 01:00, set the crontab to this:
----------------------------
* 1 * * * cd bash-completion/test && ./cron.sh < /dev/tty40
----------------------------

Here's an example batch file `cron.sh`, to be put in the bash-completion `test`
directory.  This batch file only e-mails the output of each test-run if the
test-run fails.

[source,bash]
---------------------------------------------------------------------
#!/bin/sh

set -e  # Exit if simple command fails
set -u  # Error if variable is undefined

CRON=running
LOG=/tmp/bash-completion.log~

    # Retrieve latest sources
git pull

    # Run tests on bash-3

./runUnit       > $LOG || cat $LOG
./runCompletion > $LOG || cat $LOG

    # Run tests on bash-4

./runUnit --outdir log/bash-4 --tool_exec /opt/bash-4.0/bin/bash > $LOG || cat $LOG
./runCompletion --outdir log/bash-4 --tool_exec /opt/bash-4.0/bin/bash > $LOG || cat $LOG

    # Clean up log file
[ -f $LOG ] && rm $LOG
---------------------------------------------------------------------

Specifying bash binary
~~~~~~~~~~~~~~~~~~~~~~

The test suite standard uses `bash` as found in the tcl path (/bin/bash).
Using `--tool_exec` you can specify which bash binary you want to run the test
suite against, e.g.:

----------------
./runUnit --tool_exec /opt/bash-4.0/bin/bash
----------------




Maintenance
-----------


Adding a completion test
~~~~~~~~~~~~~~~~~~~~~~~~

You can run `cd test && ./generate` to add a test.


Fixing a completion test
~~~~~~~~~~~~~~~~~~~~~~~~
Let's consider this real-life example where an ssh completion bug is fixed.
First you're triggered by unsuccessful tests:

----------------------------------
$ ./runCompletion
...
        === completion Summary ===

# of expected passes        283
# of unexpected failures    8
# of unresolved testcases   2
# of unsupported tests      47
----------------------------------

Take a look in `log/completion.log` to find out which specific command is
failing.

-----------------------
$ vi log/completion.log
-----------------------

Search for `UNRESOLVED` or `FAIL`.  From there scroll up to see which `.exp`
test is failing:

---------------------------------------------------------
/@Running ./completion/ssh.exp ...
...
UNRESOLVED: Tab should complete ssh known-hosts at prompt
---------------------------------------------------------

In this case it appears `ssh.exp` is causing the problem.  Isolate the `ssh`
tests by specifying just `ssh.exp` to run.  Furthermore add the `--debug` flag,
so output gets logged in `dbg.log`:

----------------------------------
$ ./runCompletion ssh.exp --debug
...
        === completion Summary ===

# of expected passes        1
# of unresolved testcases   1
----------------------------------

Now we can have a detailed look in `dbg.log` to find out what's going wrong.
Open `dbg.log` and search for `UNRESOLVED` (or `FAIL` if that's what you're
looking for):

---------------------------------------------------------
UNRESOLVED: Tab should complete ssh known-hosts at prompt
---------------------------------------------------------

From there, search up for the first line saying: 

-------------------------------------------------
expect: does "..." match regular expression "..."
-------------------------------------------------

This tells you where the actual output differs from the expected output.  In
this case it looks like the test "ssh -F fixtures/ssh/config <TAB>" is
expecting just hostnames, whereas the actual completion is containing commands
- but no hostnames.
So what should be expected after "ssh -F fixtures/ssh/config <TAB>" are *both*
commands and hostnames.  This means both the test and the completion need
fixing.  Let's start with the test.

----------------------------
$ vi lib/completions/ssh.exp
----------------------------

Search for the test "Tab should complete ssh known-hosts".  Here you could've
seen that what was expected were hostnames ($hosts):

-----------------------------------------
set expected "^$cmd\r\n$hosts\r\n/@$cmd$"
-----------------------------------------

Adding *all* commands (which could well be over 2000) to 'expected', seems a
bit overdone so we're gonna change things here.  Lets expect the unit test for
`_known_hosts` assures all hosts are returned.  Then all we need to do here is
expect one host and one command, just to be kind of sure that both hosts and
commands are completed.

Looking in the fixture for ssh:

-----------------------------
$ vi fixtures/ssh/known_hosts
-----------------------------

it looks like we can add an additional host 'ls_known_host'.  Now if we would
perform the test "ssh -F fixtures/ssh/config ls<TAB>" both the command `ls` and
the host `ls_known_host` should come up.  Let's modify the test so:

--------------------------------------------------------
$ vi lib/completions/ssh.exp
...
set expected "^$cmd\r\n.*ls.*ls_known_host.*\r\n/@$cmd$"
--------------------------------------------------------

Running the test reveals we still have an unresolved test:

----------------------------------
$ ./runCompletion ssh.exp --debug
...
        === completion Summary ===

# of expected passes        1
# of unresolved testcases   1
----------------------------------

But if now look into the log file `dbg.log` we can see the completion only
returns commands starting with 'ls' but fails to match our regular expression
which also expects the hostname `ls_known_host':

-----------------------
$ vi dbg.log
...
expect: does "ssh -F fixtures/ssh/config ls\r\nls           lsattr       lsb_release  lshal        lshw         lsmod        lsof         lspci        lspcmcia     lspgpot      lss16toppm\r\nlsusb\r\n/@ssh -F fixtures/ssh/config ls" (spawn_id exp9) match regular expression "^ssh -F fixtures/ssh/config ls\r\n.*ls.*ls_known_host.*\r\n/@ssh -F fixtures/ssh/config ls$"? no
-----------------------

Now let's fix ssh completion:

-------------------
$ vi ../contrib/ssh
...
-------------------

until the test shows:

----------------------------------
$ ./runCompletion ssh.exp
...
        === completion Summary ===

# of expected passes        2
----------------------------------

Fixing a unit test
~~~~~~~~~~~~~~~~~~
Now let's consider a unit test failure.  First you're triggered by unsuccessful
tests:

----------------------------------
$ ./runUnit
...
        === unit Summary ===

# of expected passes        1
# of unexpected failures    1
----------------------------------

Take a look in `log/unit.log` to find out which specific command is failing.

-----------------
$ vi log/unit.log
-----------------

Search for `UNRESOLVED` or `FAIL`.  From there scroll up to see which `.exp`
test is failing:

------------------------------------------
/@Running ./unit/_known_hosts_real.exp ...
...
FAIL: Environment should stay clean
------------------------------------------

In this case it appears `_known_hosts_real.exp` is causing the problem.
Isolate the `_known_hosts_real` test by specifying just `_known_hosts_real.exp`
to run.  Furthermore add the `--debug` flag, so output gets logged in
`dbg.log`:

----------------------------------
$ ./runUnit _known_hosts_real.exp --debug
...
        === completion Summary ===

# of expected passes        1
# of unexpected failures    1
----------------------------------

Now, if we haven't already figured out the problem, we can have a detailed look
in `dbg.log` to find out what's going wrong.  Open `dbg.log` and search for
`UNRESOLVED` (or `FAIL` if that's what you're looking for):

-----------------------------------
FAIL: Environment should stay clean
-----------------------------------

From there, search up for the first line saying: 

-------------------------------------------------
expect: does "..." match regular expression "..."
-------------------------------------------------

This tells you where the actual output differs from the expected output.  In
this case it looks like the the function `_known_hosts_real` is unexpectedly
modifying global variables `cur` and `flag`.  In case you need to modify the
test:

-----------------------------------
$ vi lib/unit/_known_hosts_real.exp
-----------------------------------

Rationale
---------


Naming conventions
~~~~~~~~~~~~~~~~~~

Test suite or testsuite
^^^^^^^^^^^^^^^^^^^^^^^
The primary Wikipedia page is called
http://en.wikipedia.org/wiki/Test_suite[test suite] and not testsuite, so
that's what this document sticks to.

script/generate
^^^^^^^^^^^^^^^
The name and location of this code generation script come from Ruby on Rails'
http://en.wikibooks.org/wiki/Ruby_on_Rails/Tools/Generators[script/generate].




== Reference

Within test scripts the following library functions can be used:

[[Test_context]]
== Test context

The test environment needs to be put to fixed states when testing.  For
instance the bash prompt (PS1) is set to the current test directory, followed
by an ampersand (@).  The default settings for `bash` reside in `config/bashrc`
and `config/inputrc`.

For each tool (completion, install, unit) a slightly different context is in
effect.

=== What happens when tests are run?

==== completion

When the completions are tested, invoking DejaGnu will result in a call to
`completion_start()` which in turn will start `bash --rcfile config/bashrc`.

.What happens when completion tests are run?
----
               | runtest --tool completion
               V
    +----------+-----------+
    |  lib/completion.exp  |
    |  lib/library.exp     |
    |  config/default.exp  |
    +----------+-----------+
               :
               V
    +----------+-----------+    +---------------+    +----------------+
    |  completion_start()  +<---+ config/bashrc +<---| config/inputrc |
    | (lib/completion.exp) |    +---------------+    +----------------+
    +----------+-----------+
               |                                   ,+----------------------------+
               |                               ,--+-+  "Actual completion tests" |
               V                              /   +------------------------------+
    +----------+-----------+    +-----------------------+
    |   completion/*.exp   +<---| lib/completions/*.exp | 
    +----------+-----------+    +-----------------------+ 
               |          \                        ,+--------------------------------+
               |           `----------------------+-+  "Completion invocation tests" |
               V                                  +----------------------------------+
    +----------+-----------+
    |   completion_exit()  |
    | (lib/completion.exp) |
    +----------------------+
----
Setting up bash once within `completion_start()` has the speed advantage that
bash - and bash-completion - need only initialize once when testing multiple
completions, e.g.:
----
    runtest --tool completion alias.exp cd.exp
----
==== install

.What happens when install tests are run?
----
                 | runtest --tool install
                 V
            +----+----+
            | DejaGnu |
            +----+----+
                 |
                 V
    +------------+---------------+
    | (file: config/default.exp) |
    +------------+---------------+
                 |
                 V
    +------------+------------+
    | (file: lib/install.exp) |
    +-------------------------+
----

==== unit

.What happens when unit tests are run?
----
               | runtest --tool unit
               V
          +----+----+
          | DejaGnu |
          +----+----+
               |
               V
    +----------+-----------+
    |          -           |
    | (file: lib/unit.exp) |
    +----------------------+
----

=== bashrc

This is the bash configuration file (bashrc) used for testing:

[source,bash]
---------------------------------------------------------------------
include::bashrc[]
---------------------------------------------------------------------


=== inputrc

This is the readline configuration file (inputrc) used for testing:

[source,bash]
---------------------------------------------------------------------
include::inputrc[]
---------------------------------------------------------------------


Index
=====