1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
|
The Automake test suite
User interface
==============
Running the tests
-----------------
To run all tests:
make -k check
You can use `-jN' for faster completion (it even helps on a
uniprocessor system, due to unavoidable sleep delays, as
noted below).
To rerun only failed tests:
make -k recheck
To run only tests that are newer than their last results:
make -k check RECHECK_LOGS=
To run only selected tests:
make -k check TESTS="foo.test bar.test" (GNU make)
env TESTS="foo.test bar.test" make -e -k check (non-GNU make)
To run the tests in cross-compilation mode:
make -k check host_alias="$host_alias" (GNU make)
env host_alias="$host_alias" make -e -k check (non-GNU make)
Here `$host_alias' should be defined to a proper value different from
configure-determined `$build_alias', and should refer to a set of
cross-compilers you have available on your system; for example, if
on Linux you have a set of MinGW-targeted cross-compilers named
'i586-mingw32msvc-cc', 'i586-mingw32msvc-c++', etc., you could use:
make -k check host_alias='i586-mingw32msvc'
Interpretation
--------------
Successes:
PASS - success
XFAIL - expected failure
Failures:
FAIL - failure
XPASS - unexpected success
Other:
SKIP - skipped tests (third party tools not available)
Getting details from failures
-----------------------------
Each test is a shell script. In a non-VPATH build you can run the
tests directly, they will be verbose. By default, verbose output of
a test foo.test is retained in the log file foo.log. A summary log
is created in the file test-suite.log.
You can limit the set of files using the TESTS variable, and enable
detailed test output at the end of the test run with the VERBOSE
variable:
env VERBOSE=x TESTS='first.test second.test ...' make -e check
Supported shells
----------------
By default, the tests are run by the $SHELL detected at configure
time. They also take care to re-execute themselves with that shell,
unless told not to. So, to run the tests with a different shell, say
`/path/to/another/sh', the user must use:
AM_TESTS_REEXEC=no /path/to/another/sh ./foo.test
to run a test directly, and:
make check TEST_LOG_COMPILER=/path/to/sh (GNU make)
TEST_LOG_COMPILER=/path/to/sh make -e check (non-GNU make)
to run the test(s) through the makefile test driver.
The test scripts are written with portability in mind, so that they
should run with any decent Bourne-compatible shell.
However, some care must be used with Zsh, since, when not directly
started in Bourne-compatibility mode, it has some incompatibilities
in the handling of `$0' which conflict with our usage. Our testsuite
can automatically work around these incompatibilities when a version
4.3 or later of Zsh is used, but unfortunately not when an older
version of Zsh is used. Thus, if you want to run a test script, say
foo.test, with Zsh 4.2, you *can't* simply do `zsh foo.test', but
you *must* resort to:
AM_TESTS_REEXEC=no zsh -o no_function_argzero foo.test
Note that this problem does not occur if Zsh is executed through
a symlink with a basename of `sh', since in that case Zsh starts
in Bourne compatibility mode. So you should be perfectly safe
when /bin/sh is Zsh, even a it's version < 4.3.
Reporting failures
------------------
Send verbose output, i.e., the contents of test-suite.log, of failing
tests to <bug-automake@gnu.org>, along with the usual version numbers
(which Automake, which Autoconf, which operating system, which make
version, which shell, etc.)
Writing test cases
==================
Do
--
If you plan to fix a bug, write the test case first. This way you'll
make sure the test catches the bug, and that it succeeds once you have
fixed the bug.
Add a copyright/license paragraph.
Explain what the test does.
Cite the PR number (if any), and the original reporter (if any), so
we can find or ask for information if needed.
Use `required=...' for required tools. Do not explicitly require
tools which can be taken for granted because they're listed in the
GNU Coding Standards (for example, `gzip').
Include ./defs in every test script (see existing tests for examples
of how to do this).
Use the `skip_' function to skip tests, with a meaningful message if
possible. Where convenient, use the `warn_' function to print generic
warnings, and the `fail_' function for test failures. Finally, you may
use the `framework_fail_' function for hard errors.
For tests that use the `parallel-tests' Automake option, set the shell
variable `parallel_tests' to "yes" before including ./defs. Also,
do not use for them a name that ends in `-p.test', since that would
risk to clash with automatically-generated tests. For tests that are
*not* meant to work with the `parallel-tests' Automake option (these
should be very very few), set the shell variable `parallel_tests' to
"no" before including ./defs.
./defs sets a skeleton configure.in. If possible, append to this
file. In some cases you'll have to overwrite it, but this should
be the exception. Note that configure.in registers Makefile.in
but do not output anything by default. If you need ./configure
to create Makefile, append AC_OUTPUT to configure.in.
By default, the testcases are run with the `errexit' shell flag on,
to make it easier to catch failures you might not have thought of.
If this is undesirable in some testcase, you can use `set +e' to
disable the `errexit' flag (but please do so only if you have a
very good reason).
End the test script with a `:' or `Exit 0'. Otherwise, when somebody
changes the test by adding a failing command after the last command,
the test will spuriously fail because $? is nonzero at the end. Note
that this is relevant even if the `errexit' shell flag is on, in case
the test contains commands like "grep ... Makefile.in && Exit 1" (and
there are indeed a lot of such tests).
Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
$PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
commands.
Use $sleep when you have to make sure that some file is newer
than another.
Use `cat' or `grep' to display (part of) files that may be interesting
for debugging, so that when a user send a verbose output we don't have
to ask him for more details. Display stderr output on the stderr file
descriptor. If some redirected command is likely to fail, display its
output even in the failure case, before exiting.
Use `Exit' rather than `exit' to abort a test.
Use `$PATH_SEPARATOR', not hard-coded `:', as the separator of
PATH's entries.
It's more important to make sure that a feature works, than
make sure that Automake's output looks correct. It might look
correct and still fail to work. In other words, prefer
running `make' over grepping `Makefile.in' (or do both).
If you run $AUTOMAKE or $AUTOCONF several times in the same test
and change `configure.in' by the meantime, do
rm -rf autom4te.cache
before the following runs. On fast machines the new `configure.in'
could otherwise have the same timestamp as the old `autom4te.cache'.
Alternatively, use `--force' for subsequent runs of the tools.
Use filenames with two consecutive spaces when testing that some
code preserves filenames with spaces. This will catch errors like
`echo $filename | ...`.
Before commit: make sure the test is executable, add the tests to
TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed,
write a ChangeLog entry, send the diff to <automake-patches@gnu.org>.
Do not
------
Do not test an Automake error with `$AUTOMAKE && Exit 1', or in three
years we'll discover that this test failed for some other bogus reason.
This happened many times. Better use something like
AUTOMAKE_fails
grep 'expected diagnostic' stderr
(Note this doesn't prevent the test from failing for another
reason, but at least it makes sure the original error is still
here.)
Do not override Makefile variables using make arguments, as in e.g.:
$MAKE DESTDIR=/foo/bar install
This is not portable for recursive targets (targets that call a
sub-make may not pass `DESTDIR=/foo/bar' along). Use the following
instead:
DESTDIR=/foo/bar $MAKE -e install
Do not send a test case without signing a copyright disclaimer.
See http://sources.redhat.com/automake/contribute.html or
ask <automake@gnu.org> for details.
|
