summaryrefslogtreecommitdiff
path: root/testsuite/driver/testglobals.py
Commit message (Collapse)AuthorAgeFilesLines
* Split off stat (benchmark) test failures into a separate section in the test ↵Gintautas Miliauskas2014-10-311-0/+2
| | | | | | | | | | | | | | | | | runner summary. Stat tests are generally less reliable than other types of tests, so it's nice to have them in a separate section rather than interspersed with potential... Summary: ...correctness issues. Reviewers: austin Reviewed By: austin Subscribers: thomie, carter, simonmar Differential Revision: https://phabricator.haskell.org/D406
* Fix #5435, adding new test config check_stdout.Edward Z. Yang2014-04-091-0/+5
| | | | | | | | | | | check_stdout(f) allows you to override the test framework's diff based output checking with another mechanism. f is a function which takes two arguments: the first is the filename containing the observed stdout, the second is the normaliser that would have been applied (in case you want to read, normalise, and then do something.) Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
* Flag to test suite: SKIP_PERF_TESTSJoachim Breitner2013-10-051-0/+3
| | | | | | More often than not the output of the performance tests is in the way, rather than helping. This allows the use of `make SKIP_PERF_TESTS=YES` to skip these tests. Fixes #8413
* Test driver: Implement different verbositiesJoachim Breitner2013-09-201-1/+1
| | | | | | | | | Select verbosity with "make VERBOSE=n". Options so far: n=0: No per-test output n=1: Only failing test results n=2: As above, plus progress information (names of all tests) n=3: As aobve, plus commands called. Default currently is n=3, although n=2 might be a nicer default.
* Test driver: Print total time spent at the endJoachim Breitner2013-09-201-1/+1
|
* Tweak the brokens list to include the directory the test is inIan Lynagh2013-02-111-1/+1
|
* Add 'make list_brokens'Ian Lynagh2013-02-111-0/+6
| | | | | | Gives a list of tickets that the testsuite thinks are broken, and what bug it thinks is the reason. This can then be pasted into trac and 'previewed', which will show any closed tickets with strikeout.
* Remove unused *_num_fields bindingsIan Lynagh2013-02-071-11/+0
|
* Add better support for .cmm test files.Austin Seipp2013-01-171-0/+3
| | | | | | Fixes Trac #7573. Signed-off-by: Austin Seipp <mad.one@gmail.com>
* Only run dynamicToo001 if we have both vanilla and dynamic librariesIan Lynagh2012-12-141-0/+3
|
* Testsuite fixes for when we only have dynamic librariesIan Lynagh2012-10-251-0/+3
|
* Add an outputdir test optionIan Lynagh2012-10-111-0/+3
| | | | | This makes it possible to share source files between tests, without having the .o/.hi files overlap
* Add support for per-test timeout adjustment (timeout_multiplier), and ↵Edward Z. Yang2012-09-251-0/+3
| | | | | | tighten up #367 test. Signed-off-by: Edward Z. Yang <ezyang@mit.edu>
* Add combined_output option.Paolo Capriotti2012-04-231-1/+3
| | | | | | | | | Added an option to combine stdout and stderr into a single file. This is useful for ghci scripts that produce interleaved errors and normal output. Also modified check_stderr_ok so that it normalizes stderr in the same way as compile tests.
* add req_smpSimon Marlow2012-02-161-0/+3
|
* Allow perf bounds to be specified as base + percentage deviationSimon Marlow2012-02-161-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | This makes it much easier to update the bounds. Instead of coming up with a suitable (min,max) pair, you just give e.g. (base, 10) to allow 10% deviation from the base figure, which can be pasted from the error. e.g. previously: - # expected value: 458700632 (amd64/Linux): - if_wordsize(64, - compiler_stats_num_field('bytes allocated', 440000000, - 480000000)), now: + if_wordsize(64, # sample from amd64/Linux 15/2/2012 + compiler_stats_range_field('bytes allocated', 360243576, 10)), Note: use stats_range_field rather than stats_num_field. I left support for the old way for now so that we can do a gradual migration. (next I suppose we should make it so that 'make accept' works for perf tests, but that's for another day)
* Make the compiler_always_flags modifiable on a per-test basisSimon Marlow2012-02-161-0/+5
| | | | Motivation: we wanted to disable -dcore-lint for the perf tests
* Count the number of tests that failed due to reqlib()Simon Marlow2012-01-061-0/+2
| | | | | | | | | | | | | | OVERALL SUMMARY for test run started at Fri Jan 6 09:46:46 GMT 2012 82 total tests, which gave rise to 820 test cases, of which 0 caused framework failures 818 were skipped 0 expected passes 2 had missing libraries <--- 0 expected failures 0 unexpected passes 0 unexpected failures
* Fix meaning of the only_ways field (fixed cgrun057(normal) in validate)Simon Marlow2011-11-161-4/+4
| | | | | | | | | | | | | The default setting for only_ways was [], which meant "all ways". However, it's really useful to have [] mean "no ways". In cgrun057 I used only_ways(prof_ways) to mean "only run this in the profiling ways", but if profiling is disabled then prof_ways = [] and this doesn't do what we want. Profiling is normally disabled in validate, but I had it enabled because I was testing profiling. So now only_ways defaults to None, which means "all ways", and [] now means "no ways".
* allow error message normalisation to be customisedSimon Marlow2011-11-091-0/+3
|
* generalise cmd_prefix to a general function to transform the commandSimon Marlow2011-10-181-2/+2
|
* Rejig how performance test stats are storedIan Lynagh2011-09-181-2/+2
| | | | | | In particular, this allows more specific results to be specified, e.g. if_wordsize(64, stats_num_field('foo', 50, 54)), if_platform('x86_64-apple-darwin', stats_num_field('foo', 62, 66)),
* Use True/False rather than 1/0 for opts.aloneIan Lynagh2011-08-081-1/+1
|
* Add support for checking whether files are written by more than one testIan Lynagh2011-08-071-0/+3
| | | | | | | | | | | | | | | | | | | | Work in progress, but largely works. Known issues: * only supported when using the timeout program * 'test.strace' files aren't cleaned, as they end up in the root directory rather than the test's directory * Doesn't yet track what the current directory is, so finds several files like "A.o" being written by multiple tests (and conversely, may be missing writes to the same file from different directories) * Lots of tests write to $HOME/.ghc/ghci_history. We should probably be passing ghci a flag to stop this from happening. * Some .strace lines aren't understood yet, causing framework failures * One .strace file can cause muiltiple framework failures, if it contains lots of lines that aren't understood Performance: Threads fast testsuite time fast testsuite time with checks 1 16:36.14 25:16.07 5 5:33.95 8:04.05
* Testcase for objective-c++ compilation (trac #5150)Peter Wortmann2011-08-061-3/+4
|
* Make and use a req_interp helperIan Lynagh2011-07-131-0/+3
| | | | | Tests can now specify that they require the interpreter (i.e. that they use one of ghci, annotations, TH, etc).
* Tests for trac #5025.Austin Seipp2011-04-031-2/+3
| | | | | Also teach the testsuite how to compile objective-c code with .m as a suffix.
* Fix the ffi002 testIan Lynagh2010-04-281-0/+3
|
* Add clean_cmd to the testsuite, and use it in bug1465Ian Lynagh2009-12-191-0/+3
|
* Add pre-command support to the testsuite, and fix annrun01 by using itIan Lynagh2009-12-191-0/+3
|
* Allow tests to behave differently depending on whether the compiler is in-treeIan Lynagh2009-12-181-0/+3
| | | | | And skip testwsdeque if it is not in-tree, as we rely on some headers from the build tree.
* Gather all tests at once, rather than doing them directory by directoryIan Lynagh2009-11-281-0/+3
| | | | | This increases the parallelism possible, and allows us to track what progress we are making.
* grab the target architecture from GHC, and add an if_arch() testSimon Marlow2009-11-241-0/+1
|
* add config.have_shared_libsSimon Marlow2009-11-041-0/+3
|
* Add support for compiler -t statsIan Lynagh2009-04-041-1/+2
|
* Improve the testsuite driver support for -t stats, and enhance space_leak_001Ian Lynagh2009-04-031-5/+7
|
* Add a space leak test, and some infrastructure for checking space usageIan Lynagh2008-11-231-0/+6
|
* Simplify, and correct, the code for determining whether we should use threadsIan Lynagh2008-06-121-1/+1
|
* Add if_os, and change derefnull/divbyzero to use itIan Lynagh2008-04-201-0/+1
| | | | | I think different Linux setups generate different output for these tests, so we just ignore the output on Linux now.
* Find compiler version-specific output files automaticallySimon Marlow2008-02-061-0/+1
| | | | | | | | | | | | | | Also, clean up the way we find the output file. From the comment: # Finding the sample output. The filename is of the form # # <test>.stdout[-<compiler>][-<version>][-<wordsize>][-<platform>] # # and we pick the most specific version available. The <version> is # the major version of the compiler (e.g. 6.8.2 would be "6.8"). For # more fine-grained control use if_compiler_lt(). I'll update the wiki too.
* allow extra normalisation to be applied to the output on a per-test basisSimon Marlow2008-01-221-0/+3
|
* Simplify the testsuite driverIan Lynagh2007-12-151-1/+1
| | | | | | | Now instead of saying, e.g. namebase_if_compiler_lt('ghc','6.9', 'foo-6.8') you say if_compiler_lt('ghc','6.9', namebase('foo-6.8'))
* remove the _with_prefix forms of compile_and_run, add cmd_prefix() config ↵Simon Marlow2007-11-221-0/+3
| | | | instead
* Add namebase_if_compiler_lt to the testsuite driverIan Lynagh2007-10-191-0/+4
| | | | | | | | Allows us to use a different name base for foo.stderr for old compilers, e.g. test('tc141', namebase_if_compiler_lt('ghc', '6.9', 'tc141-6.8'), compile_fail, [''])
* Allow files-to-cleanup to be attached to a test using extra_clean(files)Simon Marlow2007-09-131-2/+5
| | | | This is better than a top-level clean() when using threads
* clean up the extra_ways machinerySimon Marlow2007-06-271-0/+4
|
* Allow the "< /dev/null" when running tests to be disabledIan Lynagh2007-06-231-0/+3
| | | | | The user001 test in the unix package can fail when stdin comes from a file.
* add extra_ways, for adding running a test in more waysSimon Marlow2007-06-131-0/+3
|
* Allow tests to be skipped based on whether or not the compiler has a tagIan Lynagh2007-04-131-0/+4
| | | | | | When working on a new foo extension, you can now put your tests in the testsuite, set ProjectTags=-foo in mk/build.mk and skip_unless_tag('foo') in the tests.
* Calibrate the testsuite timeout if a value of -1 is givenIan Lynagh2007-04-051-0/+3
|
View Lorry Controller Status