summaryrefslogtreecommitdiff
path: root/testsuite/driver/testlib.py
Commit message (Collapse)AuthorAgeFilesLines
...
* Refactor testsuite with normalise_version()Edward Z. Yang2015-03-101-4/+16
| | | | | | | | | | | | | | | | | | | | Summary: This function generalizes the normaliseBytestringPackage and other similar one-off functions into normalise_version() with takes a package name to normalize against. This JUST manages package versions; we also could use a normalize for keys. In the process, I modified all the normalization functions to be accumulative; I don't think this makes a difference for current test cases but I think it makes things nicer. Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu> Reviewers: austin Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D725
* testsuite: format commands using config dictThomas Miedema2015-03-061-7/+17
| | | | | | | | | | | | | | | | | | | | | | Summary: Allow `cmd_wrapper` to return a format string that can refer to config values. Very useful! This allows for many tests to be defined in pure Python, instead of in an additional script or Makefile. Example: def Thpc(cmd): return(cmd + ' && {hpc} report Thpc.tix') test('Thpc', [cmd_wrapper(Thpc), only_ways['hpc']), compile_and_run, ['']) The `{hpc}` is replaced by the value of `config.hpc`. The result is that the module `Thpc` first gets compiled, then the binary `Thpc` is run, and then the `hpc report` command is run. The output of all of this is redirected (and later appended) to Thpc.run.stdout/stderr as normally. Reviewed By: austin Differential Revision: https://phabricator.haskell.org/D706
* Always ignore user-package-db when running testsThomas Miedema2015-02-231-2/+2
| | | | | | | | | | | | | | | | | | | | Summary: The user package database was already ignored for systems that `have_subprocess`. To [wiki:Debugging/InstallingPackagesInplace install] a package inplace: `cabal install --with-compiler=<inplace-ghc> --with-package-db=<inplace-package-db>` <package> Reviewers: austin Reviewed By: austin Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D668
* The test runner now also works under the msys-native Python.Gintautas Miliauskas2014-11-191-0/+16
| | | | | | | Msys binaries apply heuristics to escape paths in arguments intended for non-msys binaries, which breaks timeout invocations, see #9626. Signed-off-by: Austin Seipp <austin@well-typed.com>
* Split off stat (benchmark) test failures into a separate section in the test ↵Gintautas Miliauskas2014-10-311-8/+20
| | | | | | | | | | | | | | | | | runner summary. Stat tests are generally less reliable than other types of tests, so it's nice to have them in a separate section rather than interspersed with potential... Summary: ...correctness issues. Reviewers: austin Reviewed By: austin Subscribers: thomie, carter, simonmar Differential Revision: https://phabricator.haskell.org/D406
* Fix test driver python3 compatibility issuesMateusz Lenik2014-10-301-5/+5
| | | | | | | | | | | | | | | | | Summary: Fixes python3 compatibility issues by replacing filter with a list comperhension and a potential issue with python2 when override_flags would be an empty list. Reviewers: austin, thomie Reviewed By: austin, thomie Subscribers: thomie, carter, simonmar, mlen Differential Revision: https://phabricator.haskell.org/D399 GHC Trac Issues: #9230
* testlib: Get rid of two_normalisersJoachim Breitner2014-10-281-24/+31
| | | | | And make normalise_fun polyvariadic. After all, this is untyped code, so lets make use of it :-)
* Add n-ary version of `two_normalisers` to testsuite libHerbert Valerio Riedel2014-10-281-2/+25
| | | | This is more readable than nesting `two_normalisers()`-invocations
* Enabled warn on tabs by default (fixes #9230)Mateusz Lenik2014-10-211-7/+55
| | | | | | | | | | | | | | | | | | | | | | Summary: This revision enables -fwarn-tabs by default and add a suppression flag, so that GHC compilation won't fail when some files contain tab characters. Test Plan: Additional test case, T9230, was added to cover that change. Reviewers: austin Reviewed By: austin Subscribers: simonmar, ezyang, carter, thomie, mlen Differential Revision: https://phabricator.haskell.org/D255 GHC Trac Issues: #9230 Conflicts: testsuite/driver/testlib.py
* Python 3 support, second attempt (Trac #9184)Krzysztof Gogolewski2014-10-191-85/+92
| | | | | | | | | | | | | | | | | | | | | | | | Summary: This is a fixup of https://phabricator.haskell.org/D233 The only difference is in findTFiles (first commit), which previously broke Windows runner; now I translated literally instead attempting to improve it, and checked it works. Test Plan: I used validate under 2,3 on Linux and under 2 on msys2. On Windows I've seen a large number of failures, but they don't seem to be connected with the patch. Reviewers: hvr, simonmar, thomie, austin Reviewed By: austin Subscribers: thomie, carter, ezyang, simonmar Differential Revision: https://phabricator.haskell.org/D310 GHC Trac Issues: #9184
* Revert "Basic Python 3 support for testsuite driver (Trac #9184)"Krzysztof Gogolewski2014-10-031-92/+93
| | | | | | This reverts commit 084d241b316bfa12e41fc34cae993ca276bf0730. This is a possible culprit of Windows breakage reported at ghc-devs.
* Basic Python 3 support for testsuite driver (Trac #9184)Krzysztof Gogolewski2014-10-011-93/+92
| | | | | | | | | | | | | | | | | | | | | | | | Summary: Most of the changes is adaptation of old Python 2 only code. My priority was not breaking Python 2, and so I avoided bigger changes to the driver. In particular, under Python 3 the output is a str and buffering cannot be disabled. To test, define PYTHON=python3 in testsuite/mk/boilerplate.mk. Thanks to aspidites <emarshall85@gmail.com> who provided the initial patch. Test Plan: validate under 2 and 3 Reviewers: hvr, simonmar, thomie, austin Reviewed By: thomie, austin Subscribers: aspidites, thomie, simonmar, ezyang, carter Differential Revision: https://phabricator.haskell.org/D233 GHC Trac Issues: #9184
* testsuite: normalise integer library name for T8958Sergei Trofimovich2014-08-301-4/+2
| | | | Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
* testsuite: add signal_exit_code function to the driverKarel Gardas2014-08-071-0/+11
| | | | | | | | | | | | | | | | | | | Summary: New function signal_exit_code hides differences between target platforms handling of fatal error signals and the applications' exit codes. E.g. on Linux the application exit code which receives fatal error signal is encoded as 128 + signal value. On the other hand on Solaris the application exit code is signal value alone. Test Plan: validated on Linux and tested on Solaris Reviewers: austin, simonmar Reviewed By: simonmar Subscribers: phaskell, simonmar, relrod, ezyang, carter Differential Revision: https://phabricator.haskell.org/D122
* Correctly round when calculating the deviationJoachim Breitner2014-07-171-1/+1
|
* Include test case name in performance resultJoachim Breitner2014-07-171-9/+11
| | | | | | | With THREADS=n, for n > 1, it becomes impossible to match the performance numbers to the test case name. Hence include it in the output. This also makes grepping through a bunch of logs for a specific test case much easier, and outweighs the extra verbosity.
* Give performance benchmark deviation also in percentsJoachim Breitner2014-07-141-2/+6
| | | | | this makes it easier to spot a “just over the mark” change (e.g. +5.1%), compared to a more radical jump (e.g. +15%).
* New testsuite verbosity level 4Joachim Breitner2014-07-141-1/+1
| | | | | which makes it print performance numbers even when the test succeeds (good for historic analysis)
* Remove external coreAustin Seipp2014-05-031-95/+0
| | | | Signed-off-by: Austin Seipp <austin@well-typed.com>
* Fix #5435, adding new test config check_stdout.Edward Z. Yang2014-04-091-1/+15
| | | | | | | | | | | check_stdout(f) allows you to override the test framework's diff based output checking with another mechanism. f is a function which takes two arguments: the first is the filename containing the observed stdout, the second is the normaliser that would have been applied (in case you want to read, normalise, and then do something.) Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
* Fix some Python brainos in testlib (except e is not valid form).Edward Z. Yang2014-02-061-2/+2
| | | | Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
* Explain skipping compiler-perf tests when debugging is onJoachim Breitner2013-11-281-0/+2
|
* Summarily skip compiler_perf tests when DEBUG is onJoachim Breitner2013-11-281-0/+3
| | | | by ensuring this in the test runner.
* Flag to test suite: SKIP_PERF_TESTSJoachim Breitner2013-10-051-0/+6
| | | | | | More often than not the output of the performance tests is in the way, rather than helping. This allows the use of `make SKIP_PERF_TESTS=YES` to skip these tests. Fixes #8413
* Driver: Fix % operator typoJoachim Breitner2013-09-231-1/+1
|
* Test driver: Implement different verbositiesJoachim Breitner2013-09-201-18/+20
| | | | | | | | | Select verbosity with "make VERBOSE=n". Options so far: n=0: No per-test output n=1: Only failing test results n=2: As above, plus progress information (names of all tests) n=3: As aobve, plus commands called. Default currently is n=3, although n=2 might be a nicer default.
* Test driver: Print total time spent at the endJoachim Breitner2013-09-201-1/+6
|
* Make test cleaning a little more consistentIan Lynagh2013-05-171-4/+6
|
* Add a multi_cpu_race helperIan Lynagh2013-04-171-0/+5
| | | | | This makes the test run alone, to increase the chance of a multi-CPU race happening.
* Fix tests for dynamic ghcIan Lynagh2013-03-151-1/+1
|
* Remove composeIan Lynagh2013-02-171-3/+0
|
* Remove uses of compose(s) in tests, and change how composition is handledIan Lynagh2013-02-141-13/+10
| | | | The driver now also supports nested lists of setup functions
* Reorder some functions to group themIan Lynagh2013-02-131-18/+17
| | | | Predicates now match the order they are documented on the wiki
* Tweak the brokens list to include the directory the test is inIan Lynagh2013-02-111-4/+4
|
* More conversionsIan Lynagh2013-02-111-6/+3
|
* More helper conversionsIan Lynagh2013-02-111-6/+3
|
* Convert more helpersIan Lynagh2013-02-111-11/+2
|
* Convert more helper functionsIan Lynagh2013-02-111-61/+22
|
* Convert more helper functions to the new schemeIan Lynagh2013-02-111-33/+6
|
* Convert more helpers to the new schemeIan Lynagh2013-02-111-21/+6
|
* Change '{if,unless}_arch' to 'arch'Ian Lynagh2013-02-111-11/+2
|
* Handle duplicates in the brokens listIan Lynagh2013-02-111-4/+8
|
* expect_broken_for should also be added to the list of brokensIan Lynagh2013-02-111-8/+10
|
* Replace 'if_os' and 'unless_os' with 'opsys'Ian Lynagh2013-02-111-11/+2
| | | | The name 'os' clashes with a python library
* Add 'make list_brokens'Ian Lynagh2013-02-111-1/+5
| | | | | | Gives a list of tickets that the testsuite thinks are broken, and what bug it thinks is the reason. This can then be pasted into trac and 'previewed', which will show any closed tickets with strikeout.
* Rearrange the summary outputIan Lynagh2013-02-101-16/+18
| | | | | | In particular, the 3 values you most need to care about (framework failures, unexpected passes, unexpected failures) are now together in a block.
* Remove unused helper functionsIan Lynagh2013-02-081-6/+0
|
* Replace 'if_platform' and 'unless_platform' with 'platform'Ian Lynagh2013-02-081-11/+2
|
* Allow a simpler form of stats_num_field where all platforms use the same valueIan Lynagh2013-02-071-5/+9
|
* Add a different sort of stats_num_field helper functionIan Lynagh2013-02-071-7/+29
| | | | | | | | | | | | | | | | Uses look like stats_num_field('bytes allocated', [(wordsize(32), 45648, 5), (wordsize(64), 49400, 5)]) where the first matching triple will be used. e.g. we could override the Win32 expected values with: [(platform('i386-unknown-mingw32'), 41000, 5), (wordsize(32), 45648, 5), (wordsize(64), 49400, 5)]) with other 32-bit platforms falling through to the wordsize(32) case. This makes it easier to give different values for different platforms, while being sure that all platforms are covered.
View Lorry Controller Status