summaryrefslogtreecommitdiff
path: root/testsuite/driver/testglobals.py
Commit message (Collapse)AuthorAgeFilesLines
* Testsuite: implement use_specs.Tamar Christina2019-02-171-0/+3
|
* Fix and Reapply "Performance tests: recover a baseline from ancestor commits ↵David Eichmann2019-02-161-3/+13
| | | | and CI results."
* testsuite: Report stdout and stderr in JUnit outputBen Gamari2019-02-101-0/+15
| | | | | | | | | | This patch makes the JUnit output more useful as now we also report the stdout/stderr in the message which can be used to quickly identify why a test is failing without downloading the log. This also introduces TestResult, previously we were simply passing around tuples, making things the implementation rather difficult to follow and harder to extend.
* Revert "Performance tests: recover a baseline from ancestor commits and CI ↵Ben Gamari2019-01-311-9/+3
| | | | | | | | | results." Unfortunately this has broken all future commits due to spurious(?) performance changes which I have been unable to work around. This reverts commit cc2261d42f6a954d88e355aaad41f001f65c95da.
* Performance tests: recover a baseline from ancestor commits and CI results.David Eichmann2019-01-301-3/+9
| | | | gitlab-ci: push performance metrics as git notes to the "GHC Performance Notes" repository.
* Revert "Batch merge"Ben Gamari2019-01-301-9/+3
| | | | This reverts commit 76c8fd674435a652c75a96c85abbf26f1f221876.
* Batch mergeBen Gamari2019-01-301-3/+9
|
* testsuite: Ensure that config.{msys,cygwin} are initializedBen Gamari2019-01-231-0/+2
| | | | | | | | | | Reviewers: monoidal Reviewed By: monoidal Subscribers: monoidal, rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5056
* testsuite: Save performance metrics in git notes.David Eichmann2018-11-071-9/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes the following improvement: - Automatically records test metrics (per test environment) so that the programmer need not supply nor update expected values in *.T files. - On expected metric changes, the programmer need only indicate the direction of change in the git commit message. - Provides a simple python tool "perf_notes.py" to compare metrics over time. Issues: - Using just the previous commit allows performance to drift with each commit. - Currently we allow drift as we have a preference for minimizing false positives. - Some possible alternatives include: - Use metrics from a fixed commit per test: the last commit that allowed a change in performance (else the oldest metric) - Or use some sort of aggregate since the last commit that allowed a change in performance (else all available metrics) - These alternatives may result in a performance issue (with the test driver) having to heavily search git commits/notes. - Run locally, performance tests will trivially pass unless the tests were run locally on the previous commit. This is often not the case e.g. after pulling recent changes. Previously, *.T files contain statements such as: ``` stats_num_field('peak_megabytes_allocated', (2, 1)) compiler_stats_num_field('bytes allocated', [(wordsize(64), 165890392, 10)]) ``` This required the programmer to give the expected values and a tolerance deviation (percentage). With this patch, the above statements are replaced with: ``` collect_stats('peak_megabytes_allocated', 5) collect_compiler_stats('bytes allocated', 10) ``` So that programmer must only enter which metrics to test and a tolerance deviation. No expected value is required. CircleCI will then run the tests per test environment and record the metrics to a git note for that commit and push them to the git.haskell.org ghc repo. Metrics will be compared to the previous commit. If they are different by the tolerance deviation from the *.T file, then the corresponding test will fail. By adding to the git commit message e.g. ``` # Metric (In|De)crease <metric(s)> <options>: <tests> Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \ (test_env='linux_x86', way='default'): Test012, Test345 Metric Decrease 'bytes allocated': Test678 Metric Increase: Test711 ``` This will allow the noted changes (letting the test pass). Note that by omitting metrics or options, the change will apply to all possible metrics/options (i.e. in the above, an increase for all metrics in all test environments is allowed for Test711) phabricator will use the message in the description Reviewers: bgamari, hvr Reviewed By: bgamari Subscribers: rwbarton, carter GHC Trac Issues: #12758 Differential Revision: https://phabricator.haskell.org/D5059
* testsuite: Use bools for booleans, not intsBen Gamari2018-09-051-11/+11
| | | | | | | | | | | | | | Summary: Just as it says on the tin. Test Plan: Validate Reviewers: bgamari, osa1 Reviewed By: osa1 Subscribers: osa1, monoidal, rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D5010
* Simplify testsuite driverKrzysztof Gogolewski2018-08-111-6/+0
| | | | | | | | | | | | | | | | | | | Summary: - remove clean_cmd - framework_failures was undefined - times_file was not used - if_verbose_dump was called only when verbose >= 1; remove the check - simplify normalise_whitespace Test Plan: validate Reviewers: bgamari, thomie Reviewed By: thomie Subscribers: rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5061
* Correctly add unwinding info in manifestSp and makeFixupBlocksBartosz Nitka2018-05-031-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | In `manifestSp` the unwind info was before the relevant instruction, not after. I added some notes to establish semantics. Also removes redundant annotation in stg_catch_frame. For `makeFixupBlocks` it looks like we were off by `wORD_SIZE dflags`. I'm not sure why, but it lines up with `manifestSp`. In fact it lines up so well so that I can consolidate the Sp unwind logic in `maybeAddUnwind`. I detected the problems with `makeFixupBlocks` by running T14779b after patching D4559. Test Plan: added a new test Reviewers: bgamari, scpmw, simonmar, erikd Reviewed By: bgamari Subscribers: thomie, carter GHC Trac Issues: #14999 Differential Revision: https://phabricator.haskell.org/D4606
* testsuite: allow accepting of fine grained results [skip ci]Tamar Christina2018-03-311-4/+6
| | | | | | | | | | | | | | | | | | | | | | Summary: Sometimes we need to be able to mass accept changes that are platform specific and that can't be normalized away using our string formatters. e.g. differences in I/O manager errors or behaviors. This allows one to do so easier than before and less error prone. I have updated the docs and made it clear this should only be used when a normalizer won't work: https://ghc.haskell.org/trac/ghc/wiki/Building/RunningTests/Updating Test Plan: Manually tested while working on new I/O manager Reviewers: bgamari Subscribers: thomie, carter Differential Revision: https://phabricator.haskell.org/D4549
* Windows: fix all failing tests.Tamar Christina2018-01-021-0/+6
| | | | | | | | | | | | | | | | | | | | | | This makes the testsuite pass clean on Windows again. It also fixes the `libstdc++-6.dll` error harbormaster was showing. I'm marking some tests as isolated tests to reduce their flakiness (mostly concurrency tests) when the test system is under heavy load. Updates process submodule. Test Plan: ./validate Reviewers: hvr, bgamari, erikd, simonmar Reviewed By: bgamari Subscribers: rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D4277
* testsuite: Produce JUnit outputBen Gamari2017-07-281-0/+1
| | | | | | | | | | | | Test Plan: Validate, try ingesting into Jenkins. Reviewers: austin Subscribers: rwbarton, thomie GHC Trac Issues: #13716 Differential Revision: https://phabricator.haskell.org/D3796
* testsuite: Classify missing expected perf numbers as merely warningsBen Gamari2017-04-021-0/+1
| | | | | | Previously these were considered to be framework failures, meaning that validate would fail. For better or worse, Windows lacks a good number of metrics and I don't see this changing any time soon. Let's consider these to be non-fatal.
* The Backpack patch.Edward Z. Yang2016-10-081-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This patch implements Backpack for GHC. It's a big patch but I've tried quite hard to keep things, by-in-large, self-contained. The user facing specification for Backpack can be found at: https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst A guide to the implementation can be found at: https://github.com/ezyang/ghc-proposals/blob/backpack-impl/proposals/0000-backpack-impl.rst Has a submodule update for Cabal, as well as a submodule update for filepath to handle more strict checking of cabal-version. Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu> Test Plan: validate Reviewers: simonpj, austin, simonmar, bgamari, goldfire Subscribers: thomie, mpickering Differential Revision: https://phabricator.haskell.org/D1482
* Testsuite: use ignore_stderr/stdout instead of ignore_outputThomas Miedema2016-06-291-1/+5
| | | | | | | | | | | | | | | | | The problem with ignore_output is that it hides errors for WAY=ghci. GHCi always returns with exit code 0 (unless it is broken itself). For example: ghci015 must have been failing with compile errors for years, but we didn't notice because all output was ignored. Therefore, replace all uses of ignore_output with either ignore_stderr or ignore_stdout. In some cases I opted for adding the expected output. Update submodule hpc and stm. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2367
* Testsuite: cleanup printing of summaryThomas Miedema2016-06-281-14/+9
| | | | | | Just use a simple list of tuples, instead of a nested map. -90 lines of code.
* Testsuite: open/close stdin/stdout/stderr explicitlyThomas Miedema2016-06-281-3/+0
| | | | | | | | | | | | | | | | | | This allows run_command's to contain `|`, and `no_stdin` isn't necessary anymore. Unfortunately it doesn't fix T7037 on Windows which I had hoped it would (testsuite driver tries to read a file that it just created itself, but the OS says it doesn't exist). The only drawback of this commit is that the command that the testsuite prints to the terminal (for debugging purposes) doesn't mention the files that stdout and stderr are redirected to anymore. This is probably ok. Update submodule unix. Differential Revision: https://phabricator.haskell.org/D1234
* Testsuite: delete dead code + cleanupThomas Miedema2016-06-201-7/+0
| | | | | | * Set config settings directly in mk/test.mk, instead of indirectly in config/ghc * passing --hpcdir for WAY=hpc is unnecessary
* Testsuite: validate the tests/stage1 directory with the stage1 compilerThomas Miedema2016-06-181-1/+4
| | | | | | | | | | | * See `Note [Why is there no stage1 setup function?]`. * Move T2632 to the tests/stage1 directory (#10382). Reviewed by: ezyang, nomeata, bgamari Differential Revision: https://phabricator.haskell.org/D2341 GHC Trac Issues: #12197
* Testsuite: run tests in <testdir>.run instead of /tmpThomas Miedema2016-06-181-0/+6
| | | | | | | | | | | | | | | | | | | | As discussed in Phab:D1187, this approach makes it a bit easier to inspect the test directory while working on a new test. The only tests that needed changes are the ones that refer to files in ancestor directories. Those files are now copied directly into the test directory. validate still runs the tests in a temporary directory in /tmp, see `Note [Running tests in /tmp]` in testsuite/driver/runtests.py. Update submodule hpc. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2333 GHC Trac Issues: #11980
* Testsuite: delete check_files_writtenThomas Miedema2016-05-241-3/+0
| | | | | | | | | The CHECK_FILES_WRITTEN feature is no longer necessary, since tests don't write to the source directory anymore (#11980). Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2162
* Testsuite: run tests in /tmp after copying required filesThomas Miedema2016-05-171-0/+3
| | | | | | | | | | | | | | | | | | | | | | | Major change to the testsuite driver. For each TEST: * create a directory `<testdir>` inside `/tmp`. * link/copy all source files that the test needs into `<testdir>`. * run the test inside `<testdir>`. * delete `<testdir>` Extra files are (temporarily) tracked in `testsuite/driver/extra_files.py`, but can also be specified using the `extra_files` setup function. Differential Revision: https://phabricator.haskell.org/D1187 Reviewed by: Rufflewind, bgamari Trac: #11980
* Testsuite: make CLEANUP=1 the default (#9758)Thomas Miedema2016-04-301-3/+0
| | | | | | | | | | Also move the `cleanup` setting from `default_testopts` to `config`. The `cleanup` setting is the same for all tests, hence it belongs in `config`. Reviewed by: austin Differential Revision: https://phabricator.haskell.org/D2148
* Testsuite: delete -fesc testsThomas Miedema2016-04-281-1/+0
| | | | | | | | | | | The -fesc flag does not exist, and has never existed. Also delete now unused config.compiler_tags, and 'Project version' never contains a '-'. Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2138
* Filter out -prof callstacks from test output (#11521)Thomas Miedema2016-02-231-0/+3
|
* Testsuite: report and error out on unfound testsThomas Miedema2015-10-291-1/+2
| | | | | | | | | | | | | | | | | | | Users are sometimes confused why their test doesn't run. It is usually because of a misspelled testname, for example using 'TEST=1234' instead of 'TEST=T1234'. After this patch it is hopefully more clear what the problem is, showing: ERROR: tests not found: ['1234'] Instead of: 0 total tests, which gave rise to 0 test cases, of which 0 were skipped Reviewed by: austin, bgamari Differential Revision: https://phabricator.haskell.org/D1388
* Testsuite: by default run all tests for a single wayThomas Miedema2015-09-021-2/+2
| | | | | | | | | | | | | | `make test` now runs all tests for a single way only. Use `make slowtest` to get the previous behaviour (i.e. run all tests for all ways). The intention is to use this new `make test` setting for Phabricator, as a reasonable compromise between `make fasttest` (what it previously used) and a fullblown `make slowtest` (which runs all tests for all ways). See Note [validate and testsuite speed] in toplevel Makefile. Differential Revision: https://phabricator.haskell.org/D1178
* Testsuite: delete remaining only_compiler_types(['ghc']) setupsThomas Miedema2015-07-141-3/+0
| | | | | No point in pretending other compilers can use the GHC testsuite. This makes the *.T files a bit shorter.
* Testsuite: delete unused with_namebaseThomas Miedema2015-07-141-4/+0
| | | | It was introduced in 39c6c735c216d259854ee31b15ec87ea653f2b5d (2007).
* Testsuite: add function compile_timeout_multiplier (#10345)Thomas Miedema2015-06-131-1/+2
| | | | | | | | | | | And rename timeout_multiplier to run_timeout_multiplier. timeout_multiplier was added in commit a00389794b839971c7d52ead9e8570bfaa25ac55. The name suggested that it would affect any test, but it actually only affected tests that had a run component, and only that run component (as needed by test T367). Differential Revision: https://phabricator.haskell.org/D982
* Make validate more quietThomas Miedema2015-06-041-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * By default use V=0, and call the testsuite with VERBOSE=2, which we did before only with validate --quiet. This disables printing the test commands it runs. * When --quiet is used, call the testsuite with VERBOSE=1. This disables printing the '====> Scanning' lines, and doesn't print which test is being run. So it only prints something when a test accidentally prints to stdout or when it fails. Don't set this option on Travis, as Travis will cancel a build if it doesn't see any output for more than 10 minutes. * When --quiet is used, set the new test option NO_PRINT_SUMMARY, which skips printing the test summary. Only the list of unexpected failures is printed, if there are any. Note that the full summary can still be found in testsuite_summary.txt * When --quiet is used, don't pass the `-v` flag to `ghc-pkg check` * When --quiet is used, don't print the Oops! header. It shoud be clear from the list of failing tests that something is wrong. This is all done to get the most out of 30 lines of logfile. These changes can be disabled later by simply not passing the --quiet flag to validate. Differential Revision: https://phabricator.haskell.org/D942
* Split off stat (benchmark) test failures into a separate section in the test ↵Gintautas Miliauskas2014-10-311-0/+2
| | | | | | | | | | | | | | | | | runner summary. Stat tests are generally less reliable than other types of tests, so it's nice to have them in a separate section rather than interspersed with potential... Summary: ...correctness issues. Reviewers: austin Reviewed By: austin Subscribers: thomie, carter, simonmar Differential Revision: https://phabricator.haskell.org/D406
* Fix #5435, adding new test config check_stdout.Edward Z. Yang2014-04-091-0/+5
| | | | | | | | | | | check_stdout(f) allows you to override the test framework's diff based output checking with another mechanism. f is a function which takes two arguments: the first is the filename containing the observed stdout, the second is the normaliser that would have been applied (in case you want to read, normalise, and then do something.) Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
* Flag to test suite: SKIP_PERF_TESTSJoachim Breitner2013-10-051-0/+3
| | | | | | More often than not the output of the performance tests is in the way, rather than helping. This allows the use of `make SKIP_PERF_TESTS=YES` to skip these tests. Fixes #8413
* Test driver: Implement different verbositiesJoachim Breitner2013-09-201-1/+1
| | | | | | | | | Select verbosity with "make VERBOSE=n". Options so far: n=0: No per-test output n=1: Only failing test results n=2: As above, plus progress information (names of all tests) n=3: As aobve, plus commands called. Default currently is n=3, although n=2 might be a nicer default.
* Test driver: Print total time spent at the endJoachim Breitner2013-09-201-1/+1
|
* Tweak the brokens list to include the directory the test is inIan Lynagh2013-02-111-1/+1
|
* Add 'make list_brokens'Ian Lynagh2013-02-111-0/+6
| | | | | | Gives a list of tickets that the testsuite thinks are broken, and what bug it thinks is the reason. This can then be pasted into trac and 'previewed', which will show any closed tickets with strikeout.
* Remove unused *_num_fields bindingsIan Lynagh2013-02-071-11/+0
|
* Add better support for .cmm test files.Austin Seipp2013-01-171-0/+3
| | | | | | Fixes Trac #7573. Signed-off-by: Austin Seipp <mad.one@gmail.com>
* Only run dynamicToo001 if we have both vanilla and dynamic librariesIan Lynagh2012-12-141-0/+3
|
* Testsuite fixes for when we only have dynamic librariesIan Lynagh2012-10-251-0/+3
|
* Add an outputdir test optionIan Lynagh2012-10-111-0/+3
| | | | | This makes it possible to share source files between tests, without having the .o/.hi files overlap
* Add support for per-test timeout adjustment (timeout_multiplier), and ↵Edward Z. Yang2012-09-251-0/+3
| | | | | | tighten up #367 test. Signed-off-by: Edward Z. Yang <ezyang@mit.edu>
* Add combined_output option.Paolo Capriotti2012-04-231-1/+3
| | | | | | | | | Added an option to combine stdout and stderr into a single file. This is useful for ghci scripts that produce interleaved errors and normal output. Also modified check_stderr_ok so that it normalizes stderr in the same way as compile tests.
* add req_smpSimon Marlow2012-02-161-0/+3
|
* Allow perf bounds to be specified as base + percentage deviationSimon Marlow2012-02-161-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | This makes it much easier to update the bounds. Instead of coming up with a suitable (min,max) pair, you just give e.g. (base, 10) to allow 10% deviation from the base figure, which can be pasted from the error. e.g. previously: - # expected value: 458700632 (amd64/Linux): - if_wordsize(64, - compiler_stats_num_field('bytes allocated', 440000000, - 480000000)), now: + if_wordsize(64, # sample from amd64/Linux 15/2/2012 + compiler_stats_range_field('bytes allocated', 360243576, 10)), Note: use stats_range_field rather than stats_num_field. I left support for the old way for now so that we can do a gradual migration. (next I suppose we should make it so that 'make accept' works for perf tests, but that's for another day)
View Lorry Controller Status