summaryrefslogtreecommitdiff
path: root/testsuite/driver
Commit message (Collapse)AuthorAgeFilesLines
* Fix colorise header as wellMatthew Pickering2019-02-201-0/+1
|
* Exit with exit code 1 when tests unexpectedly passMatthew Pickering2019-02-191-0/+1
|
* Testsuite: implement use_specs.Tamar Christina2019-02-172-0/+29
|
* Fix and Reapply "Performance tests: recover a baseline from ancestor commits ↵David Eichmann2019-02-165-65/+277
| | | | and CI results."
* testsuite: Report unexpected passes in junit outputMatthew Pickering2019-02-111-2/+5
|
* testsuite: Report stdout and stderr in JUnit outputBen Gamari2019-02-104-34/+69
| | | | | | | | | | This patch makes the JUnit output more useful as now we also report the stdout/stderr in the message which can be used to quickly identify why a test is failing without downloading the log. This also introduces TestResult, previously we were simply passing around tuples, making things the implementation rather difficult to follow and harder to extend.
* testsuite: Mark print037 as broken when GHC is built with LLVMBen Gamari2019-02-041-0/+9
| | | | As noted in #16205 this configuration reliably segfaults.
* Revert "Performance tests: recover a baseline from ancestor commits and CI ↵Ben Gamari2019-01-315-259/+62
| | | | | | | | | results." Unfortunately this has broken all future commits due to spurious(?) performance changes which I have been unable to work around. This reverts commit cc2261d42f6a954d88e355aaad41f001f65c95da.
* Performance tests: recover a baseline from ancestor commits and CI results.David Eichmann2019-01-305-62/+259
| | | | gitlab-ci: push performance metrics as git notes to the "GHC Performance Notes" repository.
* testsuite: Use makefile_testBen Gamari2019-01-301-1/+1
| | | | | This eliminates most uses of run_command in the testsuite in favor of the more structured makefile_test.
* testsuite: Introduce makefile_testBen Gamari2019-01-301-0/+7
|
* Revert "Batch merge"Ben Gamari2019-01-305-267/+63
| | | | This reverts commit 76c8fd674435a652c75a96c85abbf26f1f221876.
* Batch mergeBen Gamari2019-01-305-63/+267
|
* testsuite: Remove directories that already exist when seeding extra_filesBen Gamari2019-01-271-0/+2
| | | | | Otherwise the testsuite driver crashes when run multiple times with CLEANUP=NO on a test containing such extra_files.
* testsuite: Add predicate for CPU feature availabilityBen Gamari2019-01-273-0/+79
| | | | | | | Previously testing code-generation for ISA extensions was nearly impossible since we had no ability to determine whether the host supports the needed extension. Here we fix this by introducing a simple /proc/cpuinfo-based testsuite predicate. We really ought to
* testsuite: Ensure that config.{msys,cygwin} are initializedBen Gamari2019-01-231-0/+2
| | | | | | | | | | Reviewers: monoidal Reviewed By: monoidal Subscribers: monoidal, rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5056
* Fix tests for `integer-simple`Alec Theriault2019-01-161-3/+0
| | | | | | | | | | | | A bunch of tests for `integer-simple` were now broken for a foolish reason: unlike the `integer-gmp` case, there is no CorePrep optimization for turning small integers directly into applications of `S#`. Rather than port this optimization to `integer-simple` (which would involve moving a bunch of `integer-simple` names into `PrelNames`), I switched as many tests as possible to use `Int`. The printing of `Integer` is already tested in `print037`.
* testsuite: Show both test name and way in JUnit outputBen Gamari2019-01-141-6/+6
|
* Introduce NCG config flag and add helperGabor Greif2018-12-301-1/+4
| | | | | | | ... for testing presence of NCG This commit adds a criterion for checking whether we can expect sensible output from --ddump-asm.
* testsuite: Fix a variety of issues when building with integer-simpleBen Gamari2018-12-241-12/+14
| | | | | | | | | | | | | | | | | | | | | | | | | * Mark arith011 as broken with integer-simple As noted in #16091, arith011 fails when run against integer-simple with a "divide by zero" exception. This suggests that integer-gmp and integer-simple are handling division by zero differently. * This also fixes broken_without_gmp; the lack of types made the previous failure silent, sadly. Improves situation of #16043. * Mark several tests implicitly depending upon integer-gmp as broken with integer-simple. These expect to see Core coming from integer-gmp, which breaks with integer-simple. * Increase runtime timeout multiplier of T11627a with integer-simple I previously saw that T11627a timed out in all profiling ways when run against integer-simple. I suspect this is due to integer-simple's rather verbose heap representation. Let's see whether increasing the runtime timeout helps. Fixes test for #11627. This is all in service of fixing #16043.
* Revert "testsuite: Fix broken_without_gmp"Ben Gamari2018-12-231-4/+5
| | | | | | | This reverts commit e59439af3222d151918ad1ad2a03942ce9e6a1ff. This is causing unexpected failures in some test ways. Further proof that no change is too trivial for CI.
* testsuite: Fix broken_without_gmpBen Gamari2018-12-231-5/+4
| | | | | The lack of types made the previous failure silent, sadly. Improves situation of #16043.
* testsuite: Fix typo: integer-gimp /= integer-gmpBen Gamari2018-12-131-1/+1
|
* testsuite: Normalise away spurious differences in out-of-scope instancesBen Gamari2018-12-121-1/+7
| | | | | | | | | | | | | | | | | | | | This fixes a variety of testsuite failures with integer-simple of the form ``` --- typecheck/should_fail/tcfail072.run/tcfail072.stderr.normalised +++ typecheck/should_fail/tcfail072.run/tcfail072.comp.stderr.normalised @@ -12,7 +12,7 @@ -- Defined in ‘integer-<IMPL>-<VERSION>:GHC.Integer.Type’ instance Ord () -- Defined in ‘GHC.Classes’ ...plus 21 others - ...plus three instances involving out-of-scope types + ...plus two instances involving out-of-scope types (use -fprint-potential-instances to see them all) In the expression: g A In an equation for ‘g’: g (B _ _) = g A ``` In service of fixing #16043.
* testsuite: Fix a number of GHCi-related failures due to integer-simpleBen Gamari2018-12-121-0/+5
| | | | Towards fixing #16043.
* testsuite: Print which ways we are going to runBen Gamari2018-12-121-1/+4
|
* Do not save performance test results if worktree is dirty.David Eichmann2018-12-112-2/+11
| | | | | | | | | | | | Reviewers: bgamari, tdammers Reviewed By: bgamari, tdammers Subscribers: rwbarton, carter GHC Trac Issues: #15924 Differential Revision: https://phabricator.haskell.org/D5368
* testsuite: Don't use git status to determine whether we are inside a repoBen Gamari2018-12-012-7/+8
| | | | | Git status is extremely expensive for this task. We instead use `git rev-parse HEAD` and throw away the output to ensure we don't spam the user.
* Skip all performance tests if not in a git repo.David Eichmann2018-11-302-6/+35
| | | | | | | | | | | | Reviewers: bgamari, tdammers, osa1 Reviewed By: tdammers Subscribers: osa1, tdammers, rwbarton, carter GHC Trac Issues: #15923 Differential Revision: https://phabricator.haskell.org/D5367
* testuite: update more windows tests outputsTamar Christina2018-11-221-2/+3
| | | | | | | | | | Test Plan: ./validate Reviewers: bgamari, simonmar Subscribers: rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5362
* Fix GhostScript detection (Trac #15856)Krzysztof Gogolewski2018-11-221-2/+2
| | | | | | | | | | | | | | | | | | | | | | | The option `confdir` (used in GhostScript test) was set correctly via `--config` in `test.mk` and incorrectly via `config/ghc`. AFAICT, some time ago this was working because the incorrect assignment was done first, and later it broke. Hardian doesn't pass `confdir`. I removed `confdir` and use `config.top` to determine the directory of the `good.ps` and `bad.ps` files. This is simpler. I also removed some redundant assignments in `config/ghc`. Test Plan: manually set config.have_profiling and make test Reviewers: bgamari Subscribers: rwbarton, carter GHC Trac Issues: #15856 Differential Revision: https://phabricator.haskell.org/D5298
* testsuite: Save performance metrics in git notes.David Eichmann2018-11-076-105/+764
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes the following improvement: - Automatically records test metrics (per test environment) so that the programmer need not supply nor update expected values in *.T files. - On expected metric changes, the programmer need only indicate the direction of change in the git commit message. - Provides a simple python tool "perf_notes.py" to compare metrics over time. Issues: - Using just the previous commit allows performance to drift with each commit. - Currently we allow drift as we have a preference for minimizing false positives. - Some possible alternatives include: - Use metrics from a fixed commit per test: the last commit that allowed a change in performance (else the oldest metric) - Or use some sort of aggregate since the last commit that allowed a change in performance (else all available metrics) - These alternatives may result in a performance issue (with the test driver) having to heavily search git commits/notes. - Run locally, performance tests will trivially pass unless the tests were run locally on the previous commit. This is often not the case e.g. after pulling recent changes. Previously, *.T files contain statements such as: ``` stats_num_field('peak_megabytes_allocated', (2, 1)) compiler_stats_num_field('bytes allocated', [(wordsize(64), 165890392, 10)]) ``` This required the programmer to give the expected values and a tolerance deviation (percentage). With this patch, the above statements are replaced with: ``` collect_stats('peak_megabytes_allocated', 5) collect_compiler_stats('bytes allocated', 10) ``` So that programmer must only enter which metrics to test and a tolerance deviation. No expected value is required. CircleCI will then run the tests per test environment and record the metrics to a git note for that commit and push them to the git.haskell.org ghc repo. Metrics will be compared to the previous commit. If they are different by the tolerance deviation from the *.T file, then the corresponding test will fail. By adding to the git commit message e.g. ``` # Metric (In|De)crease <metric(s)> <options>: <tests> Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \ (test_env='linux_x86', way='default'): Test012, Test345 Metric Decrease 'bytes allocated': Test678 Metric Increase: Test711 ``` This will allow the noted changes (letting the test pass). Note that by omitting metrics or options, the change will apply to all possible metrics/options (i.e. in the above, an increase for all metrics in all test environments is allowed for Test711) phabricator will use the message in the description Reviewers: bgamari, hvr Reviewed By: bgamari Subscribers: rwbarton, carter GHC Trac Issues: #12758 Differential Revision: https://phabricator.haskell.org/D5059
* testsuite: Use bools for booleans, not intsBen Gamari2018-09-052-26/+26
| | | | | | | | | | | | | | Summary: Just as it says on the tin. Test Plan: Validate Reviewers: bgamari, osa1 Reviewed By: osa1 Subscribers: osa1, monoidal, rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D5010
* Simplify testsuite driver, part 2Krzysztof Gogolewski2018-08-122-14/+13
| | | | | | | | | | | | | | | | | | Summary: - Avoid import *; this helps tools such as pyflakes. The last occurrence in runtests.py is not easy to remove as it's used by .T files. - Use False/True instead of 0/1. Test Plan: validate Reviewers: bgamari, thomie, simonmar Reviewed By: thomie Subscribers: rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5062
* Simplify testsuite driverKrzysztof Gogolewski2018-08-113-22/+9
| | | | | | | | | | | | | | | | | | | Summary: - remove clean_cmd - framework_failures was undefined - times_file was not used - if_verbose_dump was called only when verbose >= 1; remove the check - simplify normalise_whitespace Test Plan: validate Reviewers: bgamari, thomie Reviewed By: thomie Subscribers: rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5061
* Testsuite driver: fix encoding issue when calling ghc-pkgKrzysztof Gogolewski2018-08-062-4/+3
| | | | | | | | | | | | | | | | Summary: In Python 3, subprocess.communicate() returns a pair of bytes, which need to be decoded. In runtests.py, we were just calling str() instead, which converts b'x' to "b'x'". As a result, the loop that was checking pkginfo for lines starting with 'library-dirs' couldn't work. Reviewers: bgamari, thomie, Phyx Reviewed By: thomie Subscribers: Phyx, rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5046
* Strip ../ from testdir (fixes #15469)Thomas Miedema2018-08-061-1/+3
| | | | | | | | | | | | | | Test Plan: Harbormaster Reviewers: bgamari Reviewed By: bgamari Subscribers: rwbarton, carter GHC Trac Issues: #15469 Differential Revision: https://phabricator.haskell.org/D5039
* Fix typosKrzysztof Gogolewski2018-08-051-1/+1
|
* Remove dead code in testsuite driverKrzysztof Gogolewski2018-07-273-35/+4
| | | | | | | | | | | | Test Plan: validate Reviewers: bgamari, O7 GHC - Testsuite Reviewed By: bgamari Subscribers: rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D4972
* #15387 Fix setting testsuite verbose to zeroAntti Siponen2018-07-161-1/+1
|
* Disable -fghci-leak-check in DEBUG modeRichard Eisenberg2018-07-141-1/+2
| | | | | The DEBUG compiler's GHCi still leaks. This commit suppresses testsuite failures due to this leak. See #15372.
* testsuite: Print summary even if interruptedBen Gamari2018-06-141-15/+18
| | | | | | | | | | | | | | Fixes #15265. Reviewers: osa1 Reviewed By: osa1 Subscribers: rwbarton, thomie, carter GHC Trac Issues: #15265 Differential Revision: https://phabricator.haskell.org/D4841
* testuite: remove strace call.Tamar Christina2018-06-131-3/+0
|
* Fix `print-explicit-runtime-reps` (#11786).HE, Tao2018-06-081-2/+2
| | | | | | | | | | | | | | By fixing splitting of IfaceTypes in splitIfaceSigmaTy. Test Plan: make test TEST="T11549 T11376 T11786" Reviewers: goldfire, bgamari Subscribers: rwbarton, thomie, carter GHC Trac Issues: #11786, #11376 Differential Revision: https://phabricator.haskell.org/D4733
* testsuite: Fix incorrectly capitalized True in testlib.pyBen Gamari2018-05-291-1/+1
|
* Clean up Windows testsuite failuresTamar Christina2018-05-281-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Another round and attempt at getting these down to 0. We really should re-enable the CI and not wait for those cloud based ones. I've disabled the backpack tests on windows as they are too broad, they test as much the shell as they do the compiler. The perf tests have been too long to track down. but the numbers are horrible but I don't see them getting fixed so just have to accept them. T9293 has new windows specific output because a Dyn way only flag was added. This will of course not work on non-Dyn way builds. Test Plan: ./validate Reviewers: bgamari, hvr, simonmar Reviewed By: bgamari Subscribers: rwbarton, thomie, carter GHC Trac Issues: #15107 Differential Revision: https://phabricator.haskell.org/D4668
* Correctly add unwinding info in manifestSp and makeFixupBlocksBartosz Nitka2018-05-032-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | In `manifestSp` the unwind info was before the relevant instruction, not after. I added some notes to establish semantics. Also removes redundant annotation in stg_catch_frame. For `makeFixupBlocks` it looks like we were off by `wORD_SIZE dflags`. I'm not sure why, but it lines up with `manifestSp`. In fact it lines up so well so that I can consolidate the Sp unwind logic in `maybeAddUnwind`. I detected the problems with `makeFixupBlocks` by running T14779b after patching D4559. Test Plan: added a new test Reviewers: bgamari, scpmw, simonmar, erikd Reviewed By: bgamari Subscribers: thomie, carter GHC Trac Issues: #14999 Differential Revision: https://phabricator.haskell.org/D4606
* Don't shadow "result" in JUnit driverMatthew Pickering2018-05-031-2/+2
| | | | | | | | | | | | Reviewers: bgamari Reviewed By: bgamari Subscribers: thomie, carter GHC Trac Issues: #15093 Differential Revision: https://phabricator.haskell.org/D4645
* testsuite: allow accepting of fine grained results [skip ci]Tamar Christina2018-03-312-6/+18
| | | | | | | | | | | | | | | | | | | | | | Summary: Sometimes we need to be able to mass accept changes that are platform specific and that can't be normalized away using our string formatters. e.g. differences in I/O manager errors or behaviors. This allows one to do so easier than before and less error prone. I have updated the docs and made it clear this should only be used when a normalizer won't work: https://ghc.haskell.org/trac/ghc/wiki/Building/RunningTests/Updating Test Plan: Manually tested while working on new I/O manager Reviewers: bgamari Subscribers: thomie, carter Differential Revision: https://phabricator.haskell.org/D4549
* Windows: fix all failing tests.Tamar Christina2018-01-023-14/+36
| | | | | | | | | | | | | | | | | | | | | | This makes the testsuite pass clean on Windows again. It also fixes the `libstdc++-6.dll` error harbormaster was showing. I'm marking some tests as isolated tests to reduce their flakiness (mostly concurrency tests) when the test system is under heavy load. Updates process submodule. Test Plan: ./validate Reviewers: hvr, bgamari, erikd, simonmar Reviewed By: bgamari Subscribers: rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D4277
View Lorry Controller Status