summaryrefslogtreecommitdiff
path: root/testsuite/driver/runtests.py
Commit message (Collapse)AuthorAgeFilesLines
* testsuite: Use system-cxx-std-lib instead of config.stdcxx_implBen Gamari2022-07-161-2/+0
|
* Testsuite driver: don't crash on empty metricssheaf2022-04-301-3/+7
| | | | | | The testsuite driver crashed when trying to display minimum/maximum performance changes when there are no metrics (i.e. there is no baseline available). This patch fixes that.
* testsuite: Report minimum and maximum stat changesBen Gamari2022-04-271-3/+7
| | | | As suggested in #20733.
* testsuite: Ensure that GHC doesn't pick up environment filesBen Gamari2022-04-221-0/+3
| | | | | | | | | | | | | | | Here we set GHC_ENVIRONMENT="-" to ensure that GHC invocations of tests don't pick up a user's local package environment. Fixes #21365. Metric Decrease: T10421 T12234 T12425 T13035 T16875 T9198
* testsuite: Clean up tests depending on C++ std libBen Gamari2022-04-061-0/+2
|
* testsuite: Run testsuite dependency calculation before GHC is builtMatthew Pickering2022-02-041-5/+11
| | | | | | | | | | | | | | | | | | | | The main motivation for this patch is to allow tests to be added to the testsuite which test things about the source tree without needing to build GHC. In particular the notes linter can easily start failing and by integrating it into the testsuite the process of observing these changes is caught by normal validation procedures rather than having to run the linter specially. With this patch I can run ``` ./hadrian/build test --flavour=devel2 --only="uniques" ``` In a clean tree to run the checkUniques linter without having to build GHC. Fixes #21029
* testsuite: Honour PERF_BASELINE_COMMIT when computing allowed metric changesMatthew Pickering2022-02-021-1/+2
| | | | | | | We now get all the commits between the PERF_BASELINE_COMMIT and HEAD and check any of them for metric changes. Fixes #20882
* Fix a few Note inconsistenciesBen Gamari2022-02-011-1/+1
|
* testsuite: Format metric results with comma separatorMatthew Pickering2021-12-171-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As noted in #20763 the way the stats were printed was quite hard for a human to compare. Therefore we now insert the comma separator so that they are easier to compare at a glance. Before: ``` Baseline Test Metric value New value Change ----------------------------------------------------------------------------- Conversions(normal) run/alloc 107088.0 107088.0 +0.0% DeriveNull(normal) run/alloc 112050656.0 112050656.0 +0.0% InlineArrayAlloc(normal) run/alloc 1600040712.0 1600040712.0 +0.0% InlineByteArrayAlloc(normal) run/alloc 1440040712.0 1440040712.0 +0.0% InlineCloneArrayAlloc(normal) run/alloc 1600040872.0 1600040872.0 +0.0% MethSharing(normal) run/alloc 480097864.0 480097864.0 +0.0% T10359(normal) run/alloc 354344.0 354344.0 +0.0% ``` After ``` Baseline Test Metric value New value Change ---------------------------------------------------------------------------------- Conversions(normal) run/alloc 107,088 107,088 +0.0% DeriveNull(normal) run/alloc 112,050,656 112,050,656 +0.0% InlineArrayAlloc(normal) run/alloc 1,600,040,712 1,600,040,712 +0.0% InlineByteArrayAlloc(normal) run/alloc 1,440,040,712 1,440,040,712 +0.0% InlineCloneArrayAlloc(normal) run/alloc 1,600,040,872 1,600,040,872 +0.0% MethSharing(normal) run/alloc 480,097,864 480,097,864 +0.0% T10359(normal) run/alloc 354,344 354,344 +0.0% ``` Closes #20763
* Test-suite: fix geometric mean of empty listsheaf2021-12-031-2/+5
| | | | | | | The geometric mean computation panicked when it was given an empty list, which happens when there are no baselines. Instead, we should simply return 1.
* testsuite: Print geometric mean of stat metricsBen Gamari2021-12-011-0/+14
| | | | As suggested in #20733.
* ci: test in-tree compiler in hadrianZubin Duggal2021-10-131-3/+3
|
* hadrian, testsuite: Teach Hadrian to query the testsuite driver for dependenciesZubin Duggal2021-10-131-0/+10
| | | | Issues #19072, #17728, #20176
* testsuite: Clean up dynlib support predicatesBen Gamari2021-10-121-0/+1
| | | | | | | | | | | | | | | | | | Previously it was unclear whether req_shared_libs should require: * that the platform supports dynamic library loading, * that GHC supports dynamic linking of Haskell code, or * that the dyn way libraries were built Clarify by splitting the predicate into two: * `req_dynamic_lib_support` demands that the platform support dynamic linking * `req_dynamic_hs` demands that the GHC support dynamic linking of Haskell code on the target platform Naturally `req_dynamic_hs` cannot be true unless `req_dynamic_lib_support` is also true.
* [darwin] stop the DYLD_LIBRARY_PATH madnessMoritz Angermann2021-05-071-5/+2
| | | | | | | | this causes *significant* slowdown on macOS as the linker ends up looking through all the paths. Slowdown can be as bad as 100% or more. (cherry picked from commit 820b0766984d42c06c977a6c32da75c429106f7f)
* tests: Allow --skip-perf-tests/--only-perf-tests to be used with ↵Matthew Pickering2021-04-101-1/+1
| | | | --ignore-perf-failures
* Revert "[ci/arm/darwin/testsuite] Forwards ports from GHC-8.10"Ben Gamari2021-04-051-2/+5
| | | | This reverts commit 0cbdba2768d84a0f6832ae5cf9ea1e98efd739da.
* [ci/arm/darwin/testsuite] Forwards ports from GHC-8.10Moritz Angermann2021-03-211-5/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a set of forward ports (cherry-picks) from 8.10 - a7d22795ed [ci] Add support for building on aarch64-darwin - 5109e87e13 [testlib/driver] denoise - 307d34945b [ci] default value for CONFIGURE_ARGS - 10a18cb4e0 [testsuite] mark ghci056 as fragile - 16c13d5acf [ci] Default value for MAKE_ARGS - ab571457b9 [ci/build] Copy config.sub around - 251892b98f [ci/darwin] bump nixpkgs rev - 5a6c36ecb4 [testsuite/darwin] fix conc059 - aae95ef0c9 [ci] add timing info - 3592d1104c [Aarch64] No div-by-zero; disable test. - 57671071ad [Darwin] mark stdc++ tests as broken - 33c4d49754 [testsuite] filter out superfluous dylib warnings - 4bea83afec [ci/nix-shell] Add Foundation and Security - 6345530062 [testsuite/json2] Fix failure with LLVM backends - c3944bc89d [ci/nix-shell] [Darwin] Stop the ld warnings about libiconv. - b821fcc714 [testsuite] static001 is not broken anymore. - f7062e1b0c [testsuite/arm64] fix section_alignment - 820b076698 [darwin] stop the DYLD_LIBRARY_PATH madness - 07b1af0362 [ci/nix-shell] uniquify NIX_LDFLAGS{_FOR_TARGET} As well as a few additional fixups needed to make this block compile: - Fixup all.T - Set CROSS_TARGET, BROKEN_TESTS, XZ, RUNTEST_ARGS, default value. - [ci] shell.nix bump happy
* testsuite: Make --ignore-perf-tests more expressiveBen Gamari2021-03-191-2/+9
| | | | Allow skipping of only increases/decreases.
* testsuite: Introduce flag to ignore performance failuresBen Gamari2021-02-241-0/+2
| | | | Needed by #19025.
* Reduce default test verbosityMatthew Pickering2021-01-281-1/+1
|
* Testsuite: Support for user supplied package dbsAndreas Klebinger2020-11-041-0/+4
| | | | | | | | | We can now supply additional package dbs to the testsuite. For make the package db can be supplied by passing PACKAGE_DB=/path/to/db. In the testsuite driver it's passed via the --test-package-db argument.
* testsuite: Add --top flag to driverGHC GitLab CI2020-11-021-1/+6
| | | | | This allows us to make `config.top` a proper Path. Previously it was a str, which caused the Ghostscript detection logic to break.
* testsuite: Sort metrics by metric typeBen Gamari2020-10-161-1/+15
| | | | Closes #18838.
* Enhance metrics outputSylvain Henry2020-09-151-5/+6
|
* testsuite: Output performance test results in tabular formatDaishi Nakajima2020-09-081-19/+48
| | | | | | | | | | | | | | | | | this was suggested in #18417. Change the print format of the values. * Shorten commit hash * Reduce precision of the "Value" field * Shorten metrics name * e.g. runtime/bytes allocated -> run/alloc * Shorten "MetricsChange" * e.g. unchanged -> unch, increased -> incr And, print the baseline environment if there are baselines that were measured in a different environment than the current environment. If all "Baseline commit" are the same, print it once.
* testsuite: Allow baseline commit to be set explicitlyBen Gamari2020-08-181-8/+12
|
* Sort deterministically metric outputKrzysztof Gogolewski2020-05-211-1/+1
| | | | | | Previously, we sorted according to the test name and way, but the metrics (max_bytes_used/peak_megabytes_allocated etc.) were appearing in nondeterministic order.
* testsuite: Print sign of performance changesBen Gamari2020-05-131-2/+2
| | | | | Executes the minor formatting change in the tabulated performance changes suggested in #18135.
* testsuite: Make hasMetricsFile RHS more descriptiveBen Gamari2020-02-261-1/+1
|
* testsuite: Allow tests to be marked as broken on the command lineBen Gamari2020-02-261-0/+4
| | | | This allows us to work-around distribution-specific breakage easily.
* testsuite: Use colors more consistentlyBen Gamari2019-12-051-8/+1
|
* testsuite: Make performance metric summary more readableBen Gamari2019-12-051-15/+35
| | | | Along with some refactoring.
* testsuite: Factor out terminal coloringBen Gamari2019-12-051-0/+3
|
* CI: Always dump performance metrics.David Eichmann2019-10-221-3/+19
|
* testsuite: More type checking fixesBen Gamari2019-07-181-2/+2
|
* testsuite: A major revamp of the driverBen Gamari2019-06-251-11/+20
| | | | | | | | | This tries to put the testsuite driver into a slightly more maintainable condition: * Add type annotations where easily done * Use pathlib.Path instead of str paths * Make it pass the mypy typechecker
* gitlab-ci: Lint testsuite for framework failuresBen Gamari2019-06-141-1/+1
| | | | | This introduces a new lint job checking for framework failures and listing broken tests.
* Update terminal title while running test-suiteOleg Grenrus2019-05-141-1/+18
| | | | | Useful progress indicator even when `make test VERBOSE=1`, and when you do something else, but have terminal title visible.
* Exit with exit code 1 when tests unexpectedly passMatthew Pickering2019-02-231-0/+1
| | | | | | This was causing gitlab to not report from builds as failing. It also highlighted a problem with the LLVM tests where some of the external interpreter tests are failing.
* Fix test runner crash when not in a git repoDavid Eichmann2019-02-211-6/+5
| | | | Respect `inside_git_repo()` when checking performance stats.
* Fix and Reapply "Performance tests: recover a baseline from ancestor commits ↵David Eichmann2019-02-161-8/+27
| | | | and CI results."
* Revert "Performance tests: recover a baseline from ancestor commits and CI ↵Ben Gamari2019-01-311-28/+8
| | | | | | | | | results." Unfortunately this has broken all future commits due to spurious(?) performance changes which I have been unable to work around. This reverts commit cc2261d42f6a954d88e355aaad41f001f65c95da.
* Performance tests: recover a baseline from ancestor commits and CI results.David Eichmann2019-01-301-8/+28
| | | | gitlab-ci: push performance metrics as git notes to the "GHC Performance Notes" repository.
* Revert "Batch merge"Ben Gamari2019-01-301-28/+8
| | | | This reverts commit 76c8fd674435a652c75a96c85abbf26f1f221876.
* Batch mergeBen Gamari2019-01-301-8/+28
|
* testsuite: Add predicate for CPU feature availabilityBen Gamari2019-01-271-0/+7
| | | | | | | Previously testing code-generation for ISA extensions was nearly impossible since we had no ability to determine whether the host supports the needed extension. Here we fix this by introducing a simple /proc/cpuinfo-based testsuite predicate. We really ought to
* testsuite: Print which ways we are going to runBen Gamari2018-12-121-1/+4
|
* Do not save performance test results if worktree is dirty.David Eichmann2018-12-111-2/+7
| | | | | | | | | | | | Reviewers: bgamari, tdammers Reviewed By: bgamari, tdammers Subscribers: rwbarton, carter GHC Trac Issues: #15924 Differential Revision: https://phabricator.haskell.org/D5368
* testsuite: Don't use git status to determine whether we are inside a repoBen Gamari2018-12-011-3/+3
| | | | | Git status is extremely expensive for this task. We instead use `git rev-parse HEAD` and throw away the output to ensure we don't spam the user.
View Lorry Controller Status