summaryrefslogtreecommitdiff
path: root/testsuite/driver
Commit message (Collapse)AuthorAgeFilesLines
* Testsuite: more precise test optionsSylvain Henry2022-10-261-0/+6
| | | | | Necessary for newer cross-compiling backends (JS, Wasm) that don't support TH yet.
* Testsuite: Add a new tables_next_to_code predicate.Andreas Klebinger2022-10-172-0/+5
| | | | | | And use it to avoid T21710a failing on non-tntc archs. Fixes #22169
* Drop make build systemBen Gamari2022-08-252-2/+2
| | | | | | | | | | | Here we at long last remove the `make`-based build system, it having been replaced with the Shake-based Hadrian build system. Users are encouraged to refer to the documentation in `hadrian/doc` and this [1] blog post for details on using Hadrian. Closes #17527. [1] https://www.haskell.org/ghc/blog/20220805-make-to-hadrian.html
* testsuite: Use system-cxx-std-lib instead of config.stdcxx_implBen Gamari2022-07-162-5/+0
|
* Testsuite driver: don't crash on empty metricssheaf2022-04-301-3/+7
| | | | | | The testsuite driver crashed when trying to display minimum/maximum performance changes when there are no metrics (i.e. there is no baseline available). This patch fixes that.
* testsuite: Deduplicate ways correctlywip/deduplicate-waysMatthew Pickering2022-04-291-1/+2
| | | | | | This was leading to a bug where we would run a profasm test twice which led to invalid junit.xml which meant the test results database was not being populated for the fedora33-perf job.
* testsuite: Report minimum and maximum stat changesBen Gamari2022-04-271-3/+7
| | | | As suggested in #20733.
* testsuite: Ensure that GHC doesn't pick up environment filesBen Gamari2022-04-222-2/+4
| | | | | | | | | | | | | | | Here we set GHC_ENVIRONMENT="-" to ensure that GHC invocations of tests don't pick up a user's local package environment. Fixes #21365. Metric Decrease: T10421 T12234 T12425 T13035 T16875 T9198
* testsuite/driver: Treat framework failures of fragile tests as non-fatalBen Gamari2022-04-061-1/+3
| | | | | | Previously we would report framework failures of tests marked as fragile as failures. Now we rather treat them as fragile test failures, which are not fatal to the testsuite run. Noticed while investigating #21293.
* testsuite: Clean up tests depending on C++ std libBen Gamari2022-04-062-0/+4
|
* testsuite: Require LLVM for T15155lPeter Trommler2022-02-261-0/+3
|
* testsuite: Don't print names of all fragile tests on all runsMatthew Pickering2022-02-221-8/+0
| | | | | This information about fragile tests is pretty useless but annoying on CI where you have to scroll up a long way to see the actual issues.
* Reinstallable GHCZubin Duggal2022-02-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch allows ghc and its dependencies to be built using a normal invocation of cabal-install. Each componenent which relied on generated files or additional configuration now has a Setup.hs file. There are also various fixes to the cabal files to satisfy cabal-install. There is a new hadrian command which will build a stage2 compiler and then a stage3 compiler by using cabal. ``` ./hadrian/build build-cabal ``` There is also a new CI job which tests running this command. For the 9.4 release we will upload all the dependent executables to hackage and then end users will be free to build GHC and GHC executables via cabal. There are still some unresolved questions about how to ensure soundness when loading plugins into a reinstalled GHC (#20742) which will be tighted up in due course. Fixes #19896
* testsuite: Make sure all tests trigger ghc rebuildMatthew Pickering2022-02-112-2/+2
| | | | | | | I made a mistake when implementing #21029 which meant that certain tests didn't trigger a GHC recompilation. By adding the `test:ghc` target to the default settings all tests will now depend on this target unless explicitly opting out via the no_deps modifier.
* testsuite: Run testsuite dependency calculation before GHC is builtMatthew Pickering2022-02-042-5/+17
| | | | | | | | | | | | | | | | | | | | The main motivation for this patch is to allow tests to be added to the testsuite which test things about the source tree without needing to build GHC. In particular the notes linter can easily start failing and by integrating it into the testsuite the process of observing these changes is caught by normal validation procedures rather than having to run the linter specially. With this patch I can run ``` ./hadrian/build test --flavour=devel2 --only="uniques" ``` In a clean tree to run the checkUniques linter without having to build GHC. Fixes #21029
* testsuite: Honour PERF_BASELINE_COMMIT when computing allowed metric changesMatthew Pickering2022-02-022-2/+21
| | | | | | | We now get all the commits between the PERF_BASELINE_COMMIT and HEAD and check any of them for metric changes. Fixes #20882
* Fix a few Note inconsistenciesBen Gamari2022-02-012-4/+5
|
* testsuite: Fix import on python 3.10Zubin Duggal2022-01-121-1/+2
|
* Multiple Home UnitsMatthew Pickering2021-12-281-17/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Multiple home units allows you to load different packages which may depend on each other into one GHC session. This will allow both GHCi and HLS to support multi component projects more naturally. Public Interface ~~~~~~~~~~~~~~~~ In order to specify multiple units, the -unit @⟨filename⟩ flag is given multiple times with a response file containing the arguments for each unit. The response file contains a newline separated list of arguments. ``` ghc -unit @unitLibCore -unit @unitLib ``` where the `unitLibCore` response file contains the normal arguments that cabal would pass to `--make` mode. ``` -this-unit-id lib-core-0.1.0.0 -i -isrc LibCore.Utils LibCore.Types ``` The response file for lib, can specify a dependency on lib-core, so then modules in lib can use modules from lib-core. ``` -this-unit-id lib-0.1.0.0 -package-id lib-core-0.1.0.0 -i -isrc Lib.Parse Lib.Render ``` Then when the compiler starts in --make mode it will compile both units lib and lib-core. There is also very basic support for multiple home units in GHCi, at the moment you can start a GHCi session with multiple units but only the :reload is supported. Most commands in GHCi assume a single home unit, and so it is additional work to work out how to modify the interface to support multiple loaded home units. Options used when working with Multiple Home Units There are a few extra flags which have been introduced specifically for working with multiple home units. The flags allow a home unit to pretend it’s more like an installed package, for example, specifying the package name, module visibility and reexported modules. -working-dir ⟨dir⟩ It is common to assume that a package is compiled in the directory where its cabal file resides. Thus, all paths used in the compiler are assumed to be relative to this directory. When there are multiple home units the compiler is often not operating in the standard directory and instead where the cabal.project file is located. In this case the -working-dir option can be passed which specifies the path from the current directory to the directory the unit assumes to be it’s root, normally the directory which contains the cabal file. When the flag is passed, any relative paths used by the compiler are offset by the working directory. Notably this includes -i and -I⟨dir⟩ flags. -this-package-name ⟨name⟩ This flag papers over the awkward interaction of the PackageImports and multiple home units. When using PackageImports you can specify the name of the package in an import to disambiguate between modules which appear in multiple packages with the same name. This flag allows a home unit to be given a package name so that you can also disambiguate between multiple home units which provide modules with the same name. -hidden-module ⟨module name⟩ This flag can be supplied multiple times in order to specify which modules in a home unit should not be visible outside of the unit it belongs to. The main use of this flag is to be able to recreate the difference between an exposed and hidden module for installed packages. -reexported-module ⟨module name⟩ This flag can be supplied multiple times in order to specify which modules are not defined in a unit but should be reexported. The effect is that other units will see this module as if it was defined in this unit. The use of this flag is to be able to replicate the reexported modules feature of packages with multiple home units. Offsetting Paths in Template Haskell splices ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When using Template Haskell to embed files into your program, traditionally the paths have been interpreted relative to the directory where the .cabal file resides. This causes problems for multiple home units as we are compiling many different libraries at once which have .cabal files in different directories. For this purpose we have introduced a way to query the value of the -working-dir flag to the Template Haskell API. By using this function we can implement a makeRelativeToProject function which offsets a path which is relative to the original project root by the value of -working-dir. ``` import Language.Haskell.TH.Syntax ( makeRelativeToProject ) foo = $(makeRelativeToProject "./relative/path" >>= embedFile) ``` > If you write a relative path in a Template Haskell splice you should use the makeRelativeToProject function so that your library works correctly with multiple home units. A similar function already exists in the file-embed library. The function in template-haskell implements this function in a more robust manner by honouring the -working-dir flag rather than searching the file system. Closure Property for Home Units ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For tools or libraries using the API there is one very important closure property which must be adhered to: > Any dependency which is not a home unit must not (transitively) depend on a home unit. For example, if you have three packages p, q and r, then if p depends on q which depends on r then it is illegal to load both p and r as home units but not q, because q is a dependency of the home unit p which depends on another home unit r. If you are using GHC by the command line then this property is checked, but if you are using the API then you need to check this property yourself. If you get it wrong you will probably get some very confusing errors about overlapping instances. Limitations of Multiple Home Units ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are a few limitations of the initial implementation which will be smoothed out on user demand. * Package thinning/renaming syntax is not supported * More complicated reexports/renaming are not yet supported. * It’s more common to run into existing linker bugs when loading a large number of packages in a session (for example #20674, #20689) * Backpack is not yet supported when using multiple home units. * Dependency chasing can be quite slow with a large number of modules and packages. * Loading wired-in packages as home units is currently not supported (this only really affects GHC developers attempting to load template-haskell). * Barely any normal GHCi features are supported, it would be good to support enough for ghcid to work correctly. Despite these limitations, the implementation works already for nearly all packages. It has been testing on large dependency closures, including the whole of head.hackage which is a total of 4784 modules from 452 packages. Internal Changes ~~~~~~~~~~~~~~~~ * The biggest change is that the HomePackageTable is replaced with the HomeUnitGraph. The HomeUnitGraph is a map from UnitId to HomeUnitEnv, which contains information specific to each home unit. * The HomeUnitEnv contains: - A unit state, each home unit can have different package db flags - A set of dynflags, each home unit can have different flags - A HomePackageTable * LinkNode: A new node type is added to the ModuleGraph, this is used to place the linking step into the build plan so linking can proceed in parralel with other packages being built. * New invariant: Dependencies of a ModuleGraphNode can be completely determined by looking at the value of the node. In order to achieve this, downsweep now performs a more complete job of downsweeping and then the dependenices are recorded forever in the node rather than being computed again from the ModSummary. * Some transitive module calculations are rewritten to use the ModuleGraph which is more efficient. * There is always an active home unit, which simplifies modifying a lot of the existing API code which is unit agnostic (for example, in the driver). The road may be bumpy for a little while after this change but the basics are well-tested. One small metric increase, which we accept and also submodule update to haddock which removes ExtendedModSummary. Closes #10827 ------------------------- Metric Increase: MultiLayerModules ------------------------- Co-authored-by: Fendor <power.walross@gmail.com>
* Fix typosKrzysztof Gogolewski2021-12-251-1/+1
|
* perf ci: Start searching form the performance baselineMatthew Pickering2021-12-221-3/+4
| | | | | | | | | | If you specify PERF_BASELINE_COMMIT then this can fail if the specific commit you selected didn't have perf test metrics. (This can happen in CI for example if a build fails on master). Therefore instead of just reporting all tests as new, we start searching downwards from this point to try and find a good commit to report numbers from.
* testsuite: Remove reqlib modifierMatthew Pickering2021-12-221-12/+0
| | | | | | | | | | | | | | | | | | | | The reqlib modifer was supposed to indicate that a test needed a certain library in order to work. If the library happened to be installed then the test would run as normal. However, CI has never run these tests as the packages have not been installed and we don't want out tests to depend on things which might get externally broken by updating the compiler. The new strategy is to run these tests in head.hackage, where the tests have been cabalised as well as possible. Some tests couldn't be transferred into the normal style testsuite but it's better than never running any of the reqlib tests. https://gitlab.haskell.org/ghc/head.hackage/-/merge_requests/169 A few submodules also had reqlib tests and have been updated to remove it. Closes #16264 #20032 #17764 #16561
* testsuite: Ensure that -dcore-lint is not set for compiler performance testsMatthew Pickering2021-12-171-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This place ensures that the default -dcore-lint option is disabled by default when collect_compiler_stats is used but you can still pass -dcore-lint as an additional option (see T1969 which tests core lint performance). Fixes #20830 ------------------------- Metric Decrease: PmSeriesS PmSeriesT PmSeriesV T10858 T11195 T11276 T11374 T11822 T14052 T14052Type T17096 T17836 T17836b T18478 T18698a T18698b -------------------------
* testsuite: Format metric results with comma separatorMatthew Pickering2021-12-171-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As noted in #20763 the way the stats were printed was quite hard for a human to compare. Therefore we now insert the comma separator so that they are easier to compare at a glance. Before: ``` Baseline Test Metric value New value Change ----------------------------------------------------------------------------- Conversions(normal) run/alloc 107088.0 107088.0 +0.0% DeriveNull(normal) run/alloc 112050656.0 112050656.0 +0.0% InlineArrayAlloc(normal) run/alloc 1600040712.0 1600040712.0 +0.0% InlineByteArrayAlloc(normal) run/alloc 1440040712.0 1440040712.0 +0.0% InlineCloneArrayAlloc(normal) run/alloc 1600040872.0 1600040872.0 +0.0% MethSharing(normal) run/alloc 480097864.0 480097864.0 +0.0% T10359(normal) run/alloc 354344.0 354344.0 +0.0% ``` After ``` Baseline Test Metric value New value Change ---------------------------------------------------------------------------------- Conversions(normal) run/alloc 107,088 107,088 +0.0% DeriveNull(normal) run/alloc 112,050,656 112,050,656 +0.0% InlineArrayAlloc(normal) run/alloc 1,600,040,712 1,600,040,712 +0.0% InlineByteArrayAlloc(normal) run/alloc 1,440,040,712 1,440,040,712 +0.0% InlineCloneArrayAlloc(normal) run/alloc 1,600,040,872 1,600,040,872 +0.0% MethSharing(normal) run/alloc 480,097,864 480,097,864 +0.0% T10359(normal) run/alloc 354,344 354,344 +0.0% ``` Closes #20763
* Test-suite: fix geometric mean of empty listsheaf2021-12-031-2/+5
| | | | | | | The geometric mean computation panicked when it was given an empty list, which happens when there are no baselines. Instead, we should simply return 1.
* testsuite: Print geometric mean of stat metricsBen Gamari2021-12-011-0/+14
| | | | As suggested in #20733.
* testsuite: debounce title updatesZubin Duggal2021-11-251-5/+20
|
* Don't include types in test outputAndreas Klebinger2021-11-231-5/+5
|
* CmmSink: Be more aggressive in removing no-op assignments.Andreas Klebinger2021-11-231-3/+20
| | | | | | | | | | No-op assignments like R1 = R1 are not only wasteful. They can also inhibit other optimizations like inlining assignments that read from R1. We now check for assignments being a no-op before and after we simplify the RHS in Cmm sink which should eliminate most of these no-ops.
* Change CaseAlt and LambdaExpr to FunRhs in deriving Foldable and Traversable ↵Artyom Kuznetsov2021-10-261-0/+6
| | | | (#20496)
* ci: test in-tree compiler in hadrianZubin Duggal2021-10-132-4/+5
|
* hadrian, testsuite: Teach Hadrian to query the testsuite driver for dependenciesZubin Duggal2021-10-133-0/+36
| | | | Issues #19072, #17728, #20176
* testsuite: Clean up dynlib support predicatesBen Gamari2021-10-123-7/+35
| | | | | | | | | | | | | | | | | | Previously it was unclear whether req_shared_libs should require: * that the platform supports dynamic library loading, * that GHC supports dynamic linking of Haskell code, or * that the dyn way libraries were built Clarify by splitting the predicate into two: * `req_dynamic_lib_support` demands that the platform support dynamic linking * `req_dynamic_hs` demands that the GHC support dynamic linking of Haskell code on the target platform Naturally `req_dynamic_hs` cannot be true unless `req_dynamic_lib_support` is also true.
* testsuite: Fix overzealous command-line manglingBen Gamari2021-10-121-1/+1
| | | | | Previously this attempt at suppressing make's -s flag would mangle otherwise valid arguments.
* Improve overlap error for polykinded constraintssheaf2021-10-061-1/+3
| | | | | | | | | | | | | | There were two problems around `mkDictErr`: 1. An outdated call to `flattenTys` meant that we missed out on some instances. As we no longer flatten type-family applications, the logic is obsolete and can be removed. 2. We reported "out of scope" errors in a poly-kinded situation because `BoxedRep` and `Lifted` were considered out of scope. We fix this by using `pretendNameIsInScope`. fixes #20465
* hadrian: Use ghc version as suffix for all executablesMatthew Pickering2021-08-231-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | ``` [matt@nixos:~/ghc-unique-spin]$ ls _build/bindist/ghc-9.3.20210813-x86_64-unknown-linux/bin/ ghc haddock runghc ghc-9.3.20210813 haddock-ghc-9.3.20210813 runghc-9.3.20210813 ghc-iserv hp2ps runhaskell ghc-iserv-dyn hp2ps-ghc-9.3.20210813 runhaskell-9.3.20210813 ghc-iserv-dyn-ghc-9.3.20210813 hpc unlit ghc-iserv-ghc-9.3.20210813 hpc-ghc-9.3.20210813 unlit-ghc-9.3.20210813 ghc-pkg hsc2hs ghc-pkg-9.3.20210813 hsc2hs-ghc-9.3.20210813 [matt@nixos:~/ghc-unique-spin]$ ls _build/bindist/ghc-9.3.20210813-x86_64-unknown-linux/wrappers/ ghc ghc-pkg-9.3.20210813 hpc runghc-9.3.20210813 ghc-9.3.20210813 haddock hpc-ghc-9.3.20210813 runhaskell ghci haddock-ghc-9.3.20210813 hsc2hs runhaskell-9.3.20210813 ghci-9.3.20210813 hp2ps hsc2hs-ghc-9.3.20210813 ghc-pkg hp2ps-ghc-9.3.20210813 runghc ``` See the discussion on #19571 where we decided that it was most sensible to use the same version number as a suffix for all executables. For those whose version number is different to normal (for example, haddock as it's own versioning scheme) the additional "ghc" suffix is used. Cabal already knows to look for this suffix so should work nicely with existing tooling.
* testsuite: Add more debug output on failure to call ghc-pkgMatthew Pickering2021-07-271-1/+1
|
* packaging: Create both versioned and unversioned executablesMatthew Pickering2021-07-271-0/+6
| | | | | | | | | Before we would just copy the unversioned executable into the bindist. Now the actual executable is copied into the bindist and a version suffix is added. Then a wrapper or symlink is added which points to the versioned executable. Fixes #20074
* testsuite: Don't try to run tests with missing librariesMatthew Pickering2021-06-232-7/+7
| | | | | | | | | | | | | | | | As noticed by sgraf, we were still running reqlib tests, even if the library was not available. The reasons for this were not clear to me as they would never work and it was causing some issues with empty stderr files being generated if you used --test-accept. Now if the required library is not there, the test is just skipped, and a counter increased to mark the fact. Perhaps in the future it would be nicer to explicitly record why certain tests are skipped. Missing libraries causing a skip is a special case at the moment. Fixes #20005
* [testsuite] fix T13702 with clangMoritz Angermann2021-06-051-0/+5
|
* Driver Rework PatchMatthew Pickering2021-06-033-7/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | This patch comprises of four different but closely related ideas. The net result is fixing a large number of open issues with the driver whilst making it simpler to understand. 1. Use the hash of the source file to determine whether the source file has changed or not. This makes the recompilation checking more robust to modern build systems which are liable to copy files around changing their modification times. 2. Remove the concept of a "stable module", a stable module was one where the object file was older than the source file, and all transitive dependencies were also stable. Now we don't rely on the modification time of the source file, the notion of stability is moot. 3. Fix TH/plugin recompilation after the removal of stable modules. The TH recompilation check used to rely on stable modules. Now there is a uniform and simple way, we directly track the linkables which were loaded into the interpreter whilst compiling a module. This is an over-approximation but more robust wrt package dependencies changing. 4. Fix recompilation checking for dynamic object files. Now we actually check if the dynamic object file exists when compiling with -dynamic-too Fixes #19774 #19771 #19758 #17434 #11556 #9121 #8211 #16495 #7277 #16093
* testsuite: Don't copy .hi-boot and .o-boot files into temp dirMatthew Pickering2021-05-191-1/+3
|
* [darwin] stop the DYLD_LIBRARY_PATH madnessMoritz Angermann2021-05-071-5/+2
| | | | | | | | this causes *significant* slowdown on macOS as the linker ends up looking through all the paths. Slowdown can be as bad as 100% or more. (cherry picked from commit 820b0766984d42c06c977a6c32da75c429106f7f)
* [testsuite] filter out superfluous dylib warningsMoritz Angermann2021-05-071-3/+18
| | | | (cherry picked from commit 33c4d497545559a38bd8d1caf6c94e5e2a77647b)
* [testlib/driver] denoiseMoritz Angermann2021-05-072-6/+11
| | | | | | | | | | this prevents the testlib/driver to be overly noisy, and will also kill some noise produiced by the aarch64-darwin cc (for now). Fixing sysctl, will allow us to run the test's properly in a nix-shell on aarch64-darwin (cherry picked from commit 5109e87e13ab45d799db2013535f54ca35f1f4dc)
* test driver: Make sure RESIDENCY_OPTS is passed for 'all' perf testsMatthew Pickering2021-05-052-12/+33
| | | | | | | | | | | | Fixes #19731 ------------------------- Metric Decrease: T11545 Metric Increase: T12545 T15304 -------------------------
* tests: Allow --skip-perf-tests/--only-perf-tests to be used with ↵Matthew Pickering2021-04-101-1/+1
| | | | --ignore-perf-failures
* Revert "[ci/arm/darwin/testsuite] Forwards ports from GHC-8.10"Ben Gamari2021-04-053-30/+13
| | | | This reverts commit 0cbdba2768d84a0f6832ae5cf9ea1e98efd739da.
* testsuite: Check test stats only after test correctnessBen Gamari2021-04-021-5/+5
| | | | | | | | | Ticket #19576 noted that a test that failed in correctness (e.g. due to stderr mismatch) *and* failed due to a metrics change would report misleading stats. This was due to the testsuite driver *first* checking stats, before checking for correctness. Fix this. Closes #19576.
* testsuite: Make passFail a booleanBen Gamari2021-04-022-24/+13
|
View Lorry Controller Status