summaryrefslogtreecommitdiff
path: root/testsuite/driver/testlib.py
Commit message (Collapse)AuthorAgeFilesLines
...
* testsuite: Classify missing expected perf numbers as merely warningsBen Gamari2017-04-021-2/+15
| | | | | | Previously these were considered to be framework failures, meaning that validate would fail. For better or worse, Windows lacks a good number of metrics and I don't see this changing any time soon. Let's consider these to be non-fatal.
* array: Clear up inconsistency in T9220 outputBen Gamari2017-04-021-0/+7
| | | | | | | ghc-8.2 and master disagreed on the order of the instances. Normalise this difference away. Updates array submodule.
* testsuite: Move echoing commands in make invocations to VERBOSE=5Reid Barton2017-03-021-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | D2894 added a new verbosity level VERBOSE=4 to strip -s/--silent flags from make invocations in test commands. This will probably cause the test to fail of course, but is useful for seeing what a test that's already failing is doing. However there was already an undocumented meaning of VERBOSE=4, added in commit cfeededf, that causes the results of performance tests to be printed unconditionally (even when they are within the expected range). nomeata's ghc builder uses these figures to collect historical data on performance test figures. The new meaning of VERBOSE=4 added in D2894 means that any test that uses make now fails on the builder. This commit moves the new behavior of D2894 to the level VERBOSE=5 so that nomeata's ghc builder again produces useful results on failing tests. It also adds documentation for both settings. Test Plan: did some manual testing Reviewers: austin, bgamari, Phyx, nomeata Reviewed By: bgamari, Phyx Subscribers: nomeata, thomie, Phyx Differential Revision: https://phabricator.haskell.org/D3141
* tests: remove extra_files.py (#12223)Reid Barton2017-02-261-1/+1
| | | | | | | | | | | | The script I used is included as testsuite/driver/kill_extra_files.py, though at this point it is for mostly historical interest. Some of the tests in libraries/hpc relied on extra_files.py, so this commit includes an update to that submodule. One test in libraries/process also relies on extra_files.py, but we cannot update that submodule so easily, so for now we special-case it in the test driver.
* Add delete retry loop. [ci skip]Tamar Christina2017-01-281-14/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: On Windows we have to retry the delete a couple of times. The reason for this is that a `FileDelete` command just marks a file for deletion. The file is really only removed when the last handle to the file is closed. Unfortunately there are a lot of system services that can have a file temporarily opened using a shared readonly lock, such as the built in AV and search indexer. We can't really guarantee that these are all off, so what we can do is whenever after a `rmtree` the folder still exists to try again and wait a bit. Based on what I've seen from the tests on CI server, is that this is relatively rare. So overall we won't be retrying a lot. If after a reasonable amount of time the folder is still locked then abort the current test by throwing an exception, this so it won't fail with an even more cryptic error. The issue is that these services often open a file using `FILE_SHARE_DELETE` permissions. So they can seemingly be removed, and for most intended purposes they are, but recreating the file with the same name will fail as the FS will prevent data loss. The MSDN docs for `DeleteFile` says: ``` The DeleteFile function marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED. ``` Retrying seems to be a common pattern, SQLite has it in their driver http://www.sqlite.org/src/info/89f1848d7f The only way to avoid this is to run each way of a test in it's own folder. This would also have the added bonus of increased parallelism. Reviewers: austin, bgamari Reviewed By: bgamari Subscribers: thomie, #ghc_windows_task_force Differential Revision: https://phabricator.haskell.org/D2936 GHC Trac Issues: #12661, #13162
* testsuite driver: don't append to existing output filesReid Barton2017-01-101-2/+2
| | | | | | | | | | | | | | | | | | | If you happen to have a T1234.run.stdout file lying aroud (probably from before the move to running tests in temporary subdirectories) it gets symlinked into the T1234.run directory since its name starts with T1234; and then program output gets appended to the existing file (through the symlink). We should open the file for writing instead, to replace the symlink with a new file. Test Plan: tested locally, + harbormaster Reviewers: austin, Phyx, bgamari Reviewed By: Phyx, bgamari Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D2946
* Fix various issues with testsuite code on WindowsTamar Christina2016-12-281-36/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Previously we would make direct calls to `diff` using `os.system`. On Windows `os.system` is implemented using the standard idiom `CreateProcess .. WaitForSingleObject ..`. This again runs afoul with the `_exec` behaviour on Windows. So we ran into some trouble where sometimes `diff` would return before it's done. On tests which run multiple ways, such as `8086` what happens is that we think the diff is done and continue. The next way tries to set things up again by removing any previous directory. This would then fail with and error saying the directory can't be removed. Which is true, because the previous diff code/child is still running. We shouldn't make any external calls to anything using `os.system`. Instead just use `runCmd` which uses `timeout`. This also ensures that if we hit the cygwin bug where diff or any other utility hangs, we kill it and continue and not hang the entire test and leave hanging processes. Further more we also: Ignore error lines from `removeFile` from tools in the testsuite. This is a rather large hammer to work around the fact that `hsc2hs` often tries to remove it's own file too early. When this is patched the workaround can be removed. See Trac #9775 We mark `prog003` as skip. Since this test randomly fails and passes. For stability it's disabled but it is a genuine bug which we should find. It's something with interface files being overwritten. See Trac #11317 when `rmtree` hits a readonly file, the `onerror` handler is raised afterwards but not during the tree walk. It doesn't allow you to recover and continue as we thought. Instead you have to explicitly start again. This is why sometimes even though we call `cleanup` before `os.mkdirs`, it would sometimes fail with an error that the folder already exists. So we now do a second walk. A new verbosity level (4) will strip the silent flags from `MAKE` invocations so you can actually see what's going on. Test Plan: ./validate on build bots. Reviewers: bgamari, austin Reviewed By: bgamari Subscribers: mpickering, thomie, #ghc_windows_task_force Differential Revision: https://phabricator.haskell.org/D2894 GHC Trac Issues: #12661, #11317, #9775
* Fix typos (not test relevant)Gabor Greif2016-12-211-1/+1
|
* testsuite: make tests respond to SIGINT properlyPhil Ruffwind2016-12-091-13/+9
| | | | | | | | | | | | | | | | | The `std*_buffer` need to be bytes to avoid breaking Python 3. Also, using a blanket `except` in Python without specifying the exception types will catch special exceptions such as `KeyboardInterrupt`, which can prevent the program from being interrupted properly. Test Plan: validate Reviewers: thomie, austin, bgamari Reviewed By: bgamari Differential Revision: https://phabricator.haskell.org/D2805
* testsuite: Remove Unicode literals from driverBen Gamari2016-12-011-12/+12
| | | | | | | | | | | | | | | They are not supported by Python 3.0, 3.1, and 3.2 (but are supported by >= 3.3; silliness!) Test Plan: Validate on python 3.2 Reviewers: austin Subscribers: simonpj, thomie Differential Revision: https://phabricator.haskell.org/D2778 GHC Trac Issues: #12909, #9184
* Fix testsuite threading, timeout, encoding and performance issues on WindowsTamar Christina2016-11-291-79/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In a land far far away, a project called Cygwin was born. Cygwin used newlib as it's standard C library implementation. But Cygwin wanted to emulate POSIX systems as closely as possible. So it implemented `execv` using the Windows function `spawnve`. Specifically ``` spawnve (_P_OVERLAY, path, argv, cur_environ ()) ``` `_P_OVERLAY` is crucial, as it makes the function behave *sort of* like execv on linux. the child process replaces the original process. With one major difference because of the difference in process models on Windows: the original process signals the caller that it's done. this is why the file is still locked. because it's still running, control was returned because the parent process was destroyed, but the child is still running. I think it's just pure dumb luck, that the older runtimes are slow enough to give the process time to terminate before we tried deleting the file. Which explains why you do have sporadic failures even on older runtimes like 2.5.0, of a test or two (like T7307). So this patch fixes a couple of things. I leverage the existing `timeout.exe` to implement a workaround for this issue. a) The old timeout used to start the process then assign it to the job. This is slightly faulty since child processes are only assigned to a job is their parent were assigned at the time they started. So this was a race condition. I now create the process suspended, assign it to the job and then resume it. Which means all child processes are not running under the same job. b) First things, Is to prevent dangling child processes. I mark the job with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` so when the last process in the job is done, it insures all processes under the job are killed. c) Secondly, I change the way we wait for results. Instead of waiting for the parent process to terminate, I wait for the job itself to terminate. There's a slight subtlety there, we can't wait on the job itself. Instead we have to create an I/O Completion port and wait for signals on it. See https://blogs.msdn.microsoft.com/oldnewthing/20130405-00/?p=4743 This fixes the issues on all runtimes for me and makes T7307 pass consistenly. The threading was also simplified by hiding all the locking in a single semaphore and a completion class. Futhermore some additional error reporting was added. For encoding the testsuite now no longer passes a file handle to the subprocess since on windows, sh.exe seems to acquire a lock on the file that is not released in a timely fashion. I suspect this because cygwin seems to emulate console handles by creating file handles and using those for std handles. So when we give it an existing file handle it just locks the file. I what's happening is that it's not releasing the handle until all shared cygwin processes are dead. Which explains why it worked in single threaded mode. So now instead we pass a pipe and do not interpret the resulting data. Any bytes written to stdin or read out of stdout/stderr are done so in binary mode and we do not interpret the data. The reason for this is that we have encoding tests in GHC which pass invalid utf-8. If we try to handle the data as text then python will throw an exception instead of a test comparison failing. Also I have fixed the ability to override `PYTHON` when calling `make tests`. This now works the same as with `.\validate`. Finally, after cleaning up the locks I was able to make the abort behavior work correctly as I believe it was intended: when you press Ctrl+C and send an interrupt signal, the testsuite finishes the active tests and then gracefully exits showing you a report of the progress it did make. So using Ctrl+C will not just *die* as it did before. These changes lift the restriction on which python version you use (msys/mingw) or which runtime or python 3 or python 2. All combinations should now be supported. Test Plan: PATH=/usr/local/bin:/mingw64/bin:$APPDATA/cabal/bin:$PATH && PYTHON=/usr/bin/python THREADS=9 make test THREADS=9 make test PATH=/usr/local/bin:/mingw64/bin:$APPDATA/cabal/bin:$PATH && PYTHON=/usr/bin/python ./validate --quiet --testsuite-only Reviewers: erikd, RyanGlScott, bgamari, austin Subscribers: jrtc27, mpickering, thomie, #ghc_windows_task_force Differential Revision: https://phabricator.haskell.org/D2684 GHC Trac Issues: #12725, #12554, #12661, #12004
* Typos in commentsGabor Greif2016-11-251-1/+1
|
* testsuite: Rip out hack for #12554Ben Gamari2016-11-171-24/+1
| | | | | | | | | | | | | | | | | @Phyx is working on correctly fixing (pun intended) the underlying issue that prompted this hack. It turns out that `timeout` it the culprit. Moreover, this hack breaks on msys python builds, which don't export `WindowsError`. Test Plan: Validate on Windows with `msys` python. Reviewers: Phyx, austin Subscribers: thomie, Phyx Differential Revision: https://phabricator.haskell.org/D2724 GHC Trac Issues: #12554
* testsuite/driver: More Unicode awarenessBen Gamari2016-10-171-9/+9
| | | | | | | | | | | | | Explicitly specify utf8 encoding in a few spots which were failing on Windows with Python 3. Test Plan: Validate Reviewers: austin, thomie Differential Revision: https://phabricator.haskell.org/D2602 GHC Trac Issues: #9184
* testsuite: Work around #12554Ben Gamari2016-10-171-2/+42
| | | | | | | | | | | | | | | | It seems that Python 2.7.11 and "recent" msys2 releases are broken, holding open file locks unexpected. This causes rmtree to intermittently fail. Even worse, it would fail silently (since we pass ignore_errors=True), causing makedirs to fail later. We now explicitly check for the existence of the test directory before attempting to delete it and disable ignore_errors. Moreover, on Windows we now try multiple times to rmtree the testdir, working around the apparently msys bug. This is all just terrible, but Phyx and I spent several hours trying to track down the issue to no available. The workaround is better than nothing.
* The Backpack patch.Edward Z. Yang2016-10-081-7/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This patch implements Backpack for GHC. It's a big patch but I've tried quite hard to keep things, by-in-large, self-contained. The user facing specification for Backpack can be found at: https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst A guide to the implementation can be found at: https://github.com/ezyang/ghc-proposals/blob/backpack-impl/proposals/0000-backpack-impl.rst Has a submodule update for Cabal, as well as a submodule update for filepath to handle more strict checking of cabal-version. Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu> Test Plan: validate Reviewers: simonpj, austin, simonmar, bgamari, goldfire Subscribers: thomie, mpickering Differential Revision: https://phabricator.haskell.org/D1482
* Testsuite: do not depend on sys.stdout.encodingThomas Miedema2016-06-301-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The cause of #12213 is in dump_stdout and dump_stderr: print(read_no_crs(<filename>)) Commit 6f6f515401a29d26eaa5daae308b8e700abd4c04 changed read_no_crs to return a unicode string. Printing a unicode strings works fine as long as sys.stdout.encoding is 'UTF-8'. There are two reasons why sys.stdout.encoding might not be 'UTF-8'. * When output is going to a file, sys.stdout and sys.stdout do not respect the locale: $ LC_ALL=en_US.utf8 python -c 'import sys; print(sys.stderr.encoding)' UTF-8 $ LC_ALL=en_US.utf8 python -c 'import sys; print(sys.stderr.encoding)' 2>/dev/null None * When output is going to the terminal, explicitly reopening sys.stdout has the side-effect of changing sys.stdout.encoding from 'UTF-8' to 'None'. sys.stdout = os.fdopen(sys.__stdout__.fileno(), "w", 0) We currently do this to set a buffersize of 0 (the actual buffersize used is irrelevant for the sys.stdout.encoding problem). Solution: fix dump_stdout and dump_stderr to not use read_no_crs.
* Testsuite: use ignore_stderr/stdout instead of ignore_outputThomas Miedema2016-06-291-22/+27
| | | | | | | | | | | | | | | | | The problem with ignore_output is that it hides errors for WAY=ghci. GHCi always returns with exit code 0 (unless it is broken itself). For example: ghci015 must have been failing with compile errors for years, but we didn't notice because all output was ignored. Therefore, replace all uses of ignore_output with either ignore_stderr or ignore_stdout. In some cases I opted for adding the expected output. Update submodule hpc and stm. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2367
* Testsuite: fixes for python2.6 supportThomas Miedema2016-06-291-3/+3
|
* Testsuite: framework failure improvements (#11165)Thomas Miedema2016-06-281-2/+5
| | | | | | * add framework failures to unexpected results list * report errors in .T files as framework failures (show in summary) * don't report missing tests when framework failures in .T files
* Testsuite: cleanup printing of summaryThomas Miedema2016-06-281-123/+41
| | | | | | Just use a simple list of tuples, instead of a nested map. -90 lines of code.
* Testsuite: open/close stdin/stdout/stderr explicitlyThomas Miedema2016-06-281-75/+62
| | | | | | | | | | | | | | | | | | This allows run_command's to contain `|`, and `no_stdin` isn't necessary anymore. Unfortunately it doesn't fix T7037 on Windows which I had hoped it would (testsuite driver tries to read a file that it just created itself, but the OS says it doesn't exist). The only drawback of this commit is that the command that the testsuite prints to the terminal (for debugging purposes) doesn't mention the files that stdout and stderr are redirected to anymore. This is probably ok. Update submodule unix. Differential Revision: https://phabricator.haskell.org/D1234
* Testsuite: simplify extra_file handlingThomas Miedema2016-06-281-55/+13
| | | | | | | | | Before, `extra_files(['.hpc/Main.mix'])` meant copy `Main.mix` to `<testdir>/.hpc/Main.mix`. This feature wasn't really necessary, so now it just means copy `Main.mix` to `<testdir>/Main.mix`. This simplifies the implementation. Some small other cleanups as well. -40 lines of code.
* Testsuite: remove one level of indentation [skip ci]Thomas Miedema2016-06-271-133/+129
| | | | Refactoring only. Move try/except out of do_test.
* Testsuite: report duplicate testnames when `make TEST=<name>`Thomas Miedema2016-06-271-9/+9
|
* Testsuite: never pick up .T files in .run directoriesThomas Miedema2016-06-271-13/+8
| | | | | And use os.walk instead of calling os.listdir many times. The testsuite driver should be able to handle backward slashes on Windows now.
* Testsuite: do not copy .hi/.o files to testdir (#12112)Thomas Miedema2016-06-241-3/+6
|
* Testsuite: assume timeout_prog always existsThomas Miedema2016-06-201-86/+28
| | | | | | | | | | | Merge the following functions into one: * rawSystem * rawSystemWithTimeout * runCmd * runCmdFor * runCmdExitCode I don't know why this wasn't done before.
* Testsuite: delete dead code + cleanupThomas Miedema2016-06-201-22/+7
| | | | | | * Set config settings directly in mk/test.mk, instead of indirectly in config/ghc * passing --hpcdir for WAY=hpc is unnecessary
* Testsuite: remove `-Wno-warn-tabs` from default flagsThomas Miedema2016-06-201-40/+9
| | | | This allows the removal of the override_flags stuff in testlib.py.
* Testsuite: remove `-fforce-recomp` from default flags (#11980)Thomas Miedema2016-06-201-24/+10
| | | | | | | | | | | | | | There is no need for this flag anymore, since each test runs in a newly created directory. Removing it cleans up testlib.py a bit. There is a small risk that this renders some tests useless. It's hard to know. Those tests should have specified -fforce-recomp` explicitly anyway, so I'm not going to worry about it. I've fixed the ones that failed without -fforce-recomp. Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2346
* Testsuite: fix WAY=ghci when LOCAL=0Thomas Miedema2016-06-201-4/+4
|
* Testsuite: recover from utf8 decoding errorsThomas Miedema2016-06-201-2/+8
|
* Testsuite: validate the tests/stage1 directory with the stage1 compilerThomas Miedema2016-06-181-1/+31
| | | | | | | | | | | * See `Note [Why is there no stage1 setup function?]`. * Move T2632 to the tests/stage1 directory (#10382). Reviewed by: ezyang, nomeata, bgamari Differential Revision: https://phabricator.haskell.org/D2341 GHC Trac Issues: #12197
* Testsuite: write "\n" instead of "\r\n" when using mingw PythonThomas Miedema2016-06-181-11/+32
| | | | | | | | | | | | | | | Mingw style Python uses '\r\n' by default for newlines. This is annoying, because it means that when a GHC developer on Windows uses mingw Python to `make accept` a test, every single line of the .stderr file is touched. This makes it difficult to spot the real changes, and it leads to unnecessary git history bloat. Prevent this from happening by using io.open instead of open. See `Note [Universal newlines]` Reviewed by: Phyx Differential Revision: https://phabricator.haskell.org/D2342
* Testsuite: run tests in <testdir>.run instead of /tmpThomas Miedema2016-06-181-29/+19
| | | | | | | | | | | | | | | | | | | | As discussed in Phab:D1187, this approach makes it a bit easier to inspect the test directory while working on a new test. The only tests that needed changes are the ones that refer to files in ancestor directories. Those files are now copied directly into the test directory. validate still runs the tests in a temporary directory in /tmp, see `Note [Running tests in /tmp]` in testsuite/driver/runtests.py. Update submodule hpc. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2333 GHC Trac Issues: #11980
* Testsuite: delete dead code [skip ci]Thomas Miedema2016-06-091-19/+0
|
* Show sources of cost centers in .profÖmer Sinan Ağacan2016-06-081-15/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes the problem with duplicate cost-centre names that was reported a couple of times before. When a module implements a typeclass multiple times for different types, methods of different implementations get same cost-centre names and are reported like this: COST CENTRE MODULE %time %alloc CAF GHC.IO.Handle.FD 0.0 32.8 CAF GHC.Read 0.0 1.0 CAF GHC.IO.Encoding 0.0 1.8 showsPrec Main 0.0 1.2 readPrec Main 0.0 19.4 readPrec Main 0.0 20.5 main Main 0.0 20.2 individual inherited COST CENTRE MODULE no. entries %time %alloc %time %alloc MAIN MAIN 53 0 0.0 0.2 0.0 100.0 CAF Main 105 0 0.0 0.3 0.0 62.5 readPrec Main 109 1 0.0 0.6 0.0 0.6 readPrec Main 107 1 0.0 0.6 0.0 0.6 main Main 106 1 0.0 20.2 0.0 61.0 == Main 114 1 0.0 0.0 0.0 0.0 == Main 113 1 0.0 0.0 0.0 0.0 showsPrec Main 112 2 0.0 1.2 0.0 1.2 showsPrec Main 111 2 0.0 0.9 0.0 0.9 readPrec Main 110 0 0.0 18.8 0.0 18.8 readPrec Main 108 0 0.0 19.9 0.0 19.9 It's not possible to tell from the report which `==` took how long. This patch adds one more column at the cost of making outputs wider. The report now looks like this: COST CENTRE MODULE SRC %time %alloc CAF GHC.IO.Handle.FD <entire-module> 0.0 32.9 CAF GHC.IO.Encoding <entire-module> 0.0 1.8 CAF GHC.Read <entire-module> 0.0 1.0 showsPrec Main Main_1.hs:7:19-22 0.0 1.2 readPrec Main Main_1.hs:7:13-16 0.0 19.5 readPrec Main Main_1.hs:4:13-16 0.0 20.5 main Main Main_1.hs:(10,1)-(20,20) 0.0 20.2 individual inherited COST CENTRE MODULE SRC no. entries %time %alloc %time %alloc MAIN MAIN <built-in> 53 0 0.0 0.2 0.0 100.0 CAF Main <entire-module> 105 0 0.0 0.3 0.0 62.5 readPrec Main Main_1.hs:7:13-16 109 1 0.0 0.6 0.0 0.6 readPrec Main Main_1.hs:4:13-16 107 1 0.0 0.6 0.0 0.6 main Main Main_1.hs:(10,1)-(20,20) 106 1 0.0 20.2 0.0 61.0 == Main Main_1.hs:7:25-26 114 1 0.0 0.0 0.0 0.0 == Main Main_1.hs:4:25-26 113 1 0.0 0.0 0.0 0.0 showsPrec Main Main_1.hs:7:19-22 112 2 0.0 1.2 0.0 1.2 showsPrec Main Main_1.hs:4:19-22 111 2 0.0 0.9 0.0 0.9 readPrec Main Main_1.hs:7:13-16 110 0 0.0 18.8 0.0 18.8 readPrec Main Main_1.hs:4:13-16 108 0 0.0 19.9 0.0 19.9 CAF Text.Read.Lex <entire-module> 102 0 0.0 0.5 0.0 0.5 To fix failing test cases because of different orderings of cost centres (e.g. optimized and non-optimized build printing in different order), with this patch we also start sorting cost centres before printing. The order depends on 1) entries (more entered cost centres come first) 2) names (using strcmp() on cost centre names). Reviewers: simonmar, austin, erikd, bgamari Reviewed By: simonmar, bgamari Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D2282 GHC Trac Issues: #11543, #8473, #7105
* Testsuite driver: always quote opts.testdirThomas Miedema2016-06-071-18/+25
| | | | | | | This makes sure the testsuite keeps working when testdir contains backward slashes. Differential Revision: https://phabricator.haskell.org/D2314
* Testsuite: also normalise platform-dependent .stdout/stderrThomas Miedema2016-05-251-26/+12
| | | | | | | | | | | | | | | | | This effectively reverses commit 429f0099ab9adfadc779ca76f3aae1c9c160fb8c (2006). I don't see why platform-dependent .stdout/stderr files should //not// get normalised. It fixes T11223_link_order_a_b_2_fail on Windows, by normalising `ghc-stage2.exe` to `ghc` when comparing stderr with .stderr-mingw32. Reviewed by: Phyx Differential Revision: https://phabricator.haskell.org/D2267 GHC Trac Issues: #12118
* Testsuite: delete check_files_writtenThomas Miedema2016-05-241-179/+1
| | | | | | | | | The CHECK_FILES_WRITTEN feature is no longer necessary, since tests don't write to the source directory anymore (#11980). Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2162
* Testsuite: delete old cleanup code (#11980)Thomas Miedema2016-05-171-146/+13
|
* Testsuite: run tests in /tmp after copying required filesThomas Miedema2016-05-171-14/+137
| | | | | | | | | | | | | | | | | | | | | | | Major change to the testsuite driver. For each TEST: * create a directory `<testdir>` inside `/tmp`. * link/copy all source files that the test needs into `<testdir>`. * run the test inside `<testdir>`. * delete `<testdir>` Extra files are (temporarily) tracked in `testsuite/driver/extra_files.py`, but can also be specified using the `extra_files` setup function. Differential Revision: https://phabricator.haskell.org/D1187 Reviewed by: Rufflewind, bgamari Trac: #11980
* Testsuite: make CLEANUP=1 the default (#9758)Thomas Miedema2016-04-301-1/+1
| | | | | | | | | | Also move the `cleanup` setting from `default_testopts` to `config`. The `cleanup` setting is the same for all tests, hence it belongs in `config`. Reviewed by: austin Differential Revision: https://phabricator.haskell.org/D2148
* Testsuite: delete -fesc testsThomas Miedema2016-04-281-3/+0
| | | | | | | | | | | The -fesc flag does not exist, and has never existed. Also delete now unused config.compiler_tags, and 'Project version' never contains a '-'. Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2138
* testsuite: Identify framework failures in testsuite summaryBen Gamari2016-03-261-0/+14
| | | | | | | | | | | | | | | | Currently the testsuite driver tells you how many tests failed due to a framework failure but you need to manually grep through the testsuite output to identify which ones. Test Plan: Validate with, e.g., a timing out testcase Reviewers: austin, thomie Reviewed By: austin, thomie Differential Revision: https://phabricator.haskell.org/D2026 GHC Trac Issues: #11165
* Testsuite: check actual_prof_file only when neededThomas Miedema2016-02-291-10/+11
| | | | | | | | | Might be a little faster. Avoids testing for #6113 (.prof file not written when process is killed with any signal but SIGINT) for tests that don't have a .prof.sample file (which is almost all of them) when running the profiling ways. Tests that were failing because of #6113: T8089, overflow1, overflow2 and overflow3.
* Testsuite: do not write empty files on 'make accept'Thomas Miedema2016-02-251-2/+10
| | | | Also prevent showing '\ No newline at end of file' in diff output.
* Filter out -prof callstacks from test output (#11521)Thomas Miedema2016-02-231-3/+16
|
* Testsuite: delete compiler_lt/le/gt/ge setup functionsThomas Miedema2016-02-171-20/+0
| | | | | | | | Since we're not consisently keeping track of which tests should pass with which compiler versions, there is no point in keeping these functions. Update submodules containers, hpc and stm.
View Lorry Controller Status