testsuite: Use separate setups for unstable tests instead of should_fail

There are two possible interpretations of "expected failure": either
the test *must* fail (exactly the inverse of an ordinary test, with
success becoming failure and failure becoming success), or the test
*may* fail (with success intended, but failure possible in some
environments). Autotools had the second interpretation, which seems
more useful in practice, but Meson has the first.

Instead of using should_fail, we can put the tests in one of two new
suites: "flaky" is intended for tests that succeed or fail unpredictably
according to the test environment or chance, while "failing" is for
tests that ought to succeed but currently never do as a result of a
bug or missing functionality. With a sufficiently new version of Meson,
the flaky and failing tests are not run by default, but can be requested
by running a setup that does not exclude them, with a command like:

    meson test --setup=x11_unstable --suite=flaky --suite=failing

As a bonus, now that we're setting up setups and their excluded suites
programmatically, the gsk-compare-broadway tests are also excluded by
default when running the test setup for a non-broadway backend.

When running the tests in CI, --suite=gtk overrides the default
exclude_suites, so we have to specify --no-suite=flaky and
--no-suite=failing explicitly.

This arrangement is inspired by GNOME/glib!2987, which was contributed
by Marco Trevisan.

Signed-off-by: Simon McVittie <smcv@debian.org>

1 files changed, 55 insertions, 15 deletions

diff --git a/.gitlab-ci/run-tests.sh b/.gitlab-ci/run-tests.sh
index e68cf5a6c9..b60c3b38cc 100755
--- a/.gitlab-ci/run-tests.sh
+++ b/.gitlab-ci/run-tests.sh
@@ -19,11 +19,21 @@ case "${backend}" in
                 --print-errorlogs \
                 --setup=${backend} \
                 --suite=gtk \
+                --no-suite=failing \
+                --no-suite=flaky \
                 --no-suite=gsk-compare-broadway
 
     # Store the exit code for the CI run, but always
     # generate the reports
     exit_code=$?
+
+    xvfb-run -a -s "-screen 0 1024x768x24 -noreset" \
+          meson test -C ${builddir} \
+                --timeout-multiplier "${MESON_TEST_TIMEOUT_MULTIPLIER}" \
+                --print-errorlogs \
+                --setup=${backend}_unstable \
+                --suite=flaky \
+                --suite=failing || true
     ;;
 
   wayland)
@@ -38,9 +48,18 @@ case "${backend}" in
                 --print-errorlogs \
                 --setup=${backend} \
                 --suite=gtk \
+                --no-suite=failing \
+                --no-suite=flaky \
                 --no-suite=gsk-compare-broadway
-
     exit_code=$?
+
+    meson test -C ${builddir} \
+                --timeout-multiplier "${MESON_TEST_TIMEOUT_MULTIPLIER}" \
+                --print-errorlogs \
+                --setup=${backend}_unstable \
+                --suite=flaky \
+                --suite=failing || true
+
     kill ${compositor}
     ;;
 
@@ -56,9 +75,18 @@ case "${backend}" in
                 --print-errorlogs \
                 --setup=${backend} \
                 --suite=gtk \
+                --no-suite=failing \
+                --no-suite=flaky \
                 --no-suite=gsk-compare-broadway
-
     exit_code=$?
+
+    meson test -C ${builddir} \
+                --timeout-multiplier "${MESON_TEST_TIMEOUT_MULTIPLIER}" \
+                --print-errorlogs \
+                --setup=${backend}_unstable \
+                --suite=flaky \
+                --suite=failing || true
+
     kill ${compositor}
     ;;
 
@@ -74,10 +102,20 @@ case "${backend}" in
                 --print-errorlogs \
                 --setup=${backend} \
                 --suite=gtk \
+                --no-suite=failing \
+                --no-suite=flaky \
                 --no-suite=gsk-compare-opengl
 
     # don't let Broadway failures fail the run, for now
     exit_code=0
+
+    meson test -C ${builddir} \
+                --timeout-multiplier "${MESON_TEST_TIMEOUT_MULTIPLIER}" \
+                --print-errorlogs \
+                --setup=${backend}_unstable \
+                --suite=flaky \
+                --suite=failing || true
+
     kill ${server}
     ;;
 
@@ -90,18 +128,20 @@ esac
 
 cd ${builddir}
 
-$srcdir/.gitlab-ci/meson-junit-report.py \
-        --project-name=gtk \
-        --backend=${backend} \
-        --job-id="${CI_JOB_NAME}" \
-        --output=report-${backend}.xml \
-        meson-logs/testlog-${backend}.json
-$srcdir/.gitlab-ci/meson-html-report.py \
-        --project-name=gtk \
-        --backend=${backend} \
-        --job-id="${CI_JOB_NAME}" \
-        --reftest-output-dir="testsuite/reftests/output/${backend}" \
-        --output=report-${backend}.html \
-        meson-logs/testlog-${backend}.json
+for suffix in "" "_unstable"; do
+    $srcdir/.gitlab-ci/meson-junit-report.py \
+            --project-name=gtk \
+            --backend="${backend}${suffix}" \
+            --job-id="${CI_JOB_NAME}" \
+            --output="report-${backend}${suffix}.xml" \
+            "meson-logs/testlog-${backend}${suffix}.json"
+    $srcdir/.gitlab-ci/meson-html-report.py \
+            --project-name=gtk \
+            --backend="${backend}${suffix}" \
+            --job-id="${CI_JOB_NAME}" \
+            --reftest-output-dir="testsuite/reftests/output/${backend}${suffix}" \
+            --output="report-${backend}${suffix}.html" \
+            "meson-logs/testlog-${backend}${suffix}.json"
+done
 
 exit $exit_code