Widen T12545 acceptance windowwip/t12545-variance

This test has been the scourge of contributors for a long time.

It has caused many failed CI runs and wasted hours debugging a test
which barely does anything. The fact is does nothing is the reason for
the flakiness and it's very sensitive to small changes in initialisation costs,
in particular adding wired-in things can cause this test to fluctuate
quite a bit.

Therefore we admit defeat and just bump the threshold up to 10% to catch
very large regressions but otherwise don't care what this test does.

Fixes #19414

1 files changed, 9 insertions, 4 deletions

diff --git a/testsuite/tests/perf/compiler/all.T b/testsuite/tests/perf/compiler/all.T
index 25672bf7e7..4136572419 100644
--- a/testsuite/tests/perf/compiler/all.T
+++ b/testsuite/tests/perf/compiler/all.T
@@ -212,12 +212,17 @@ test('T12234',
      [''])
 
 # T12545 is sensitive to -dunique-increments changes, see #19414. I've seen
-# variations of as much as 4.8% by playing with that parameter, but I think
-# it's better to check with T12545.measure.sh that lower and upper bounds of
-# allocations indeed haven't changed and then simply accept the metric increase.
+# variations of as much as 4.8% by playing with that parameter,
+#
+# The issue with the test is that it does too little so is very sensitive to
+# any small variations during initialisation and in particular populating the
+# initial environments with wired-in things. Therefore it has a very high change
+# threshold so we catch if it regresses a lot but don't worry if it regresses a little.
+#
+# You can measure the variance by running T12545.measure.sh.
 test('T12545',
      [ only_ways(['normal']),
-       collect_compiler_stats('bytes allocated', 3), # Don't increase it, run T12545.measure.sh and leave a comment on #19414!
+       collect_compiler_stats('bytes allocated', 10), #
        extra_clean(['T12545a.hi', 'T12545a.o'])
      ],
      multimod_compile,