Revert "build: do not randomize tests by default"th/test-randomize-random-seed

Tests that use nmtst_*rand*() randomize test inputs and code paths.
The purpose is to cover more test cases (on average).

These may be test cases that we didn't even think about, or where
testing all combinations (every time) is not feasable.

The idea is that if you run the tests often enough, you will hit those
cases eventually. Optimally, we already perform these randomized tests
in a loop and would hit always all error cases. In practice, that is not
possible because we don't have time to extensivley loop and we don't
explicitly know all interesting cases we'd have to test. We we would
have a well-known, small set of interesting cases, we wouldn't need to
randomize in the first place.

So, "random" failures due to nmtst_*rand*() are not something to be
afraid of. You hunt them down by:

  1) first run the test in a loop with randomization enabled.

  2) once you get the failure, note the NMTST_SEED_RAND and reproduce
     the failure reliably.

Note that we want that failures are reliably reproducable (2) for a
given seed. For that, all randomization must happen for internal reasons.
Meaning: the randomization in a unit tests must not depend on external factors
like the run time of a test step. If a randomized test cannot reliably
reproduce the failure after setting NMTST_SEED_RAND, then the test has a
bug that needs fixing.

Regardless of that, note that fixing NMTST_SEED_RAND to reproduce the failure
is only guaranteed to work if you run the test in the exact same configuration.
That means on the same machine, during the same boot, with the same
library versions, with the same compiler options, same environment variables,
and same command line ("-p TEST"). If you vary any of these, the failure may
not be reliably reproducable. But that's not a problem: just run the test in a
loop so that you find the offending NMTST_SEED_RAND that reproduce the issue for
*your* current configuration.

So: fixing the NMTST_SEED_RAND to 0 by default is harmful:

 a) it's not guaranteed that NMTST_SEED_RAND=0 will yield the same
    results in different configurations anyway. What does it help when the
    test "reliably" fails in gitlab-ci but you still cannot reproduce
    it on your system by using the same seed. Instead: you have to
    find a seed that reproduces the issue in your environment.

 b) despite a), by fixing the seed we wrongly limit the search space of
    inputs and code paths that are tested. It defeats the purpose of
    randomization. As explained, it is a given that one fixed seed is not
    sufficient to find all possible failures. We must find all possible
    failures by running often and with different seeds.

This reverts commit 9c6ff7fe1800ecbb6181a80c39e940d1bcc2da82.

1 files changed, 11 insertions, 17 deletions

diff --git a/shared/nm-utils/nm-test-utils.h b/shared/nm-utils/nm-test-utils.h
index 26190a7460..99e33c826d 100644
--- a/shared/nm-utils/nm-test-utils.h
+++ b/shared/nm-utils/nm-test-utils.h
@@ -46,10 +46,9 @@
  *   and g_test_assert_expected_messages() will not fail.
  *
  * NMTST_SEED_RAND environment variable:
- *   Tests that use random numbers from nmtst_get_rand() get seeded at each start.
- *   You can specify the seed by setting NMTST_SEED_RAND to a particular number or empty ("")
- *   for a random one. If NMTST_SEED_RAND is not set (default) a stable seed gets chosen.
- *   Tests will print the seed to stdout, so that you know which one was chosen or generated.
+ *   Tests that use random numbers from nmtst_get_rand() get seeded randomly at each start.
+ *   You can specify the seed by setting NMTST_SEED_RAND. Also, tests will print the seed
+ *   to stdout, so that you know the chosen seed.
  *
  *
  * NMTST_DEBUG environment variable:
@@ -827,20 +826,10 @@ nmtst_get_rand (void)
 
 	if (G_UNLIKELY (!__nmtst_internal.rand)) {
 		guint32 seed;
-		const char *str = g_getenv ("NMTST_SEED_RAND");
+		const char *str;
 
-		if (!str) {
-			/* No NMTST_SEED_RAND. Pick a stable one. */
-			seed = 0;
-			__nmtst_internal.rand = g_rand_new_with_seed (seed);
-		} else if (str[0] == '\0') {
-			/* NMTST_SEED_RAND is set but empty. Pick a random one. */
-			__nmtst_internal.rand = g_rand_new ();
-
-			seed = g_rand_int (__nmtst_internal.rand);
-			g_rand_set_seed (__nmtst_internal.rand, seed);
-		} else {
-			/* NMTST_SEED_RAND is set. Use it as a seed. */
+		if (   (str = g_getenv ("NMTST_SEED_RAND"))
+		    && str[0] != '\0') {
 			char *s;
 			gint64 i;
 
@@ -849,6 +838,11 @@ nmtst_get_rand (void)
 
 			seed = i;
 			__nmtst_internal.rand = g_rand_new_with_seed (seed);
+		} else {
+			__nmtst_internal.rand = g_rand_new ();
+
+			seed = g_rand_int (__nmtst_internal.rand);
+			g_rand_set_seed (__nmtst_internal.rand, seed);
 		}
 		__nmtst_internal.rand_seed = seed;