ctdb-tests: Wait for child process when killing cluster mutex helper

The following test sometimes fails:

==================================================
Running "cluster_mutex_test lock-unlock-lock-unlock ./tests/var/cluster_mutex.lockfile"
--------------------------------------------------
Output (Exit status: 134):
--------------------------------------------------
LOCK
UNLOCK
CONTENTION
NOLOCK
cluster_mutex_test: ../../tests/src/cluster_mutex_test.c:307: test_lock_unlock_lock_unlock: Assertion `dl2->mh != NULL' failed.
--------------------------------------------------
Required output (Exit status: 0):
--------------------------------------------------
LOCK
UNLOCK
LOCK
UNLOCK

FAILED
==========================================================================
TEST FAILED: tests/cunit/cluster_mutex_001.sh (status 1) (duration: 0s)
==========================================================================

This is due to a race in the test.  For the first UNLOCK a signal is
sent to the cluster mutex handler but the test tries to retake the
lock before that process is scheduled and the signal is processed.
Therefore, the fcntl() lock is still held and contention is seen.

After unlocking, tests need to wait until the child has gone, so build
this into ctdb_kill().  This is one of the only places where the PID
is accessible.

Outside of testing, on a real system, nothing will never try
to (re)take the lock so quickly.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=14085

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

1 files changed, 17 insertions, 1 deletions

diff --git a/ctdb/tests/src/cluster_mutex_test.c b/ctdb/tests/src/cluster_mutex_test.c
index 3bf653a3b00..34398a98ea9 100644
--- a/ctdb/tests/src/cluster_mutex_test.c
+++ b/ctdb/tests/src/cluster_mutex_test.c
@@ -53,7 +53,23 @@ static pid_t ctdb_fork(struct ctdb_context *ctdb)
 
 static int ctdb_kill(struct ctdb_context *ctdb, pid_t pid, int signum)
 {
-	return kill(pid, signum);
+	/*
+	 * Tests need to wait for the child to exit to ensure that the
+	 * lock really has been released.  The PID is only accessible
+	 * in ctdb_cluster_mutex.c, so make a best attempt to ensure
+	 * that the child process is waited for after it is killed.
+	 * Avoid waiting if the process is already gone.
+	 */
+	int ret;
+
+	if (signum == 0) {
+		return kill(pid, signum);
+	}
+
+	ret = kill(pid, signum);
+	waitpid(pid, NULL, 0);
+
+	return ret;
 }
 
 #include "server/ctdb_cluster_mutex.c"