MDEV-24845 Oddities around innodb_fatal_semaphore_wait_threshold and global.innodb_disallow_writes

This commit adds a mtr test for reproducing a test scenario where despite of
innodb_disallow_writes blocking, writes to file system can still happen.

The test launches a garbd node, which triggers one of the cluster node to switch to
SST donor state. In this state, all disk activity should be halted, and e.g.
innodb_disallow_writes has been set. The test records md5sum aggregate over mariadb
data directory when the node enters the donor state, and records another md5sum
when the node leaves the donor state. If there is no IO activity in data directory, these
hashes should be equal.

For this test, the Donor state processing, has beeen instrumented so that, SST donor thread can be
stopped when entering the donor state. The test uses this new dbug sync point,
to control when to record the md5sums.

New SST script was added: wsrep_sst_backup, and garbd uses backup method to lauch the donor
node to call this script, and to enter in donor state.

The backup script could be later extended as general purpose backup method for the cluster.

This commit fixes also one race condition happening in wsrep_sst_rsync, like this:
* wsrep_rsync_sst script requests for flush tables,
  and then waits in a loop until mariadbd has created file tables_flushed,
  as confirmation that FLUSH TABLES has completed
* mariadbd's SST donor thread, wakes for the flush table request and then performs FTWRL,
  and after this it creates the tables_flushed file
* note that SST script will now continue to startup rsync sending
* mariadbd's SST donor thread now calls for sst_disallow_writes(),
  so that innodb would setup disk IO blockage, however rsyncing may already be ongoing at this point

This race condition is fixed in this commit, by performing all disk IO blocking before
creating the tables_flushed file.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

1 files changed, 41 insertions, 0 deletions

diff --git a/mysql-test/suite/galera_3nodes/r/galera_garbd_backup.result b/mysql-test/suite/galera_3nodes/r/galera_garbd_backup.result
new file mode 100644
index 00000000000..f176ef1dd7f
--- /dev/null
+++ b/mysql-test/suite/galera_3nodes/r/galera_garbd_backup.result
@@ -0,0 +1,41 @@
+connection node_1;
+connection node_1;
+connection node_2;
+connection node_3;
+connection node_1;
+SET GLOBAL innodb_max_dirty_pages_pct=99;
+SET GLOBAL innodb_max_dirty_pages_pct_lwm=99;
+connection node_1;
+CREATE TABLE t1 (f1 INTEGER, f2 varchar(1024)) Engine=InnoDB;
+CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB;
+INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10);
+INSERT INTO t1 (f2) SELECT REPEAT('x', 1024) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;
+connection node_2;
+Killing node #3 to free ports for garbd ...
+connection node_3;
+connection node_1;
+SET GLOBAL debug_dbug = "+d,sync.wsrep_donor_state";
+Starting garbd ...
+SET SESSION debug_sync = "now WAIT_FOR sync.wsrep_donor_state_reached";
+SET GLOBAL innodb_max_dirty_pages_pct_lwm=0;
+SET GLOBAL innodb_max_dirty_pages_pct=0;
+SET SESSION debug_sync = "now SIGNAL signal.wsrep_donor_state";
+SET GLOBAL debug_dbug = "";
+SET debug_sync='RESET';
+connection node_2;
+Killing garbd ...
+connection node_1;
+connection node_2;
+DROP TABLE t1;
+DROP TABLE ten;
+Restarting node #3 to satisfy MTR's end-of-test checks
+connection node_3;
+connection node_1;
+SET GLOBAL innodb_max_dirty_pages_pct = 75.000000;
+SET GLOBAL innodb_max_dirty_pages_pct_lwm = 0.000000;
+connection node_1;
+CALL mtr.add_suppression("WSREP: Protocol violation\. JOIN message sender 1\.0 \(.*\) is not in state transfer \(SYNCED\)");
+connection node_2;
+CALL mtr.add_suppression("WSREP: Protocol violation\. JOIN message sender 1\.0 \(.*\) is not in state transfer \(SYNCED\)");
+connection node_3;
+CALL mtr.add_suppression("WSREP: Protocol violation\. JOIN message sender 1\.0 \(.*\) is not in state transfer \(SYNCED\)");