MDEV-21910 : KIlling thread on Galera could cause mutex deadlockbb-10.1-MDEV-21910

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets consider a example starting from Galera BF transaction deciding to abort
conflicting transaction (let's call this thread1):

thread1:
lock_rec_other_has_conflicting
  here we hold both lock_sys mutex and trx mutex for conflicting lock transaction.
wsrep_innobase_kill_one_trx
  For victim we call wsrep_thd_awake

Next thread2 we assume user to execute KILL QUERY to some other executing
query (note it can't be BF query).

thread2:
sql_kill()
kill_one_thread
find_thread_by_id
  takes LOCK_thd_kill
thd->awake_no_mutex()

thread1:
thd->awake(KILL_QUERY)
  Tries to have LOCK_thd_kill but it is hold by thread2 so we wait (note that
  we still hold lock_sys mutex and trx mutex).

thread2:
ha_kill_query()
kill_handlerton
innobase_kill_query
lock_trx_handle_wait
  lock_mutex_enter() must wait as thread1 is holding it

Thus thread1 lock_sys, trx_mutex waits -> thread2 LOCK_thd_kill waits lock_sys -> thread1
==> thread1 waits -> thread2 waits -> thread1 ==> mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
	Here we introduce a list where victim threads are stored,
	condition variable to be used to wake up background thread
	and mutex to protect list.

wsrep_thd.cc
	Create a new background thread to handle victim thread
	abort. We may take wsrep_thd_LOCK mutex here but not any
	InnoDB mutexes.

wsrep_innobase_kill_one_trx
	Remove all the wsrep code that was moved to wsrep_thd.cc
	We just enqueue required information to background kill
	list and cancel victim trx lock wait if there is such.
	Here we have InnoDB lock sys mutex and trx mutex so here
	we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
	Cleanup only.

19 files changed, 294 insertions, 51 deletions

diff --git a/mysql-test/suite/galera/r/galera_bf_kill.result b/mysql-test/suite/galera/r/galera_bf_kill.result
new file mode 100644
index 00000000000..1ce2974f158
--- /dev/null
+++ b/mysql-test/suite/galera/r/galera_bf_kill.result
@@ -0,0 +1,49 @@
+connection node_2;
+CREATE TABLE t1(a int not null primary key auto_increment,b int) engine=InnoDB;
+insert into t1 values (NULL,1);
+connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2;
+connection node_2a;
+begin;
+update t1 set a = 5;
+connection node_2;
+select * from t1;
+a	b
+2	1
+disconnect node_2a;
+connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2;
+connection node_2a;
+begin;
+update t1 set a =5;
+connection node_2;
+select * from t1;
+a	b
+2	1
+disconnect node_2a;
+connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2;
+connection node_2a;
+begin;
+update t1 set a =5, b=2;
+connection node_2;
+ALTER TABLE t1 ADD UNIQUE KEY(b);
+select * from t1;
+a	b
+2	1
+disconnect node_2a;
+connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2;
+connection node_2a;
+begin;
+update t1 set a =5, b=2;
+connect node_2b, 127.0.0.1, root, , test, $NODE_MYPORT_2;
+connection node_2b;
+begin;
+update t1 set a =6, b=7;
+connection node_2;
+ALTER TABLE t1 ADD UNIQUE KEY(b);
+Warnings:
+Note	1831	Duplicate index `b_2`. This is deprecated and will be disallowed in a future release
+select * from t1;
+a	b
+2	1
+disconnect node_2a;
+disconnect node_2b;
+drop table t1;
diff --git a/mysql-test/suite/galera/r/galera_insert_multi.result b/mysql-test/suite/galera/r/galera_insert_multi.result
index 33717781f2c..467684eb988 100644
--- a/mysql-test/suite/galera/r/galera_insert_multi.result
+++ b/mysql-test/suite/galera/r/galera_insert_multi.result
@@ -1,32 +1,32 @@
 CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
 INSERT INTO t1 VALUES (1),(2);
 INSERT INTO t1 VALUES (3),(4);
-SELECT COUNT(*) = 4 FROM t1;
-COUNT(*) = 4
-1
-SELECT COUNT(*) = 4 FROM t1;
-COUNT(*) = 4
-1
+SELECT COUNT(*) AS EXPECT_4 FROM t1;
+EXPECT_4
+4
+SELECT COUNT(*) AS EXPECT_4 FROM t1;
+EXPECT_4
+4
 DROP TABLE t1;
 CREATE TABLE t1 (f1 INTEGER, KEY (f1)) ENGINE=InnoDB;
 INSERT INTO t1 VALUES (1),(1);
 INSERT INTO t1 VALUES (2),(2);
-SELECT COUNT(*) = 4 FROM t1;
-COUNT(*) = 4
-1
-SELECT COUNT(*) = 4 FROM t1;
-COUNT(*) = 4
-1
+SELECT COUNT(*) AS EXPECT_4 FROM t1;
+EXPECT_4
+4
+SELECT COUNT(*) AS EXPECT_4 FROM t1;
+EXPECT_4
+4
 DROP TABLE t1;
 CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
 INSERT INTO t1 VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (1);
 ERROR 23000: Duplicate entry '1' for key 'PRIMARY'
-SELECT COUNT(*) = 0 FROM t1;
-COUNT(*) = 0
-1
-SELECT COUNT(*) = 0 FROM t1;
-COUNT(*) = 0
-1
+SELECT COUNT(*) AS EXPECT_0 FROM t1;
+EXPECT_0
+0
+SELECT COUNT(*) AS EXPECT_0 FROM t1;
+EXPECT_0
+0
 DROP TABLE t1;
 CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
 SET AUTOCOMMIT = OFF;
@@ -49,10 +49,10 @@ START TRANSACTION;
 INSERT INTO t1 VALUES (2), (1);
 ROLLBACK;
 COMMIT;
-SELECT COUNT(*) = 2 FROM t1;
-COUNT(*) = 2
-1
-SELECT COUNT(*) = 2 FROM t1;
-COUNT(*) = 2
-1
+SELECT COUNT(*) AS EXPECT_2 FROM t1;
+EXPECT_2
+2
+SELECT COUNT(*) AS EXPECT_2 FROM t1;
+EXPECT_2
+2
 DROP TABLE t1;
diff --git a/mysql-test/suite/galera/r/galera_serializable.result b/mysql-test/suite/galera/r/galera_serializable.result
index 90fe628e505..fb9e403892e 100644
--- a/mysql-test/suite/galera/r/galera_serializable.result
+++ b/mysql-test/suite/galera/r/galera_serializable.result
@@ -24,4 +24,10 @@ INSERT INTO t1 VALUES (1,1);
 INSERT INTO t1 VALUES (1,2);
 COMMIT;
 ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
+SELECT * from t1;
+id	f2
+1	2
+SELECT * from t1;
+id	f2
+1	2
 DROP TABLE t1;
diff --git a/mysql-test/suite/galera/r/galera_var_slave_threads.result b/mysql-test/suite/galera/r/galera_var_slave_threads.result
index 5a78d84c24e..160b2c1872b 100644
--- a/mysql-test/suite/galera/r/galera_var_slave_threads.result
+++ b/mysql-test/suite/galera/r/galera_var_slave_threads.result
@@ -14,7 +14,7 @@ SELECT @@wsrep_slave_threads = 1;
 SET GLOBAL wsrep_slave_threads = 1;
 SELECT COUNT(*) FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user';
 COUNT(*)
-2
+3
 SELECT COUNT(*) FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND STATE LIKE '%wsrep aborter%';
 COUNT(*)
 1
diff --git a/mysql-test/suite/galera/t/MW-286.cnf b/mysql-test/suite/galera/t/MW-286.cnf
new file mode 100644
index 00000000000..8ec4c2333cb
--- /dev/null
+++ b/mysql-test/suite/galera/t/MW-286.cnf
@@ -0,0 +1,10 @@
+!include ../galera_2nodes.cnf
+
+[mysqld.1]
+wsrep-debug=ON
+wsrep-log-conflicts=ON
+
+[mysqld.2]
+wsrep-debug=ON
+wsrep-log-conflicts=ON
+
diff --git a/mysql-test/suite/galera/t/MW-286.test b/mysql-test/suite/galera/t/MW-286.test
index 426b4493bb7..fcfbf4c62c3 100644
--- a/mysql-test/suite/galera/t/MW-286.test
+++ b/mysql-test/suite/galera/t/MW-286.test
@@ -3,8 +3,6 @@
 #
 
 --source include/galera_cluster.inc
---source include/have_innodb.inc
---source include/big_test.inc
 
 --connection node_1
 CREATE TABLE ten (f1 INTEGER) Engine=InnoDB;
diff --git a/mysql-test/suite/galera/t/galera_as_slave_gtid_myisam.test b/mysql-test/suite/galera/t/galera_as_slave_gtid_myisam.test
index faa9ddfd5c8..d8760d2b639 100644
--- a/mysql-test/suite/galera/t/galera_as_slave_gtid_myisam.test
+++ b/mysql-test/suite/galera/t/galera_as_slave_gtid_myisam.test
@@ -57,9 +57,13 @@ DROP TABLE t1;
 reset master;
 
 --connection node_2
+--let $wait_condition = SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1';
+--source include/wait_condition.inc
 STOP SLAVE;
 RESET SLAVE ALL;
 reset master;
 
 --connection node_3
+--let $wait_condition = SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1';
+--source include/wait_condition.inc
 reset master;
diff --git a/mysql-test/suite/galera/t/galera_bf_abort_sleep.test b/mysql-test/suite/galera/t/galera_bf_abort_sleep.test
index 8d135dc7d42..304e13fafd1 100644
--- a/mysql-test/suite/galera/t/galera_bf_abort_sleep.test
+++ b/mysql-test/suite/galera/t/galera_bf_abort_sleep.test
@@ -8,8 +8,8 @@
 CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
 
 --connection node_2
-SET AUTOCOMMIT=OFF;
 --let $wsrep_local_bf_aborts_before = `SELECT VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_bf_aborts'`
+SET AUTOCOMMIT=OFF;
 INSERT INTO t1 VALUES (1);
 --send SELECT SLEEP(1000);
 
diff --git a/mysql-test/suite/galera/t/galera_bf_kill.cnf b/mysql-test/suite/galera/t/galera_bf_kill.cnf
new file mode 100644
index 00000000000..8ec4c2333cb
--- /dev/null
+++ b/mysql-test/suite/galera/t/galera_bf_kill.cnf
@@ -0,0 +1,10 @@
+!include ../galera_2nodes.cnf
+
+[mysqld.1]
+wsrep-debug=ON
+wsrep-log-conflicts=ON
+
+[mysqld.2]
+wsrep-debug=ON
+wsrep-log-conflicts=ON
+
diff --git a/mysql-test/suite/galera/t/galera_bf_kill.test b/mysql-test/suite/galera/t/galera_bf_kill.test
new file mode 100644
index 00000000000..0ad86c13dba
--- /dev/null
+++ b/mysql-test/suite/galera/t/galera_bf_kill.test
@@ -0,0 +1,118 @@
+--source include/galera_cluster.inc
+
+#
+# Test case 1: Start a transaction on node_2a and kill it
+# from other connection on same node
+#
+
+--connection node_2
+CREATE TABLE t1(a int not null primary key auto_increment,b int) engine=InnoDB;
+insert into t1 values (NULL,1);
+
+--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2
+--connection node_2a
+begin;
+update t1 set a = 5;
+
+--connection node_2
+
+--let $wait_condition = SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'root' AND COMMAND = 'Sleep' LIMIT 1
+--source include/wait_condition.inc
+
+--let $k_thread = `SELECT ID FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'root' AND COMMAND = 'Sleep' LIMIT 1`
+
+--disable_query_log
+--eval KILL $k_thread
+--enable_query_log
+
+select * from t1;
+--disconnect node_2a
+
+#
+# Test case 2: Start a transaction on node_2a and use
+# kill query from other connection on same node
+#
+
+--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2
+--connection node_2a
+begin;
+update t1 set a =5;
+
+--connection node_2
+--let $wait_condition = SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'root' AND COMMAND = 'Sleep' LIMIT 1
+--source include/wait_condition.inc
+
+--let $k_thread = `SELECT ID FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'root' AND COMMAND = 'Sleep' LIMIT 1`
+
+--disable_query_log
+--eval KILL QUERY $k_thread
+--enable_query_log
+
+select * from t1;
+--disconnect node_2a
+#
+# Test case 3: Start a transaction on node_2a and start a DDL on other transaction
+# that will then abort node_2a transaction
+#
+--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2
+--connection node_2a
+begin;
+update t1 set a =5, b=2;
+
+--connection node_2
+ALTER TABLE t1 ADD UNIQUE KEY b1(b);
+ALTER TABLE t1 DROP KEY b1;
+
+select * from t1;
+
+--disconnect node_2a
+
+#
+# Test case 4: Start a transaction on node_2a and conflicting transaction on node_2b
+# and start a DDL on other transaction that will then abort node_2a and node_2b
+# transactions
+#
+
+--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2
+--connection node_2a
+begin;
+update t1 set a =5, b=2;
+
+--connect node_2b, 127.0.0.1, root, , test, $NODE_MYPORT_2
+--connection node_2b
+begin;
+send update t1 set a =6, b=7;
+
+--connection node_2
+ALTER TABLE t1 ADD UNIQUE KEY b2(b);
+ALTER TABLE t1 DROP KEY b2;
+
+select * from t1;
+
+--disconnect node_2a
+--disconnect node_2b
+
+#
+# Test case 5: Start a transaction on node_2a with wsrep disabled 
+# and start a DDL on other transaction that will then abort node_2a
+# transactions
+#
+
+--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2
+--connection node_2a
+SET SESSION wsrep_on=OFF;
+begin;
+update t1 set a =5, b=2;
+
+--connection node_2
+ALTER TABLE t1 ADD UNIQUE KEY b3(b);
+
+select * from t1;
+
+--disconnect node_2a
+
+
+drop table t1;
+
+
+
diff --git a/mysql-test/suite/galera/t/galera_drop_database.cnf b/mysql-test/suite/galera/t/galera_drop_database.cnf
new file mode 100644
index 00000000000..8ec4c2333cb
--- /dev/null
+++ b/mysql-test/suite/galera/t/galera_drop_database.cnf
@@ -0,0 +1,10 @@
+!include ../galera_2nodes.cnf
+
+[mysqld.1]
+wsrep-debug=ON
+wsrep-log-conflicts=ON
+
+[mysqld.2]
+wsrep-debug=ON
+wsrep-log-conflicts=ON
+
diff --git a/mysql-test/suite/galera/t/galera_insert_multi.test b/mysql-test/suite/galera/t/galera_insert_multi.test
index d62283aff69..e225b4bb199 100644
--- a/mysql-test/suite/galera/t/galera_insert_multi.test
+++ b/mysql-test/suite/galera/t/galera_insert_multi.test
@@ -13,10 +13,16 @@ INSERT INTO t1 VALUES (1),(2);
 INSERT INTO t1 VALUES (3),(4);
 
 --connection node_1
-SELECT COUNT(*) = 4 FROM t1;
+--let $wait_condition = SELECT COUNT(*) = 4 FROM t1;
+--source include/wait_condition.inc
+
+SELECT COUNT(*) AS EXPECT_4 FROM t1;
 
 --connection node_2
-SELECT COUNT(*) = 4 FROM t1;
+--let $wait_condition = SELECT COUNT(*) = 4 FROM t1;
+--source include/wait_condition.inc
+
+SELECT COUNT(*) AS EXPECT_4 FROM t1;
 
 DROP TABLE t1;
 
@@ -32,10 +38,16 @@ INSERT INTO t1 VALUES (1),(1);
 INSERT INTO t1 VALUES (2),(2);
 
 --connection node_2
-SELECT COUNT(*) = 4 FROM t1;
+--let $wait_condition = SELECT COUNT(*) = 4 FROM t1;
+--source include/wait_condition.inc
+
+SELECT COUNT(*) AS EXPECT_4 FROM t1;
 
 --connection node_1
-SELECT COUNT(*) = 4 FROM t1;
+--let $wait_condition = SELECT COUNT(*) = 4 FROM t1;
+--source include/wait_condition.inc
+
+SELECT COUNT(*) AS EXPECT_4 FROM t1;
 
 DROP TABLE t1;
 
@@ -49,10 +61,10 @@ CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
 --error ER_DUP_ENTRY
 INSERT INTO t1 VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (1);
 
-SELECT COUNT(*) = 0 FROM t1;
+SELECT COUNT(*) AS EXPECT_0 FROM t1;
 
 --connection node_2
-SELECT COUNT(*) = 0 FROM t1;
+SELECT COUNT(*) AS EXPECT_0 FROM t1;
 
 DROP TABLE t1;
 
@@ -107,10 +119,13 @@ ROLLBACK;
 
 --connection node_2
 COMMIT;
-SELECT COUNT(*) = 2 FROM t1;
+SELECT COUNT(*) AS EXPECT_2 FROM t1;
 
 --connection node_1
-SELECT COUNT(*) = 2 FROM t1;
+--let $wait_condition = SELECT COUNT(*) = 2 FROM t1;
+--source include/wait_condition.inc
+
+SELECT COUNT(*) AS EXPECT_2 FROM t1;
 
 DROP TABLE t1;
 
diff --git a/mysql-test/suite/galera/t/galera_parallel_autoinc_largetrx.test b/mysql-test/suite/galera/t/galera_parallel_autoinc_largetrx.test
index 34558283462..3188ee19880 100644
--- a/mysql-test/suite/galera/t/galera_parallel_autoinc_largetrx.test
+++ b/mysql-test/suite/galera/t/galera_parallel_autoinc_largetrx.test
@@ -21,7 +21,7 @@ CREATE TABLE t1 (f1 INTEGER AUTO_INCREMENT PRIMARY KEY, f2 INTEGER) Engine=InnoD
 set session wsrep_sync_wait=15;
 --let $wsrep_slave_threads_orig = `SELECT @@wsrep_slave_threads`
 SET GLOBAL wsrep_slave_threads = 4;
---let $wait_condition = SELECT COUNT(*) = @@wsrep_slave_threads + 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
+--let $wait_condition = SELECT COUNT(*) = @@wsrep_slave_threads + 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
 --source include/wait_condition.inc
 
 
diff --git a/mysql-test/suite/galera/t/galera_parallel_autoinc_manytrx.test b/mysql-test/suite/galera/t/galera_parallel_autoinc_manytrx.test
index d04603891db..1db5ee3bdc7 100644
--- a/mysql-test/suite/galera/t/galera_parallel_autoinc_manytrx.test
+++ b/mysql-test/suite/galera/t/galera_parallel_autoinc_manytrx.test
@@ -25,7 +25,7 @@ CREATE TABLE t1 (f1 INTEGER AUTO_INCREMENT PRIMARY KEY, f2 INTEGER) Engine=InnoD
 set session wsrep_sync_wait=15;
 --let $wsrep_slave_threads_orig = `SELECT @@wsrep_slave_threads`
 SET GLOBAL wsrep_slave_threads = 4;
---let $wait_condition = SELECT COUNT(*) = @@wsrep_slave_threads + 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
+--let $wait_condition = SELECT COUNT(*) = @@wsrep_slave_threads + 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
 --source include/wait_condition.inc
 
 --connection node_1
diff --git a/mysql-test/suite/galera/t/galera_serializable.test b/mysql-test/suite/galera/t/galera_serializable.test
index b12d57fd488..acbf99810a2 100644
--- a/mysql-test/suite/galera/t/galera_serializable.test
+++ b/mysql-test/suite/galera/t/galera_serializable.test
@@ -7,7 +7,6 @@
 #
 
 --source include/galera_cluster.inc
---source include/have_innodb.inc
 
 --connection node_1
 
@@ -26,7 +25,12 @@ SELECT * FROM t1;
 --connection node_2
 INSERT INTO t1 VALUES (1,1);
 
---sleep 2
+--connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1
+--connection node_1a
+--let $wait_condition = SELECT COUNT(*) = 1 FROM t1;
+--error 0,ER_LOCK_DEADLOCK
+--source include/wait_condition.inc
+
 --connection node_1
 --error ER_LOCK_DEADLOCK
 SELECT * FROM t1;
@@ -46,7 +50,11 @@ SELECT * FROM t1;
 --connection node_2
 UPDATE t1 SET f2 = 2;
 
---sleep 2
+--connection node_1a
+--let $wait_condition = SELECT COUNT(*) = 1 FROM t1;
+--error 0,ER_LOCK_DEADLOCK
+--source include/wait_condition.inc
+
 --connection node_1
 --error ER_LOCK_DEADLOCK
 UPDATE t1 SET f2 = 3;
@@ -62,8 +70,6 @@ DELETE FROM t1;
 
 --connection node_1
 START TRANSACTION;
-
---connection node_1
 INSERT INTO t1 VALUES (1,1);
 
 --connection node_2
@@ -73,4 +79,18 @@ INSERT INTO t1 VALUES (1,2);
 --error ER_LOCK_DEADLOCK
 COMMIT;
 
+--let $wait_condition = SELECT COUNT(*) = 1 FROM t1;
+--source include/wait_condition.inc
+
+SELECT * from t1;
+
+--connection node_2
+
+--let $wait_condition = SELECT COUNT(*) = 1 FROM t1;
+--source include/wait_condition.inc
+
+SELECT * from t1;
+
 DROP TABLE t1;
+
+--disconnect node_1a
diff --git a/mysql-test/suite/galera/t/galera_unicode_pk.test b/mysql-test/suite/galera/t/galera_unicode_pk.test
index 0d571f5cfd7..242d27b2fa7 100644
--- a/mysql-test/suite/galera/t/galera_unicode_pk.test
+++ b/mysql-test/suite/galera/t/galera_unicode_pk.test
@@ -12,6 +12,9 @@ CREATE TABLE t1 (
 INSERT INTO t1 VALUES ('текст');
 
 --connection node_2
+--let $wait_condition = SELECT COUNT(*) = 1 FROM t1;
+--source include/wait_condition.inc
+
 SELECT f1 = 'текст' FROM t1;
 
 #
diff --git a/mysql-test/suite/galera/t/galera_var_slave_threads.test b/mysql-test/suite/galera/t/galera_var_slave_threads.test
index 95cd2aac163..940b7ac0e99 100644
--- a/mysql-test/suite/galera/t/galera_var_slave_threads.test
+++ b/mysql-test/suite/galera/t/galera_var_slave_threads.test
@@ -43,7 +43,7 @@ INSERT INTO t1 VALUES (1);
 SELECT COUNT(*) FROM t1;
 
 --echo # wsrep_slave_threads = 64
---let $wait_condition = SELECT COUNT(*) = @@wsrep_slave_threads + 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user';
+--let $wait_condition = SELECT COUNT(*) = 64 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE LIKE '%wsrep aborter%' OR STATE IS NULL)
 --source include/wait_condition.inc
 
 SELECT COUNT(*) FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND STATE LIKE '%wsrep aborter%';
@@ -75,7 +75,7 @@ while ($count)
 SELECT COUNT(*) FROM t2;
 
 --echo # wsrep_slave_threads = 1
---let $wait_condition = SELECT COUNT(*) = @@wsrep_slave_threads + 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user';
+--let $wait_condition = SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE LIKE '%wsrep aborter%' OR STATE IS NULL);
 --source include/wait_condition.inc
 
 SELECT COUNT(*) FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND STATE LIKE '%wsrep aborter%';
@@ -94,7 +94,7 @@ CREATE TABLE t1 (i INT AUTO_INCREMENT PRIMARY KEY) ENGINE=INNODB;
 
 --connection node_2
 SET GLOBAL wsrep_slave_threads = 4;
---let $wait_condition = SELECT COUNT(*) = @@wsrep_slave_threads + 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user'
+--let $wait_condition = SELECT COUNT(*) = 4 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE LIKE '%wsrep aborter%' OR STATE IS NULL);
 --source include/wait_condition.inc
 
 SET GLOBAL wsrep_slave_threads = 1;
@@ -119,7 +119,7 @@ SELECT NAME FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE NAME LIKE 'test/t%';
 #
 # make sure that we are left with exactly one applier thread before we leaving the test
 #
---let $wait_condition = SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user'
+--let $wait_condition = SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND STATE LIKE '%wsrep aborter%';
 --source include/wait_condition.inc
 
 SELECT COUNT(*) FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND STATE LIKE '%wsrep aborter%';
diff --git a/mysql-test/suite/wsrep/r/variables.result b/mysql-test/suite/wsrep/r/variables.result
index d11cffba116..800fce6b01c 100644
--- a/mysql-test/suite/wsrep/r/variables.result
+++ b/mysql-test/suite/wsrep/r/variables.result
@@ -144,7 +144,7 @@ Variable_name	Value
 Threads_connected	1
 SHOW STATUS LIKE 'wsrep_thread_count';
 Variable_name	Value
-wsrep_thread_count	2
+wsrep_thread_count	3
 
 SET @wsrep_slave_threads_saved= @@global.wsrep_slave_threads;
 SET GLOBAL wsrep_slave_threads= 10;
@@ -154,7 +154,7 @@ Variable_name	Value
 Threads_connected	1
 SHOW STATUS LIKE 'wsrep_thread_count';
 Variable_name	Value
-wsrep_thread_count	11
+wsrep_thread_count	12
 set wsrep_on=0;
 set wsrep_on=1;
 create user test@localhost;
diff --git a/mysql-test/suite/wsrep/t/variables.test b/mysql-test/suite/wsrep/t/variables.test
index 1a2ab2579a5..59d52e04c75 100644
--- a/mysql-test/suite/wsrep/t/variables.test
+++ b/mysql-test/suite/wsrep/t/variables.test
@@ -104,7 +104,7 @@ SET GLOBAL wsrep_cluster_address= 'gcomm://';
 
 --echo # Wait for applier threads to get created.
 --let $wait_timeout=600
---let $wait_condition = SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
+--let $wait_condition = SELECT COUNT(*) = 3 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
 --source include/wait_condition.inc
 
 --replace_regex /.*libgalera_smm.*/libgalera_smm.so/
@@ -118,7 +118,7 @@ SET @wsrep_slave_threads_saved= @@global.wsrep_slave_threads;
 SET GLOBAL wsrep_slave_threads= 10;
 
 --echo # Wait for applier threads to get created.
---let $wait_condition = SELECT COUNT(*) = 11 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
+--let $wait_condition = SELECT COUNT(*) = 12 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
 --source include/wait_condition.inc
 
 SHOW STATUS LIKE 'threads_connected';