MDEV-23536 : Race condition between KILL and transaction commitbb-10.4-MDEV-23536

A race condition may occur between the execution of transaction commit,
and an execution of a KILL statement that would attempt to abort that
transaction.

MDEV-17092 worked around this race condition by modifying InnoDB code.
After that issue was closed, Sergey Vojtovich pointed out that this
race condition would better be fixed above the storage engine layer:

If you look carefully into the above, you can conclude that
thd->free_connection() can be called concurrently with
KILL/thd->awake(). Which is the bug. And it is partially fixed in
THD::~THD(), that is destructor waits for KILL completion:

Fix: Add necessary mutex operations to THD::free_connection()
and move WSREP specific code also there. This ensures that no
one is using THD while we do free_connection(). These mutexes
will also ensures that there can't be concurrent KILL/THD::awake().

innobase_kill_query
  We can now remove usage of trx_sys_mutex introduced on MDEV-17092.

trx_t::free()
  Poison trx->state and trx->mysql_thd

This patch is validated with an RQG run similar to the one that
reproduced MDEV-17092.

1 files changed, 41 insertions, 19 deletions

diff --git a/sql/sql_class.cc b/sql/sql_class.cc
index ceb8dc1ade8..2efd7685be6 100644
--- a/sql/sql_class.cc
+++ b/sql/sql_class.cc
@@ -1512,16 +1512,28 @@ void THD::reset_db(const LEX_CSTRING *new_db)
 
 /* Do operations that may take a long time */
 
-void THD::cleanup(void)
+void THD::cleanup(bool have_mutex)
 {
   DBUG_ENTER("THD::cleanup");
   DBUG_ASSERT(cleanup_done == 0);
 
-  set_killed(KILL_CONNECTION);
+  if (have_mutex)
+    set_killed_no_mutex(KILL_CONNECTION,0,0);
+  else
+    set_killed(KILL_CONNECTION);
 #ifdef WITH_WSREP
   if (wsrep_cs().state() != wsrep::client_state::s_none)
   {
-    wsrep_cs().cleanup();
+    if (have_mutex)
+    {
+      mysql_mutex_assert_owner(static_cast<mysql_mutex_t*>(m_wsrep_mutex.native()));
+      // Below wsrep-lib function will not take any mutexes
+      wsrep::unique_lock<wsrep::mutex> lock(m_wsrep_mutex, std::adopt_lock);
+      wsrep_cs().cleanup(lock);
+      lock.release();
+    }
+    else
+      wsrep_cs().cleanup();
   }
   wsrep_client_thread= false;
 #endif /* WITH_WSREP */
@@ -1598,6 +1610,28 @@ void THD::cleanup(void)
 void THD::free_connection()
 {
   DBUG_ASSERT(free_connection_done == 0);
+  /* Make sure threads are not available via server_threads.  */
+  assert_not_linked();
+
+  /*
+    Other threads may have a lock on THD::LOCK_thd_data or
+    THD::LOCK_thd_kill to ensure that this THD is not deleted
+    while they access it. The following mutex_lock ensures
+    that no one else is using this THD and it's now safe to
+    continue.
+
+    For example consider KILL-statement execution on
+    sql_parse.cc kill_one_thread() that will use
+    THD::LOCK_thd_data to protect victim thread during
+    THD::awake().
+  */
+  mysql_mutex_lock(&LOCK_thd_data);
+  mysql_mutex_lock(&LOCK_thd_kill);
+
+#ifdef WITH_WSREP
+  delete wsrep_rgi;
+  wsrep_rgi= 0;
+#endif
   my_free((char*) db.str);
   db= null_clex_str;
 #ifndef EMBEDDED_LIBRARY
@@ -1606,8 +1640,8 @@ void THD::free_connection()
   net.vio= 0;
   net_end(&net);
 #endif
- if (!cleanup_done)
-   cleanup();
+  if (!cleanup_done)
+    cleanup(true); // We have locked THD::LOCK_thd_kill
   ha_close_connection(this);
   plugin_thdvar_cleanup(this);
   mysql_audit_free_thd(this);
@@ -1618,6 +1652,8 @@ void THD::free_connection()
 #if defined(ENABLED_PROFILING)
   profiling.restart();                          // Reset profiling
 #endif
+  mysql_mutex_unlock(&LOCK_thd_kill);
+  mysql_mutex_unlock(&LOCK_thd_data);
 }
 
 /*
@@ -1671,24 +1707,10 @@ THD::~THD()
   if (!status_in_global)
     add_status_to_global();
 
-  /*
-    Other threads may have a lock on LOCK_thd_kill to ensure that this
-    THD is not deleted while they access it. The following mutex_lock
-    ensures that no one else is using this THD and it's now safe to delete
-  */
-  if (WSREP_NNULL(this)) mysql_mutex_lock(&LOCK_thd_data);
-  mysql_mutex_lock(&LOCK_thd_kill);
-  mysql_mutex_unlock(&LOCK_thd_kill);
-  if (WSREP_NNULL(this)) mysql_mutex_unlock(&LOCK_thd_data);
-
   if (!free_connection_done)
     free_connection();
 
 #ifdef WITH_WSREP
-  if (wsrep_rgi != NULL) {
-    delete wsrep_rgi;
-    wsrep_rgi = NULL;
-  }
   mysql_cond_destroy(&COND_wsrep_thd);
 #endif
   mdl_context.destroy();