MDEV-19826 10.4 seems to crash with "pool-of-threads" (#1370)

MariaDB 10.4 was crashing when thread-handling was set to
pool-of-threads and wsrep was enabled.

There were two apparent reasons for the crash:
- Connection handling in threadpool_common.cc was missing calls to
  control wsrep client state.
- Thread specific storage which contains thread variables (THR_KEY_mysys)
  was not handled appropriately by wsrep patch when pool-of-threads
  was configured.

This patch addresses the above issues in the following way:
- Wsrep client state open/close was moved in thd_prepare_connection() and
  end_connection() to have common handling for one-thread-per-connection
  and pool-of-threads.
- Thread local storage handling in wsrep patch was reworked by introducing
  set of wsrep_xxx_threadvars() calls which replace calls to
  THD store_globals()/reset_globals() and deal with thread handling
  specifics internally.

Wsrep-lib was updated to version which relaxes internal concurrency
related sanity checks.

Rollback code from wsrep_rollback_process() was extracted to separate calls
for better readability.

Post rollback thread was removed as it was completely unused.

1 files changed, 11 insertions, 7 deletions

diff --git a/sql/sql_connect.cc b/sql/sql_connect.cc
index b8fff8c864a..6bf43d3df5e 100644
--- a/sql/sql_connect.cc
+++ b/sql/sql_connect.cc
@@ -1188,6 +1188,16 @@ void end_connection(THD *thd)
 {
   NET *net= &thd->net;
 
+#ifdef WITH_WSREP
+  if (thd->wsrep_cs().state() == wsrep::client_state::s_exec)
+  {
+    /* Error happened after the thread acquired ownership to wsrep
+       client state, but before command was processed. Clean up the
+       state before wsrep_close(). */
+    wsrep_after_command_ignore_result(thd);
+  }
+  wsrep_close(thd);
+#endif /* WITH_WSREP */
   if (thd->user_connect)
   {
     /*
@@ -1321,6 +1331,7 @@ bool thd_prepare_connection(THD *thd)
   prepare_new_connection_state(thd);
 #ifdef WITH_WSREP
   thd->wsrep_client_thread= true;
+  wsrep_open(thd);
 #endif /* WITH_WSREP */
   return FALSE;
 }
@@ -1393,9 +1404,6 @@ void do_handle_one_connection(CONNECT *connect)
       create_user= FALSE;
       goto end_thread;
     }      
-#ifdef WITH_WSREP
-    wsrep_open(thd);
-#endif /* WITH_WSREP */
 
     while (thd_is_connection_alive(thd))
     {
@@ -1406,10 +1414,6 @@ void do_handle_one_connection(CONNECT *connect)
     }
     end_connection(thd);
 
-#ifdef WITH_WSREP
-    wsrep_close(thd);
-#endif /* WITH_WSREP */
-
 end_thread:
     close_connection(thd);