MDEV-16678 Prefer MDL to dict_sys.latch for innodb background tasks

This is joint work with Thirunarayanan Balathandayuthapani.
The MDL interface between InnoDB and the rest of the server
(in storage/innobase/dict/dict0dict.cc and in include/)
is my work, while most everything else is Thiru's.

The collection of InnoDB persistent statistics and the
defragmentation were not refactored to use MDL. They will
keep relying on lower-level interlocking with
fil_check_pending_operations().

The purge of transaction history and the background operations on
fulltext indexes will use MDL. We will revert
commit 2c4844c9e76427525e8c39a2d72686085efe89c3
(MDEV-17813) because thanks to MDL, purge cannot conflict
with DDL operations anymore. For a similar reason, we will remove
the MDEV-16222 test case from gcol.innodb_virtual_debug_purge.

Purge is essentially replacing all use of the global dict_sys.latch
with MDL. Purge will skip the undo log records for tables whose names
start with #sql-ib or #sql2. Theoretically, such tables might
be renamed back to visible table names if TRUNCATE fails to
create a new table, or the final rename in ALTER TABLE...ALGORITHM=COPY
fails. In that case, purge could permanently leave some garbage
in the table. Such garbage will be tolerated; the table would not
be considered corrupted.

To avoid repeated MDL releases and acquisitions,
trx_purge_attach_undo_recs() will sort undo log records by table_id,
and purge_node_t will keep the MDL and table handle open for multiple
successive undo log records.

get_purge_table(): A new accessor, used during the purge of
history for indexed virtual columns. This interface should ideally
not exist at all.

thd_mdl_context(): Accessor of THD::mdl_context.
Wrapped in a new thd_mdl_service.

dict_get_db_name_len(): Define inline.

dict_acquire_mdl_shared(): Acquire explicit shared MDL on a table name
if needed.

dict_table_open_on_id(): Return MDL_ticket, if requested.

dict_table_close(): Release MDL ticket, if requested.

dict_fts_index_syncing(), dict_index_t::index_fts_syncing: Remove.
row_drop_table_for_mysql() no longer needs to check these, because
MDL guarantees that a fulltext index sync will not be in progress
while MDL_EXCLUSIVE is protecting a DDL operation.

dict_table_t::parse_name(): Parse the table name for acquiring MDL.

purge_node_t::undo_recs: Change the type to std::list<trx_purge_rec_t*>
(different container, and storing also roll_ptr).

purge_node_t: Add mdl_ticket, last_table_id, purge_thd, mdl_hold_recs
for acquiring MDL and for keeping the table open across multiple
undo log records.

purge_vcol_info_t, row_purge_store_vsec_cur(), row_purge_restore_vsec_cur():
Remove. We will acquire the MDL earlier.

purge_sys_t::heap: Added, for reading undo log records.

fts_sync_during_ddl(): Invoked during ALGORITHM=INPLACE operations
to ensure that fts_sync_table() will not conflict with MDL_EXCLUSIVE.
Uses fts_t::sync_message for bookkeeping.

5 files changed, 1 insertions, 136 deletions

diff --git a/mysql-test/suite/gcol/r/innodb_virtual_debug_purge.result b/mysql-test/suite/gcol/r/innodb_virtual_debug_purge.result
index 1bbc577ed93..552805110b1 100644
--- a/mysql-test/suite/gcol/r/innodb_virtual_debug_purge.result
+++ b/mysql-test/suite/gcol/r/innodb_virtual_debug_purge.result
@@ -204,32 +204,5 @@ connection truncate;
 disconnect truncate;
 connection default;
 DROP TABLE t1, t2;
-#
-# MDEV-16222 Assertion `0' failed in row_purge_remove_sec_if_poss_leaf
-# on table with virtual columns and indexes
-#
-set @saved_dbug= @@global.debug_dbug;
-set global debug_dbug= "+d,ib_purge_virtual_mdev_16222_1,ib_purge_virtual_mdev_16222_2";
-create table t1 (
-pk serial, vb tinyblob as (b) virtual, b tinyblob,
-primary key(pk), index (vb(64)))
-engine innodb;
-insert ignore into t1 (b) values ('foo');
-select * into outfile 'load.data' from t1;
-load data infile 'load.data' replace into table t1;
-set debug_sync= "now WAIT_FOR latch_released";
-set global debug_dbug= "-d,ib_purge_virtual_mdev_16222_1";
-drop table t1;
-set debug_sync= "now SIGNAL drop_started WAIT_FOR got_no_such_table";
-create table t1 (
-pk serial, vb tinyblob as (b) virtual, b tinyblob,
-primary key(pk), index (vb(64)))
-engine innodb;
-insert ignore into t1 (b) values ('foo');
-select * into outfile 'load.data' from t1;
-load data infile 'load.data' replace into table t1;
-set debug_sync= "now WAIT_FOR got_no_such_table";
-set global debug_dbug= @saved_dbug;
-drop table t1;
 set debug_sync=reset;
 SET GLOBAL innodb_purge_rseg_truncate_frequency = @saved_frequency;
diff --git a/mysql-test/suite/gcol/t/innodb_virtual_debug_purge.test b/mysql-test/suite/gcol/t/innodb_virtual_debug_purge.test
index 04ab8a88488..8568c66eccc 100644
--- a/mysql-test/suite/gcol/t/innodb_virtual_debug_purge.test
+++ b/mysql-test/suite/gcol/t/innodb_virtual_debug_purge.test
@@ -259,54 +259,6 @@ disconnect truncate;
 connection default;
 DROP TABLE t1, t2;
 
---echo #
---echo # MDEV-16222 Assertion `0' failed in row_purge_remove_sec_if_poss_leaf
---echo # on table with virtual columns and indexes
---echo #
-
---let $datadir= `select @@datadir`
-set @saved_dbug= @@global.debug_dbug;
-set global debug_dbug= "+d,ib_purge_virtual_mdev_16222_1,ib_purge_virtual_mdev_16222_2";
-
-create table t1 (
-  pk serial, vb tinyblob as (b) virtual, b tinyblob,
-  primary key(pk), index (vb(64)))
-engine innodb;
-
-insert ignore into t1 (b) values ('foo');
-
-select * into outfile 'load.data' from t1;
-load data infile 'load.data' replace into table t1;
-
-set debug_sync= "now WAIT_FOR latch_released";
-set global debug_dbug= "-d,ib_purge_virtual_mdev_16222_1";
-drop table t1;
---remove_file $datadir/test/load.data
-
-set debug_sync= "now SIGNAL drop_started WAIT_FOR got_no_such_table";
-
-create table t1 (
-  pk serial, vb tinyblob as (b) virtual, b tinyblob,
-  primary key(pk), index (vb(64)))
-engine innodb;
-
-insert ignore into t1 (b) values ('foo');
-
-select * into outfile 'load.data' from t1;
-load data infile 'load.data' replace into table t1;
-
-set debug_sync= "now WAIT_FOR got_no_such_table";
-
-# FIXME: Race condition here:
-# 1. purge thread goes into sending got_no_such_table
-# 2. test thread finishes debug_sync= "RESET" below
-# 3. purge thread sends got_no_such_table
-set global debug_dbug= @saved_dbug;
-
-# cleanup
-drop table t1;
---remove_file $datadir/test/load.data
-
 --source include/wait_until_count_sessions.inc
 set debug_sync=reset;
 SET GLOBAL innodb_purge_rseg_truncate_frequency = @saved_frequency;
diff --git a/mysql-test/suite/innodb/r/instant_alter_purge.result b/mysql-test/suite/innodb/r/instant_alter_purge.result
index a3643610f04..1179ff62ecc 100644
--- a/mysql-test/suite/innodb/r/instant_alter_purge.result
+++ b/mysql-test/suite/innodb/r/instant_alter_purge.result
@@ -21,26 +21,4 @@ ALTER TABLE t1 DROP extra;
 disconnect prevent_purge;
 InnoDB		0 transactions not purged
 DROP TABLE t1;
-#
-# MDEV-17813 Crash in instant ALTER TABLE due to purge
-# concurrently emptying table
-#
-CREATE TABLE t1 (f2 INT) ENGINE=InnoDB;
-INSERT INTO t1 SET f2=1;
-ALTER TABLE t1 ADD COLUMN f1 INT;
-connect  purge_control,localhost,root;
-START TRANSACTION WITH CONSISTENT SNAPSHOT;
-connection default;
-DELETE FROM t1;
-SET DEBUG_SYNC='innodb_commit_inplace_alter_table_enter SIGNAL go WAIT_FOR do';
-ALTER TABLE t1 ADD COLUMN f3 INT;
-connection purge_control;
-SET DEBUG_SYNC='now WAIT_FOR go';
-COMMIT;
-InnoDB		0 transactions not purged
-SET DEBUG_SYNC='now SIGNAL do';
-disconnect purge_control;
-connection default;
-SET DEBUG_SYNC=RESET;
-DROP TABLE t1;
 SET GLOBAL innodb_purge_rseg_truncate_frequency = @saved_frequency;
diff --git a/mysql-test/suite/innodb/r/row_format_redundant.result b/mysql-test/suite/innodb/r/row_format_redundant.result
index 2df6eaceb34..0b42d297395 100644
--- a/mysql-test/suite/innodb/r/row_format_redundant.result
+++ b/mysql-test/suite/innodb/r/row_format_redundant.result
@@ -76,7 +76,7 @@ DROP TABLE t1;
 Warnings:
 Warning	1932	Table 'test.t1' doesn't exist in engine
 DROP TABLE t2,t3;
-FOUND 50 /\[ERROR\] InnoDB: Table `test`\.`t1` in InnoDB data dictionary contains invalid flags\. SYS_TABLES\.TYPE=1 SYS_TABLES\.MIX_LEN=511\b/ in mysqld.1.err
+FOUND 49 /\[ERROR\] InnoDB: Table `test`\.`t1` in InnoDB data dictionary contains invalid flags\. SYS_TABLES\.TYPE=1 SYS_TABLES\.MIX_LEN=511\b/ in mysqld.1.err
 # restart
 ib_buffer_pool
 ib_logfile0
diff --git a/mysql-test/suite/innodb/t/instant_alter_purge.test b/mysql-test/suite/innodb/t/instant_alter_purge.test
index 152d200d977..9ccf3347d7b 100644
--- a/mysql-test/suite/innodb/t/instant_alter_purge.test
+++ b/mysql-test/suite/innodb/t/instant_alter_purge.test
@@ -34,42 +34,4 @@ disconnect prevent_purge;
 let $wait_all_purged= 0;
 --source include/wait_all_purged.inc
 DROP TABLE t1;
-
---echo #
---echo # MDEV-17813 Crash in instant ALTER TABLE due to purge
---echo # concurrently emptying table
---echo #
-CREATE TABLE t1 (f2 INT) ENGINE=InnoDB;
-INSERT INTO t1 SET f2=1;
-ALTER TABLE t1 ADD COLUMN f1 INT;
-
-connect (purge_control,localhost,root);
-START TRANSACTION WITH CONSISTENT SNAPSHOT;
-
-connection default;
-DELETE FROM t1;
-
-if ($have_debug) {
-SET DEBUG_SYNC='innodb_commit_inplace_alter_table_enter SIGNAL go WAIT_FOR do';
-}
-send ALTER TABLE t1 ADD COLUMN f3 INT;
-
-connection purge_control;
-if ($have_debug) {
-SET DEBUG_SYNC='now WAIT_FOR go';
-}
-COMMIT;
---source include/wait_all_purged.inc
-if ($have_debug) {
-SET DEBUG_SYNC='now SIGNAL do';
-}
-disconnect purge_control;
-
-connection default;
-reap;
-if ($have_debug) {
-SET DEBUG_SYNC=RESET;
-}
-DROP TABLE t1;
-
 SET GLOBAL innodb_purge_rseg_truncate_frequency = @saved_frequency;