Bug#23713 LOCK TABLES + CREATE TRIGGER + FLUSH TABLES WITH READ LOCK = deadlock

This bug is actually two bugs in one, one of which is CREATE TRIGGER under
LOCK TABLES and the other is CREATE TRIGGER under LOCK TABLES simultaneous
to a FLUSH TABLES WITH READ LOCK (global read lock). Both situations could
lead to a server crash or deadlock.

The first problem arises from the fact that when under LOCK TABLES, if the
table is in the set of locked tables, the table is already open and it doesn't
need to be reopened (not a placeholder). Also in this case, if the table is
not write locked, a exclusive lock can't be acquired because of a possible
deadlock with another thread also holding a (read) lock on the table. The
second issue arises from the fact that one should never wait for a global
read lock if it's holding any locked tables, because the global read lock
is waiting for these tables and this leads to a circular wait deadlock.

The solution for the first case is to check if the table is write locked
and upgraded the write lock to a exclusive lock and fail otherwise for non
write locked tables. Grabbin the exclusive lock in this case also means
to ensure that the table is opened only by the calling thread. The second
issue is partly fixed by not waiting for the global read lock if the thread
is holding any locked tables.

The second issue is only partly addressed in this patch because it turned
out to be much wider and also affects other DDL statements. Reported as
Bug#32395


mysql-test/r/trigger.result:
  Add test case result for Bug#23713
mysql-test/r/trigger_notembedded.result:
  Add test case result for Bug#23713
mysql-test/t/trigger.test:
  Add test case for Bug#23713
mysql-test/t/trigger_notembedded.test:
  Add test case for Bug#23713
sql/mysql_priv.h:
  Locally export wait_while_table_is_used and name_lock_locked_table
  and add flag to mysql_ha_rm_tables to signal that LOCK_open is locked.
sql/sql_base.cc:
  Introduce name_lock_locked_table function and match
  close_old_data_files function declaration and definition.
sql/sql_handler.cc:
  Add flag to mysql_ha_rm_tables to signal that LOCK_open is locked.
sql/sql_rename.cc:
  Fix mysql_ha_rm_tables caller.
sql/sql_table.cc:
  Export wait_while_table_is_used and assert that LOCK_open is locked
  and fix mysql_ha_rm_tables caller.
sql/sql_trigger.cc:
  Upgrade write locked tables to a exclusive lock and fail if
  the table is not write locked. Also, don't wait for the global
  read lock if under LOCK TABLES.

1 files changed, 47 insertions, 0 deletions

diff --git a/mysql-test/t/trigger.test b/mysql-test/t/trigger.test
index e1037d8bf9c..1aeac91e5ad 100644
--- a/mysql-test/t/trigger.test
+++ b/mysql-test/t/trigger.test
@@ -2257,4 +2257,51 @@ DROP TRIGGER IF EXISTS trg;
 --error ER_TRG_DOES_NOT_EXIST
 SHOW CREATE TRIGGER trg;
 
+#
+# Bug#23713 LOCK TABLES + CREATE TRIGGER + FLUSH TABLES WITH READ LOCK = deadlock
+#
+# Test of trigger creation and removal under LOCK TABLES
+#
+
+--disable_warnings
+drop table if exists t1;
+--enable_warnings
+
+create table t1 (i int, j int);
+
+create trigger t1_bi before insert on t1 for each row begin end;
+--error ER_NOT_SUPPORTED_YET
+create trigger t1_bi before insert on t1 for each row begin end;
+drop trigger t1_bi;
+--error ER_TRG_DOES_NOT_EXIST
+drop trigger t1_bi;
+
+lock tables t1 read;
+--error ER_TABLE_NOT_LOCKED_FOR_WRITE
+create trigger t1_bi before insert on t1 for each row begin end;
+--error ER_TABLE_NOT_LOCKED_FOR_WRITE
+create trigger t1_bi before insert on t1 for each row begin end;
+--error ER_TRG_DOES_NOT_EXIST
+drop trigger t1_bi;
+unlock tables;
+
+create trigger t1_bi before insert on t1 for each row begin end;
+lock tables t1 read;
+--error ER_TABLE_NOT_LOCKED_FOR_WRITE
+create trigger t1_bi before insert on t1 for each row begin end;
+--error ER_TABLE_NOT_LOCKED_FOR_WRITE
+drop trigger t1_bi;
+unlock tables;
+drop trigger t1_bi;
+
+lock tables t1 write;
+create trigger b1_bi before insert on t1 for each row set new.i = new.i + 10;
+insert into t1 values (10, 10);
+drop trigger b1_bi;
+insert into t1 values (10, 10);
+select * from t1;
+unlock tables;
+
+drop table t1;
+
 --echo End of 5.1 tests.