MDEV-9500 Bug after upgrade to 10.1.10 (and 10.1.11)

Case: table with a NOT NULL field, BEFORE UPDATE trigger,
and UPDATE with a subquery that uses GROUP BY on that
NOT NULL field, and needs a temporary table for it.

Because of the BEFORE trigger, the field becomes nullable
temporarily. But its Item_field (used in GROUP BY) doesn't.
When working with the temptable some code looked at
item->maybe_null, some - at field->null_ptr.
The fix: make Item_field nullable when its field is.

This triggers an assert. The group key size is calculated
before the item is made nullable, so the group key doesn't
have a null byte. The fix: make fields/items nullable
before the group key size is calculated.

1 files changed, 3 insertions, 2 deletions

diff --git a/sql/sql_update.cc b/sql/sql_update.cc
index f343a17ee11..05c3ae1a666 100644
--- a/sql/sql_update.cc
+++ b/sql/sql_update.cc
@@ -368,6 +368,9 @@ int mysql_update(THD *thd,
   if (check_unique_table(thd, table_list))
     DBUG_RETURN(TRUE);
 
+  switch_to_nullable_trigger_fields(fields, table);
+  switch_to_nullable_trigger_fields(values, table);
+
   /* Apply the IN=>EXISTS transformation to all subqueries and optimize them. */
   if (select_lex->optimize_unflattened_subqueries(false))
     DBUG_RETURN(TRUE);
@@ -455,8 +458,6 @@ int mysql_update(THD *thd,
   }
   init_ftfuncs(thd, select_lex, 1);
 
-  switch_to_nullable_trigger_fields(fields, table);
-  switch_to_nullable_trigger_fields(values, table);
   table->mark_columns_needed_for_update();
 
   table->update_const_key_parts(conds);