WL#874 "Extended LOAD DATA".

Now one can use user variables as target for data loaded from file
(besides table's columns). Also LOAD DATA got new SET-clause in which
one can specify values for table columns as expressions.

For example the following is possible:
LOAD DATA INFILE 'words.dat' INTO TABLE t1 (a, @b) SET c = @b + 1;

This patch also implements new way of replicating LOAD DATA.
Now we do it similarly to other queries.
We store LOAD DATA query in new Execute_load_query event
(which is last in the sequence of events representing LOAD DATA).
When we are executing this event we simply rewrite part of query which
holds name of file (we use name of temporary file) and then execute it
as usual query. In the beggining of this sequence we use Begin_load_query
event which is almost identical to Append_file event


client/mysqlbinlog.cc:
  Added support of two new binary log events Begin_load_query_log_event and
  Execute_load_query_log_Event which are used to replicate LOAD DATA INFILE.
mysql-test/r/ctype_ucs.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
mysql-test/r/insert_select.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
mysql-test/r/loaddata.result:
  Added tests for new LOAD DATA features.
mysql-test/r/mix_innodb_myisam_binlog.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results (don't dare to get rid from binlog positions
  completely since it seems that this test uses them).
mysql-test/r/mysqlbinlog.result:
  New approach for binlogging of LOAD DATA statement. Now we store it as
  usual query and rewrite part in which file is specified when needed.
  So now mysqlbinlog output for LOAD DATA much more closer to its initial
  form. Updated test'd results accordingly.
mysql-test/r/mysqldump.result:
  Made test more robust to other tests failures.
mysql-test/r/rpl000015.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_change_master.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results.
mysql-test/r/rpl_charset.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly
mysql-test/r/rpl_deadlock.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly
mysql-test/r/rpl_error_ignored_table.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
mysql-test/r/rpl_flush_log_loop.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_flush_tables.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
mysql-test/r/rpl_loaddata.result:
  New way of replicating LOAD DATA. Now we do it similarly to other
  queries. We store LOAD DATA query in new Execute_load_query event
  (which is last in the sequence of events representing LOAD DATA).
  When we are executing this event we simply rewrite part of query which
  holds name of file (we use name of temporary file) and then execute it
  as usual query. In the beggining of this sequence we use Begin_load_query
  event which is almost identical to Append_file event...
  
  Updated test's results wwith new binlog positions.
mysql-test/r/rpl_loaddata_rule_m.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
  Since now LOAD DATA is replicated much in the same way as usual query
  --binlog_do/ignore_db work for it inthe same way as for usual queries.
mysql-test/r/rpl_loaddata_rule_s.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_loaddatalocal.result:
  Added nice test for case when it is important that LOAD DATA LOCAL
  ignores duplicates.
mysql-test/r/rpl_log.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly (don't dare to get rid from binlog 
  positions completely since it seems that this test uses them).
mysql-test/r/rpl_log_pos.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_max_relay_size.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_multi_query.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_relayrotate.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_replicate_do.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_reset_slave.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_rotate_logs.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_server_id1.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_server_id2.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly.
mysql-test/r/rpl_temporary.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
mysql-test/r/rpl_timezone.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
mysql-test/r/rpl_until.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results accordingly and tweaked test a bit to bring it
  back to good shape.
mysql-test/r/rpl_user_variables.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
mysql-test/r/user_var.result:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test's results and made it more robust for future similar 
  changes.
mysql-test/t/ctype_ucs.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly and made it more robust for future similar
  changes.
mysql-test/t/insert_select.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly and made it more robust for future similar
  changes.
mysql-test/t/loaddata.test:
  Added test cases for new LOAD DATA functionality.
mysql-test/t/mix_innodb_myisam_binlog.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly.
mysql-test/t/mysqlbinlog.test:
  New way of replicating LOAD DATA local. Now we do it similarly to other
  queries. We store LOAD DATA query in new Execute_load_query event
  (which is last in the sequence of events representing LOAD DATA).
  When we are executing this event we simply rewrite part of query which
  holds name of file (we use name of temporary file) and then execute it
  as usual query. In the beggining of this sequence we use Begin_load_query
  event which is almost identical to Append_file event...
  
  Thus we need new binlog positions for LOAD DATA events.
mysql-test/t/mysqlbinlog2.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly.
mysql-test/t/mysqldump.test:
  Made test more robust for failures of other tests.
mysql-test/t/rpl_charset.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly.
mysql-test/t/rpl_deadlock.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly.
mysql-test/t/rpl_error_ignored_table.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly and made it more robust for future similar
  changes.
mysql-test/t/rpl_flush_tables.test:
  Addition of two new types of binary log events shifted binlog positions.
  Made test more robust for future similar changes.
mysql-test/t/rpl_loaddata.test:
  New way of replicating LOAD DATA. Now we do it similarly to other
  queries. We store LOAD DATA query in new Execute_load_query event
  (which is last in the sequence of events representing LOAD DATA).
  When we are executing this event we simply rewrite part of query which
  holds name of file (we use name of temporary file) and then execute it
  as usual query. In the beggining of this sequence we use Begin_load_query
  event which is almost identical to Append_file event...
  
  Apropritely updated comments in test.
mysql-test/t/rpl_loaddata_rule_m.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly and made it more robust for future similar 
  changes.
  Since now LOAD DATA is replicated much in the same way as usual query
  --binlog_do/ignore_db work for it inthe same way as for usual queries.
mysql-test/t/rpl_loaddata_rule_s.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly.
mysql-test/t/rpl_loaddatalocal.test:
  Added nice test for case when it is important that LOAD DATA LOCAL
  ignores duplicates.
mysql-test/t/rpl_log.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly (don't dare to get rid from binlog positions
  completely since it seems that this test uses them).
mysql-test/t/rpl_log_pos.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly.
mysql-test/t/rpl_multi_query.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly.
mysql-test/t/rpl_temporary.test:
  Addition of two new types of binary log events shifted binlog positions.
  Made test more robust for future similar changes.
mysql-test/t/rpl_timezone.test:
  Addition of two new types of binary log events shifted binlog positions.
  Made test more robust for future similar changes.
mysql-test/t/rpl_until.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly and tweaked it a bit to bring it back to good
  shape.
mysql-test/t/rpl_user_variables.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly and made it more robust for future similar
  changes.
mysql-test/t/user_var.test:
  Addition of two new types of binary log events shifted binlog positions.
  Updated test accordingly and made it more robust for future similar
  changes.
sql/item_func.cc:
  Added Item_user_var_as_out_param class that represents user variable
  which used as out parameter in LOAD DATA.
  
  Moved code from Item_func_set_user_var::update_hash() function to
  separate static function to be able to reuse it in this new class.
sql/item_func.h:
  Added Item_user_var_as_out_param class that represents user variable
  which used as out parameter in LOAD DATA.
sql/log_event.cc:
  New way of replicating LOAD DATA. Now we do it similarly to other
  queries. We store LOAD DATA query in new Execute_load_query event
  (which is last in the sequence of events representing LOAD DATA).
  When we are executing this event we simply rewrite part of query which
  holds name of file (we use name of temporary file) and then execute it
  as usual query. In the beggining of this sequence we use Begin_load_query
  event which is almost identical to Append_file event.
sql/log_event.h:
  New way of replicating LOAD DATA. Now we do it similarly to other
  queries. We store LOAD DATA query in new Execute_load_query event
  (which is last in the sequence of events representing LOAD DATA).
  When we are executing this event we simply rewrite part of query which
  holds name of file (we use name of temporary file) and then execute it
  as usual query. In the beggining of this sequence we use Begin_load_query
  event which is almost identical to Append_file event.
sql/mysql_priv.h:
  Now mysql_load() has two more arguments. They are needed to pass list of
  columns and corresponding expressions from new LOAD DATA's SET clause.
sql/share/errmsg.txt:
  Added new error message which is used to forbid loading of data from
  fixed length rows to variables.
sql/sql_lex.h:
  Added LEX::fname_start/fname_end members. 
  They are pointers to part of LOAD DATA statement which should be
  rewritten during replication (file name + little extra).
sql/sql_load.cc:
  Added support for extended LOAD DATA. 
  Now one can use user variables as target for data loaded from file 
  (besides table's columns). Also LOAD DATA got new SET-clause in which
  one can specify values for table columns as expressions.
  
  Updated mysql_load()/read_fixed_length()/read_sep_field() to support
  this functionality (now they can read data from file to both columns and
  variables and assign do calculations and assignments specified in SET
  clause).
  
  We also use new approach for LOAD DATA binlogging/replication.
sql/sql_parse.cc:
  mysql_execute_command():
    Since now we have SET clause in LOAD DATA we should also check
    permissions for tables used in its expressions. Also mysql_load()
    has two more arguments to pass information about this clause.
sql/sql_repl.cc:
  New way of replicating LOAD DATA. Now we do it similarly to other
  queries. We store LOAD DATA query in new Execute_load_query event
  (which is last in the sequence of events representing LOAD DATA).
  When we are executing this event we simply rewrite part of query which
  holds name of file (we use name of temporary file) and then execute it
  as usual query. In the beggining of this sequence we use Begin_load_query
  event which is almost identical to Append_file event.
sql/sql_repl.h:
  struct st_load_file_info:
    Removed memebers which are no longer needed for LOAD DATA binnlogging.
sql/sql_yacc.yy:
  Added support for extended LOAD DATA syntax. Now one can use
  user variables as target for data loaded from file (besides table's 
  columns). Also LOAD DATA got new SET-clause in which one can specify
  values for table columns as expressions.
  
  For example the following is possible:
  LOAD DATA INFILE 'words.dat' INTO TABLE t1 (a, @b) SET c = @b + 1;
  
  Also now we save pointers to the beginning and to the end of part of 
  LOAD DATA statement which should be rewritten during replication.

1 files changed, 121 insertions, 5 deletions

diff --git a/sql/log_event.h b/sql/log_event.h
index 7a041959f92..43a801da851 100644
--- a/sql/log_event.h
+++ b/sql/log_event.h
@@ -163,10 +163,12 @@ struct sql_ex_info
 
   See the #defines below for the format specifics.
 
-  The events which really update data are Query_log_event and
-  Load_log_event/Create_file_log_event/Execute_load_log_event (these 3 act
-  together to replicate LOAD DATA INFILE, with the help of
-  Append_block_log_event which prepares temporary files to load into the table).
+  The events which really update data are Query_log_event,
+  Execute_load_query_log_event and old Load_log_event and
+  Execute_load_log_event events (Execute_load_query is used together with
+  Begin_load_query and Append_block events to replicate LOAD DATA INFILE.
+  Create_file/Append_block/Execute_load (which includes Load_log_event)
+  were used to replicate LOAD DATA before the 5.0.3).
 
  ****************************************************************************/
 
@@ -194,6 +196,8 @@ struct sql_ex_info
 #define EXEC_LOAD_HEADER_LEN   4
 #define DELETE_FILE_HEADER_LEN 4
 #define FORMAT_DESCRIPTION_HEADER_LEN (START_V3_HEADER_LEN+1+LOG_EVENT_TYPES)
+#define EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN (4 + 4 + 4 + 1)
+#define EXECUTE_LOAD_QUERY_HEADER_LEN  (QUERY_HEADER_LEN + EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN)
 
 /*
    Event header offsets;
@@ -284,6 +288,12 @@ struct sql_ex_info
 /* DF = "Delete File" */
 #define DF_FILE_ID_OFFSET  0
 
+/* ELQ = "Execute Load Query" */
+#define ELQ_FILE_ID_OFFSET QUERY_HEADER_LEN
+#define ELQ_FN_POS_START_OFFSET ELQ_FILE_ID_OFFSET + 4
+#define ELQ_FN_POS_END_OFFSET ELQ_FILE_ID_OFFSET + 8
+#define ELQ_DUP_HANDLING_OFFSET ELQ_FILE_ID_OFFSET + 12
+
 /* 4 bytes which all binlogs should begin with */
 #define BINLOG_MAGIC        "\xfe\x62\x69\x6e"
 
@@ -387,6 +397,8 @@ enum Log_event_type
   RAND_EVENT, USER_VAR_EVENT,
   FORMAT_DESCRIPTION_EVENT,
   XID_EVENT,
+  BEGIN_LOAD_QUERY_EVENT,
+  EXECUTE_LOAD_QUERY_EVENT,
 
   /*
     add new events here - right above this comment!
@@ -711,13 +723,17 @@ public:
 #ifdef HAVE_REPLICATION
   void pack_info(Protocol* protocol);
   int exec_event(struct st_relay_log_info* rli);
+  int exec_event(struct st_relay_log_info* rli, const char *query_arg,
+                 uint32 q_len_arg);
 #endif /* HAVE_REPLICATION */
 #else
+  void print_query_header(FILE* file, bool short_form = 0, LAST_EVENT_INFO* last_event_info= 0);
   void print(FILE* file, bool short_form = 0, LAST_EVENT_INFO* last_event_info= 0);
 #endif
 
   Query_log_event(const char* buf, uint event_len,
-                  const Format_description_log_event *description_event);
+                  const Format_description_log_event *description_event,
+                  Log_event_type event_type);
   ~Query_log_event()
   {
     if (data_buf)
@@ -728,6 +744,14 @@ public:
   Log_event_type get_type_code() { return QUERY_EVENT; }
   bool write(IO_CACHE* file);
   bool is_valid() const { return query != 0; }
+
+  /*
+    Returns number of bytes additionaly written to post header by derived
+    events (so far it is only Execute_load_query event).
+  */
+  virtual ulong get_post_header_size_for_derived() { return 0; }
+  /* Writes derived event-specific part of post header. */
+  virtual bool write_post_header_for_derived(IO_CACHE* file) { return FALSE; }
 };
 
 #ifdef HAVE_REPLICATION
@@ -779,6 +803,10 @@ public:
  ****************************************************************************/
 class Load_log_event: public Log_event
 {
+private:
+  uint get_query_buffer_length();
+  void print_query(bool need_db, char *buf, char **end,
+                   char **fn_start, char **fn_end);
 protected:
   int copy_log_event(const char *buf, ulong event_len,
                      int body_offset, const Format_description_log_event* description_event);
@@ -1312,6 +1340,7 @@ public:
 #ifdef HAVE_REPLICATION
   int exec_event(struct st_relay_log_info* rli);
   void pack_info(Protocol* protocol);
+  virtual int get_open_mode() const;
 #endif /* HAVE_REPLICATION */
 #else
   void print(FILE* file, bool short_form = 0, LAST_EVENT_INFO* last_event_info= 0);
@@ -1395,6 +1424,93 @@ public:
 };
 
 
+/***************************************************************************
+
+  Begin load query Log Event class
+
+  Event for the first block of file to be loaded, its only difference from
+  Append_block event is that this event creates or truncates existing file
+  before writing data.
+
+****************************************************************************/
+class Begin_load_query_log_event: public Append_block_log_event
+{
+public:
+#ifndef MYSQL_CLIENT
+  Begin_load_query_log_event(THD* thd_arg, const char *db_arg,
+                             char* block_arg, uint block_len_arg,
+                             bool using_trans);
+#ifdef HAVE_REPLICATION
+  Begin_load_query_log_event(THD* thd);
+  int get_open_mode() const;
+#endif /* HAVE_REPLICATION */
+#endif
+  Begin_load_query_log_event(const char* buf, uint event_len,
+                             const Format_description_log_event* description_event);
+  ~Begin_load_query_log_event() {}
+  Log_event_type get_type_code() { return BEGIN_LOAD_QUERY_EVENT; }
+};
+
+
+/*
+  Elements of this enum describe how LOAD DATA handles duplicates.
+*/
+enum enum_load_dup_handling { LOAD_DUP_ERROR= 0, LOAD_DUP_IGNORE,
+                              LOAD_DUP_REPLACE };
+
+/****************************************************************************
+
+  Execute load query Log Event class
+
+  Event responsible for LOAD DATA execution, it similar to Query_log_event
+  but before executing the query it substitutes original filename in LOAD DATA
+  query with name of temporary file.
+
+****************************************************************************/
+class Execute_load_query_log_event: public Query_log_event
+{
+public:
+  uint file_id;       // file_id of temporary file
+  uint fn_pos_start;  // pointer to the part of the query that should
+                      // be substituted
+  uint fn_pos_end;    // pointer to the end of this part of query
+  /*
+    We have to store type of duplicate handling explicitly, because
+    for LOAD DATA it also depends on LOCAL option. And this part
+    of query will be rewritten during replication so this information
+    may be lost...
+  */
+  enum_load_dup_handling dup_handling;
+
+#ifndef MYSQL_CLIENT
+  Execute_load_query_log_event(THD* thd, const char* query_arg,
+                       ulong query_length, uint fn_pos_start_arg,
+                       uint fn_pos_end_arg,
+                       enum_load_dup_handling dup_handling_arg,
+                       bool using_trans, bool suppress_use);
+#ifdef HAVE_REPLICATION
+  void pack_info(Protocol* protocol);
+  int exec_event(struct st_relay_log_info* rli);
+#endif /* HAVE_REPLICATION */
+#else
+  void print(FILE* file, bool short_form = 0,
+             LAST_EVENT_INFO* last_event_info= 0);
+  /* Prints the query as LOAD DATA LOCAL and with rewritten filename */
+  void print(FILE* file, bool short_form, LAST_EVENT_INFO* last_event_info,
+             const char *local_fname);
+#endif
+  Execute_load_query_log_event(const char* buf, uint event_len,
+                               const Format_description_log_event *description_event);
+  ~Execute_load_query_log_event() {}
+
+  Log_event_type get_type_code() { return EXECUTE_LOAD_QUERY_EVENT; }
+  bool is_valid() const { return Query_log_event::is_valid() && file_id != 0; }
+
+  ulong get_post_header_size_for_derived();
+  bool write_post_header_for_derived(IO_CACHE* file);
+ };
+
+
 #ifdef MYSQL_CLIENT
 class Unknown_log_event: public Log_event
 {