Fix all warnings given by UBSAN

The easiest way to compile and test the server with UBSAN is to run:
./BUILD/compile-pentium64-ubsan
and then run mysql-test-run.
After this commit, one should be able to run this without any UBSAN
warnings. There is still a few compiler warnings that should be fixed
at some point, but these do not expose any real bugs.

The 'special' cases where we disable, suppress or circumvent UBSAN are:
- ref10 source (as here we intentionally do some shifts that UBSAN
  complains about.
- x86 version of optimized int#korr() methods. UBSAN do not like unaligned
  memory access of integers.  Fixed by using byte_order_generic.h when
  compiling with UBSAN
- We use smaller thread stack with ASAN and UBSAN, which forced me to
  disable a few tests that prints the thread stack size.
- Verifying class types does not work for shared libraries. I added
  suppression in mysql-test-run.pl for this case.
- Added '#ifdef WITH_UBSAN' when using integer arithmetic where it is
  safe to have overflows (two cases, in item_func.cc).

Things fixed:
- Don't left shift signed values
  (byte_order_generic.h, mysqltest.c, item_sum.cc and many more)
- Don't assign not non existing values to enum variables.
- Ensure that bool and enum values are properly initialized in
  constructors.  This was needed as UBSAN checks that these types has
  correct values when one copies an object.
  (gcalc_tools.h, ha_partition.cc, item_sum.cc, partition_element.h ...)
- Ensure we do not called handler functions on unallocated objects or
  deleted objects.
  (events.cc, sql_acl.cc).
- Fixed bugs in Item_sp::Item_sp() where we did not call constructor
  on Query_arena object.
- Fixed several cast of objects to an incompatible class!
  (Item.cc, Item_buff.cc, item_timefunc.cc, opt_subselect.cc, sql_acl.cc,
   sql_select.cc ...)
- Ensure we do not do integer arithmetic that causes over or underflows.
  This includes also ++ and -- of integers.
  (Item_func.cc, Item_strfunc.cc, item_timefunc.cc, sql_base.cc ...)
- Added JSON_VALUE_UNITIALIZED to json_value_types and ensure that
  value_type is initialized to this instead of to -1, which is not a valid
  enum value for json_value_types.
- Ensure we do not call memcpy() when second argument could be null.
- Fixed that Item_func_str::make_empty_result() creates an empty string
  instead of a null string (safer as it ensures we do not do arithmetic
  on null strings).

Other things:

- Changed struct st_position to an OBJECT and added an initialization
  function to it to ensure that we do not copy or use uninitialized
  members. The change to a class was also motived that we used "struct
  st_position" and POSITION randomly trough the code which was
  confusing.
- Notably big rewrite in sql_acl.cc to avoid using deleted objects.
- Changed in sql_partition to use '^' instead of '-'. This is safe as
  the operator is either 0 or 0x8000000000000000ULL.
- Added check for select_nr < INT_MAX in JOIN::build_explain() to
  avoid bug when get_select() could return NULL.
- Reordered elements in POSITION for better alignment.
- Changed sql_test.cc::print_plan() to use pointers instead of objects.
- Fixed bug in find_set() where could could execute '1 << -1'.
- Added variable have_sanitizer, used by mtr.  (This variable was before
  only in 10.5 and up).  It can now have one of two values:
  ASAN or UBSAN.
- Moved ~Archive_share() from ha_archive.cc to ha_archive.h and marked
  it virtual. This was an effort to get UBSAN to work with loaded storage
  engines. I kept the change as the new place is better.
- Added in CONNECT engine COLBLK::SetName(), to get around a wrong cast
  in tabutil.cpp.
- Added HAVE_REPLICATION around usage of rgi_slave, to get embedded
  server to compile with UBSAN. (Patch from Marko).
- Added #ifdef for powerpc64 to avoid a bug in old gcc versions related
  to integer arithmetic.

Changes that should not be needed but had to be done to suppress warnings
from UBSAN:

- Added static_cast<<uint16_t>> around shift to get rid of a LOT of
  compiler warnings when using UBSAN.
- Had to change some '/' of 2 base integers to shift to get rid of
  some compile time warnings.

Reviewed by:
- Json changes: Alexey Botchkov
- Charset changes in ctype-uca.c: Alexander Barkov
- InnoDB changes & Embedded server: Marko Mäkelä
- sql_acl.cc changes: Vicențiu Ciorbaru
- build_explain() changes: Sergey Petrunia

1 files changed, 49 insertions, 48 deletions

diff --git a/sql/sql_select.h b/sql/sql_select.h
index 1d928334bd8..d21d1bcc305 100644
--- a/sql/sql_select.h
+++ b/sql/sql_select.h
@@ -698,8 +698,6 @@ end_write_group(JOIN *join, JOIN_TAB *join_tab __attribute__((unused)),
 		bool end_of_records);
 
 
-struct st_position;
-
 class Semi_join_strategy_picker
 {
 public:
@@ -710,7 +708,7 @@ public:
     Update internal state after another table has been added to the join
     prefix
   */
-  virtual void set_from_prev(struct st_position *prev) = 0;
+  virtual void set_from_prev(POSITION *prev) = 0;
   
   virtual bool check_qep(JOIN *join,
                          uint idx,
@@ -720,7 +718,7 @@ public:
                          double *read_time,
                          table_map *handled_fanout,
                          sj_strategy_enum *strategy,
-                         struct st_position *loose_scan_pos) = 0;
+                         POSITION *loose_scan_pos) = 0;
 
   virtual void mark_used() = 0;
 
@@ -751,7 +749,7 @@ public:
     first_dupsweedout_table= MAX_TABLES;
     is_used= FALSE;
   }
-  void set_from_prev(struct st_position *prev);
+  void set_from_prev(POSITION *prev);
   
   bool check_qep(JOIN *join,
                  uint idx,
@@ -761,7 +759,7 @@ public:
                  double *read_time,
                  table_map *handled_fanout,
                  sj_strategy_enum *stratey,
-                 struct st_position *loose_scan_pos);
+                 POSITION *loose_scan_pos);
 
   void mark_used() { is_used= TRUE; }
   friend void fix_semijoin_strategies_for_picked_join_order(JOIN *join);
@@ -797,7 +795,7 @@ public:
     is_used= FALSE;
   }
 
-  void set_from_prev(struct st_position *prev);
+  void set_from_prev(POSITION *prev);
   bool check_qep(JOIN *join,
                  uint idx,
                  table_map remaining_tables, 
@@ -806,7 +804,7 @@ public:
                  double *read_time,
                  table_map *handled_fanout,
                  sj_strategy_enum *strategy,
-                 struct st_position *loose_scan_pos);
+                 POSITION *loose_scan_pos);
 
   void mark_used() { is_used= TRUE; }
   friend void fix_semijoin_strategies_for_picked_join_order(JOIN *join);
@@ -815,6 +813,7 @@ public:
 
 class LooseScan_picker : public Semi_join_strategy_picker
 {
+public:
   /* The first (i.e. driving) table we're doing loose scan for */
   uint        first_loosescan_table;
   /* 
@@ -833,14 +832,13 @@ class LooseScan_picker : public Semi_join_strategy_picker
   uint loosescan_parts; /* Number of keyparts to be kept distinct */
   
   bool is_used;
-public:
   void set_empty()
   {
     first_loosescan_table= MAX_TABLES; 
     is_used= FALSE;
   }
 
-  void set_from_prev(struct st_position *prev);
+  void set_from_prev(POSITION *prev);
   bool check_qep(JOIN *join,
                  uint idx,
                  table_map remaining_tables, 
@@ -849,19 +847,19 @@ public:
                  double *read_time,
                  table_map *handled_fanout,
                  sj_strategy_enum *strategy,
-                 struct st_position *loose_scan_pos);
+                 POSITION *loose_scan_pos);
   void mark_used() { is_used= TRUE; }
 
   friend class Loose_scan_opt;
   friend void best_access_path(JOIN      *join,
                                JOIN_TAB  *s,
                                table_map remaining_tables,
-                               const struct st_position *join_positions,
+                               const POSITION *join_positions,
                                uint      idx,
                                bool      disable_jbuf,
                                double    record_count,
-                               struct st_position *pos,
-                               struct st_position *loose_scan_pos);
+                               POSITION *pos,
+                               POSITION *loose_scan_pos);
   friend bool get_best_combination(JOIN *join);
   friend int setup_semijoin_loosescan(JOIN *join);
   friend void fix_semijoin_strategies_for_picked_join_order(JOIN *join);
@@ -888,7 +886,7 @@ public:
     sjm_scan_last_inner= 0;
     is_used= FALSE;
   }
-  void set_from_prev(struct st_position *prev);
+  void set_from_prev(POSITION *prev);
   bool check_qep(JOIN *join,
                  uint idx,
                  table_map remaining_tables, 
@@ -897,7 +895,7 @@ public:
                  double *read_time,
                  table_map *handled_fanout,
                  sj_strategy_enum *strategy,
-                 struct st_position *loose_scan_pos);
+                 POSITION *loose_scan_pos);
   void mark_used() { is_used= TRUE; }
 
   friend void fix_semijoin_strategies_for_picked_join_order(JOIN *join);
@@ -912,8 +910,9 @@ class Rowid_filter;
   Information about a position of table within a join order. Used in join
   optimization.
 */
-typedef struct st_position
+class POSITION
 {
+public:
   /* The table that's put into join order */
   JOIN_TAB *table;
 
@@ -925,7 +924,7 @@ typedef struct st_position
   double records_read;
 
   /* The selectivity of the pushed down conditions */
-  double cond_selectivity; 
+  double cond_selectivity;
 
   /* 
     Cost accessing the table in course of the entire complete join execution,
@@ -934,8 +933,6 @@ typedef struct st_position
   */
   double read_time;
 
-  /* Cumulative cost and record count for the join prefix */
-  Cost_estimate prefix_cost;
   double    prefix_record_count;
 
   /*
@@ -944,29 +941,46 @@ typedef struct st_position
   */
   KEYUSE *key;
 
+  /* Info on splitting plan used at this position */
+  SplM_plan_info *spl_plan;
+
+  /* Cost info for the range filter used at this position */
+  Range_rowid_filter_cost_info *range_rowid_filter_info;
+
   /* If ref-based access is used: bitmap of tables this table depends on  */
   table_map ref_depend_map;
- 
+
   /*
-    TRUE <=> join buffering will be used. At the moment this is based on 
-    *very* imprecise guesses made in best_access_path(). 
+    Bitmap of semi-join inner tables that are in the join prefix and for
+    which there's no provision for how to eliminate semi-join duplicates
+    they produce.
   */
-  bool use_join_buffer;
- 
+  table_map dups_producing_tables;
+
+  table_map inner_tables_handled_with_other_sjs;
+
+  Duplicate_weedout_picker  dups_weedout_picker;
+  Firstmatch_picker         firstmatch_picker;
+  LooseScan_picker          loosescan_picker;
+  Sj_materialization_picker sjmat_picker;
+
+  /* Cumulative cost and record count for the join prefix */
+  Cost_estimate prefix_cost;
+
   /*
     Current optimization state: Semi-join strategy to be used for this
     and preceding join tables.
-    
+
     Join optimizer sets this for the *last* join_tab in the
-    duplicate-generating range. That is, in order to interpret this field, 
+    duplicate-generating range. That is, in order to interpret this field,
     one needs to traverse join->[best_]positions array from right to left.
     When you see a join table with sj_strategy!= SJ_OPT_NONE, some other
-    field (depending on the strategy) tells how many preceding positions 
+    field (depending on the strategy) tells how many preceding positions
     this applies to. The values of covered_preceding_positions->sj_strategy
     must be ignored.
   */
   enum sj_strategy_enum sj_strategy;
-  
+
   /*
     Valid only after fix_semijoin_strategies_for_picked_join_order() call:
     if sj_strategy!=SJ_OPT_NONE, this is the number of subsequent tables that
@@ -975,26 +989,12 @@ typedef struct st_position
   uint n_sj_tables;
 
   /*
-    Bitmap of semi-join inner tables that are in the join prefix and for
-    which there's no provision for how to eliminate semi-join duplicates
-    they produce.
+    TRUE <=> join buffering will be used. At the moment this is based on
+    *very* imprecise guesses made in best_access_path().
   */
-  table_map dups_producing_tables;
-
-  table_map inner_tables_handled_with_other_sjs;
-   
-  Duplicate_weedout_picker  dups_weedout_picker;
-  Firstmatch_picker         firstmatch_picker;
-  LooseScan_picker          loosescan_picker;
-  Sj_materialization_picker sjmat_picker;
-
-  /* Info on splitting plan used at this position */  
-  SplM_plan_info *spl_plan;
-
-  /* Cost info for the range filter used at this position */
-  Range_rowid_filter_cost_info *range_rowid_filter_info;
-
-} POSITION;
+  bool use_join_buffer;
+  POSITION();
+};
 
 typedef Bounds_checked_array<Item_null_result*> Item_null_array;
 
@@ -1590,6 +1590,7 @@ public:
       fields_list= fields_arg;
     non_agg_fields.empty();
     bzero((char*) &keyuse,sizeof(keyuse));
+    having_value= Item::COND_UNDEF;
     tmp_table_param.init();
     tmp_table_param.end_write_records= HA_POS_ERROR;
     rollup.state= ROLLUP::STATE_NONE;