Bug#24985 - UTF8 ENUM primary key on MEMORY using BTREE

            causes incorrect duplicate entries

Keys for BTREE indexes on ENUM and SET columns of MEMORY tables
with character set UTF8 were computed incorrectly. Many
different column values got the same key value.

Apart of possible performance problems, it made unique indexes
of this type unusable because it rejected many different
values as duplicates.

The problem was that multibyte character detection was tried
on the internal numeric column value. Many values were not
identified as characters. Their key value became blank filled.

Thanks to Alexander Barkov and Ramil Kalimullin for the patch,
which sets the character set of ENUM and SET key segments to
the pseudo binary character set.


mysql-test/r/heap_btree.result:
  Bug#24985 - UTF8 ENUM primary key on MEMORY using BTREE
              causes incorrect duplicate entries
  Added test result.
mysql-test/t/heap_btree.test:
  Bug#24985 - UTF8 ENUM primary key on MEMORY using BTREE
              causes incorrect duplicate entries
  Added test.
sql/ha_heap.cc:
  Bug#24985 - UTF8 ENUM primary key on MEMORY using BTREE
              causes incorrect duplicate entries
  Set key segment charset to my_charset_bin for ENUM and SET
  columns.

1 files changed, 12 insertions, 0 deletions

diff --git a/mysql-test/r/heap_btree.result b/mysql-test/r/heap_btree.result
index e6492e90b80..512a8a52845 100644
--- a/mysql-test/r/heap_btree.result
+++ b/mysql-test/r/heap_btree.result
@@ -280,4 +280,16 @@ a
 1
 1
 drop table t1;
+CREATE TABLE t1 (
+c1 ENUM('1', '2'),
+UNIQUE USING BTREE(c1)
+) ENGINE= MEMORY DEFAULT CHARSET= utf8;
+INSERT INTO t1 VALUES('1'), ('2');
+DROP TABLE t1;
+CREATE TABLE t1 (
+c1 SET('1', '2'),
+UNIQUE USING BTREE(c1)
+) ENGINE= MEMORY DEFAULT CHARSET= utf8;
+INSERT INTO t1 VALUES('1'), ('2');
+DROP TABLE t1;
 End of 4.1 tests