MDEV-10425 Assertion `collation.derivation == DERIVATION_IMPLICIT' failed in Item_func_conv_charset::fix_length_and_dec()

MDEV-10850 Wrong result for WHERE .. (f2=TO_BASE64('test') OR f2=TO_BASE64('TEST'))

Problem N1: MDEV-10425
Item_func_{md5|sha|sha2}::fix_length_and_dec() changed args[0]->collation
to force binary comparison in args[0]->eq().
It was done to treat e.g. MD5('a') and MD5('A') as different values.
It is wrong for a Item_func_xxx to modify its arguments.
Item_func_conv_charset did not expect that and crashed on assert.

Problem N2: MDEV-10850
Item_func_to_base64, Item_func_password, Item_func_hex are also case sensitive
hash functions, but they did not compare their arguments as binary.

Solution:
- Removing the code changing args[0]->collation
- Introducing Item_str_ascii_checksum_func as a common parent
  for Item_func_{md5|sha|sha2|password|hex|to_base64}
  and overriding its eq() method to compare arguments binary.

1 files changed, 24 insertions, 0 deletions

diff --git a/mysql-test/include/func_str_ascii_checksum.inc b/mysql-test/include/func_str_ascii_checksum.inc
new file mode 100644
index 00000000000..8e51c92c1ac
--- /dev/null
+++ b/mysql-test/include/func_str_ascii_checksum.inc
@@ -0,0 +1,24 @@
+--echo # Start of func_str_ascii_checksum.inc
+
+--echo #
+--echo # MDEV-10850 Wrong result for WHERE .. (f2=TO_BASE64('test') OR f2=TO_BASE64('TEST'))
+--echo #
+
+--eval CREATE TABLE t1 (f1 VARCHAR(4), f2 VARCHAR(255), UNIQUE KEY k1 (f1,f2))
+--eval INSERT INTO t1 VALUES ('test',$func('test')), ('TEST', $func('TEST'))
+--eval SELECT * FROM t1 IGNORE INDEX(k1) WHERE f1='test' AND (f2= $func("test") OR f2= $func("TEST"))
+--eval SELECT * FROM t1                  WHERE f1='test' AND (f2= $func("test") OR f2= $func("TEST"))
+--eval SELECT * FROM t1                  WHERE f1='test' AND (f2= $func("TEST") OR f2= $func("test"))
+DROP TABLE t1;
+
+
+--echo #
+--echo # MDEV-10425 Assertion `collation.derivation == DERIVATION_IMPLICIT' failed in Item_func_conv_charset::fix_length_and_dec()
+--echo #
+
+--eval PREPARE stmt FROM "SELECT $func(CONVERT('foo' USING latin1))"
+EXECUTE stmt;
+DEALLOCATE PREPARE stmt;
+
+
+--echo # End of func_str_ascii_checksum.inc