Fix for bug #30885: MEMORY returns incorrect data if BTREE index is used for NULL lookup

Problem: creating an rb-tree key we store length (2 bytes) before the actual data for 
varchar key parts. The fact was missed for NULL key parts, when we set NULL byte and 
skip the rest.

Fix: take into account the length of the varchar key parts for NULLs.


heap/hp_hash.c:
  Fix for bug #30885: MEMORY returns incorrect data if BTREE index is used for NULL lookup
    - skip varchar key parts length (2 byte) for null key parts.
mysql-test/r/heap_btree.result:
  Fix for bug #30885: MEMORY returns incorrect data if BTREE index is used for NULL lookup
    -test result.
mysql-test/t/heap_btree.test:
  Fix for bug #30885: MEMORY returns incorrect data if BTREE index is used for NULL lookup
    - test case.

1 files changed, 6 insertions, 0 deletions

diff --git a/heap/hp_hash.c b/heap/hp_hash.c
index d8eee9c794c..251be1a3fe2 100644
--- a/heap/hp_hash.c
+++ b/heap/hp_hash.c
@@ -808,6 +808,12 @@ uint hp_rb_pack_key(HP_KEYDEF *keydef, uchar *key, const uchar *old,
       if (!(*key++= (char) 1 - *old++))
       {
         k_len-= seg->length;
+        /* 
+          Take into account length (2 bytes) of varchar key parts 
+          stored before the data.
+         */
+        if (seg->flag & (HA_VAR_LENGTH_PART | HA_BLOB_PART))
+          k_len-= 2;
         continue;
       }
     }