diff options
| author | Varun Gupta <varun.gupta@mariadb.com> | 2020-10-30 14:56:57 +0530 |
|---|---|---|
| committer | Marko Mäkelä <marko.makela@mariadb.com> | 2020-10-30 12:22:01 +0200 |
| commit | 5a0c34e4c2fd951119efb432eedcaa65a1d36606 (patch) | |
| tree | 5e6e21e19580aa9ac77c62b9a6409c6eca3c7d29 /sql/filesort.cc | |
| parent | 5482d62760bcbdcf44f1340fb5846c3942419dc5 (diff) | |
| download | mariadb-git-5a0c34e4c2fd951119efb432eedcaa65a1d36606.tar.gz | |
MDEV-24033: SIGSEGV in __memcmp_avx2_movbe from queue_insert | SIGSEGV in __memcmp_avx2_movbe from native_compare
The issue here was the system variable max_sort_length was being applied
to decimals and it was truncating the value for decimals to the number
of bytes set by max_sort_length.
This was leading to a buffer overflow as the values were written
to the buffer without truncation and then we moved the offset to
the number of bytes(set by max_sort_length), that are needed for comparison.
The fix is to not apply max_sort_length for fixed size types like INT,
DECIMALS and only apply max_sort_length for CHAR, VARCHARS, TEXT and
BLOBS.
Diffstat (limited to 'sql/filesort.cc')
| -rw-r--r-- | sql/filesort.cc | 14 |
1 files changed, 13 insertions, 1 deletions
diff --git a/sql/filesort.cc b/sql/filesort.cc index d76c39c3bd4..96f9aa874da 100644 --- a/sql/filesort.cc +++ b/sql/filesort.cc @@ -1971,7 +1971,14 @@ sortlength(THD *thd, SORT_FIELD *sortorder, uint s_length, if (sortorder->field) { CHARSET_INFO *cs= sortorder->field->sort_charset(); + sortorder->type= sortorder->field->is_packable() ? + SORT_FIELD_ATTR::VARIABLE_SIZE : + SORT_FIELD_ATTR::FIXED_SIZE; + sortorder->length= sortorder->field->sort_length(); + if (sortorder->is_variable_sized()) + set_if_smaller(sortorder->length, thd->variables.max_sort_length); + if (use_strnxfrm((cs=sortorder->field->sort_charset()))) { *multi_byte_charset= true; @@ -1982,6 +1989,10 @@ sortlength(THD *thd, SORT_FIELD *sortorder, uint s_length, } else { + sortorder->type= sortorder->item->type_handler()->is_packable() ? + SORT_FIELD_ATTR::VARIABLE_SIZE : + SORT_FIELD_ATTR::FIXED_SIZE; + sortorder->item->sortlength(thd, sortorder->item, sortorder); if (use_strnxfrm(sortorder->item->collation.collation)) { @@ -1990,7 +2001,8 @@ sortlength(THD *thd, SORT_FIELD *sortorder, uint s_length, if (sortorder->item->maybe_null) length++; // Place for NULL marker } - set_if_smaller(sortorder->length, thd->variables.max_sort_length); + if (sortorder->is_variable_sized()) + set_if_smaller(sortorder->length, thd->variables.max_sort_length); length+=sortorder->length; } sortorder->field= (Field*) 0; // end marker |