diff options
author | Jan Lindström <jan.lindstrom@mariadb.com> | 2018-03-14 16:13:14 +0200 |
---|---|---|
committer | Jan Lindström <jan.lindstrom@mariadb.com> | 2018-03-14 16:13:14 +0200 |
commit | bc2c0e1e27b1d1003429dc5afad29281a0b63ccb (patch) | |
tree | 0ad47be9c29fa1975cbbab129f23593ae97ec4a9 /storage/xtradb/buf/buf0checksum.cc | |
parent | b4026efaacbfc935d3ce8fc0fdb0a88920c4f426 (diff) | |
download | mariadb-git-bc2c0e1e27b1d1003429dc5afad29281a0b63ccb.tar.gz |
MDEV-13103: InnoDB crash recovery fails to decompress a page in buf_dblwr_process()bb-10.1-MDEV-13103-MDEV-15032
There were several problems. Firstly, page decompression code did not handle
possible decompression errors correctly. Secondly, not all compression methods
tolerate corrupted input (e.g. lz4 did not tolerate input that was compressed
using snappy method). Finally, if page is actually also encrypted we can't
decompress page. Solutions: Add proper error handling to decompression code
and add post compression checksum to page. As whole page including page checksum
is compressed we can reuse the original checksum field for post compression
checksum. With post compression checksum we can detect most of the corruptions.
If no corruption is detected hopefully decompression code can detect
remaining problems.
Doublewrite buffer page recovery for page compressed pages require
that post compression checksum matches. For pages from old releases
supporting page compression checksum must be BUF_NO_CHECKSUM_MAGIC.
Upgrade from older versions is supported as post compression
checksum check accepts the BUF_NO_CHECKSUM_MAGIC that they stored
in checksum filed.
Downgrade to older versions is not supported (assuming that there
were some changes to compressed tables) as page compression code
would not tolerate any other checksum except BUF_NO_CHECKSUM_MAGIC.
innochecksum.cc is_page_corrupted:
If page is compressed verify post compression checksum
buf_page_decrypt_after_read
Return DB_PAGE_CORRUPTED if page is found to be corrupted
after post compression checksum check.
buf_page_io_complete
If page is found corrupted after buf_page_decrypt_after_read
there is no need to continue page check.
buf_page_decrypt_after_read
Verify post compression checksum before decompression and
if it does not match mark page corrupted. Note that old
compressed pages do not really have post compression
checksum so they are treated as not corrupted and then
we need to hope that decompression code can handle the
possible corruptions by returning error.
buf_calc_compressed_crc32
New function to calculate post compression checksum
so that necessary compression metadata fields are
included.
buf_dblwr_decompress
New function that handles post compression checksum check
and page decompression if it is ok.
buf_dblwr_process
Verify post compression checksum before trying to decompress
page.
fil_space_verify_crypt_checksum
Remove incorrect code as compressed and encrypted pages
do have post encryption checksum.
fil_compress_page
Calculate and store post compression checksum to FIL_SPACE_OR_CHKSUM
field as original value is stored on compressed image.
fil_decompress_page
Add error handling if decompression fails.
fil_verify_compression_checksum
New function to verify post compression checksum.
Compressed tablespaces before this change have BUF_NO_CHECKSUM_MAGIC
in checksum field and they must be treated as not corrupted.
fil_iterate
Add post compression checksum check before actual decompression.
convert_error_code_to_mysql
Handle also page corruptions DB_PAGE_CORRUPTED as HA_ERR_CRASHED.
Note that there are cases when we do not know for certain
is page corrupted, corrupted and compressed, or still encrypted
after failed decrypt, thus tablespace could be marked just corrupted.
Tests modified
innodb-page_compression_[zip, lz4, lzma, lzo, bzip2, snappy]
to use innodb-page-compression.inc
innodb-page-compression.inc add innochecksum and intentional tablespace
corruption tests.
innodb-force-corrupt, innodb_bug14147491 add new error
messages to mtr suppression and new error codes.
New tests
encryption/innodb-corrupted.test test intentionally corrupted
tablespaces containing encryption and compression.
doublewrite-compressed test doublewrite recovery for page
compressed tables
innodb-import import files from both big_endian and little_endian
machine
Diffstat (limited to 'storage/xtradb/buf/buf0checksum.cc')
-rw-r--r-- | storage/xtradb/buf/buf0checksum.cc | 45 |
1 files changed, 45 insertions, 0 deletions
diff --git a/storage/xtradb/buf/buf0checksum.cc b/storage/xtradb/buf/buf0checksum.cc index 01b646a78e0..98abba575ef 100644 --- a/storage/xtradb/buf/buf0checksum.cc +++ b/storage/xtradb/buf/buf0checksum.cc @@ -1,6 +1,7 @@ /***************************************************************************** Copyright (c) 1995, 2015, Oracle and/or its affiliates. All Rights Reserved. +Copyright (c) 2017, 2018, MariaDB Corporation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software @@ -28,6 +29,7 @@ Created Aug 11, 2011 Vasil Dimov #include "ut0crc32.h" /* ut_crc32() */ #include "ut0rnd.h" /* ut_fold_binary() */ #include "buf0types.h" +#include "mach0data.h" #ifndef UNIV_INNOCHECKSUM @@ -154,3 +156,46 @@ buf_checksum_algorithm_name( ut_error; return(NULL); } + +/** Calculates the CRC32 checksum of a page compressed page. The value is +stored to the page when it is written to a file and also checked for +a match when reading from the file. Checksum is calculated from +actual payload of the compressed page and some header fields. + +@param[in] page buffer page (UNIV_PAGE_SIZE bytes) +@return checksum */ +UNIV_INTERN +uint32_t +buf_calc_compressed_crc32( + const byte* page) +{ + /* In page compressed pages compression method is stored to field + FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION, thus add it to checksum. + In pages first compressed and then encrypted same field + contains key version after compression. */ + + ulint page_type = mach_read_from_2(page + FIL_PAGE_TYPE); + + ulint header_len = page_type == FIL_PAGE_PAGE_COMPRESSED ? + FIL_PAGE_SPACE_ID - FIL_PAGE_OFFSET : + FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION - FIL_PAGE_OFFSET; + + const uint32_t c1 = ut_crc32( + page + FIL_PAGE_OFFSET, + header_len); + + /* Calculate checksum from actual payload including stored size + field. In encrypted case add also compression method field. */ + ulint payload_len = mach_read_from_2(page+FIL_PAGE_DATA)+FIL_PAGE_COMPRESSED_SIZE; + + if (page_type == FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED) { + payload_len += FIL_PAGE_COMPRESSION_METHOD_SIZE; + } + + const uint32_t c2 = ut_crc32( + page + FIL_PAGE_DATA, + payload_len); + + return(c1 ^ c2); +} + |