MDEV-13103: InnoDB crash recovery fails to decompress a page in buf_dblwr_process()bb-10.1-MDEV-13103-MDEV-15032

There were several problems. Firstly, page decompression code did not handle
possible decompression errors correctly. Secondly, not all compression methods
tolerate corrupted input (e.g. lz4 did not tolerate input that was compressed
using snappy method). Finally, if page is actually also encrypted we can't
decompress page. Solutions: Add proper error handling to decompression code
and add post compression checksum to page. As whole page including page checksum
is compressed we can reuse the original checksum field for post compression
checksum. With post compression checksum we can detect most of the corruptions.
If no corruption is detected hopefully decompression code can detect
remaining problems.

Doublewrite buffer page recovery for page compressed pages require
that post compression checksum matches. For pages from old releases
supporting page compression checksum must be BUF_NO_CHECKSUM_MAGIC.

Upgrade from older versions is supported as post compression
checksum check accepts the BUF_NO_CHECKSUM_MAGIC that they stored
in checksum filed.

Downgrade to older versions is not supported (assuming that there
were some changes to compressed tables) as page compression code
would not tolerate any other checksum except BUF_NO_CHECKSUM_MAGIC.

innochecksum.cc is_page_corrupted:
    If page is compressed verify post compression checksum

buf_page_decrypt_after_read
     Return DB_PAGE_CORRUPTED if page is found to be corrupted
     after post compression checksum check.

buf_page_io_complete
     If page is found corrupted after buf_page_decrypt_after_read
     there is no need to continue page check.

buf_page_decrypt_after_read
     Verify post compression checksum before decompression and
     if it does not match mark page corrupted. Note that old
     compressed pages do not really have post compression
     checksum so they are treated as not corrupted and then
     we need to hope that decompression code can handle the
     possible corruptions by returning error.

buf_calc_compressed_crc32
     New function to calculate post compression checksum
     so that necessary compression metadata fields are
     included.

buf_dblwr_decompress
     New function that handles post compression checksum check
     and page decompression if it is ok.

buf_dblwr_process
     Verify post compression checksum before trying to decompress
     page.

fil_space_verify_crypt_checksum
     Remove incorrect code as compressed and encrypted pages
     do have post encryption checksum.

fil_compress_page
     Calculate and store post compression checksum to FIL_SPACE_OR_CHKSUM
     field as original value is stored on compressed image.

fil_decompress_page
     Add error handling if decompression fails.

fil_verify_compression_checksum
     New function to verify post compression checksum.
     Compressed tablespaces before this change have BUF_NO_CHECKSUM_MAGIC
     in checksum field and they must be treated as not corrupted.

fil_iterate
     Add post compression checksum check before actual decompression.

convert_error_code_to_mysql
      Handle also page corruptions DB_PAGE_CORRUPTED as HA_ERR_CRASHED.
      Note that there are cases when we do not know for certain
      is page corrupted, corrupted and compressed, or still encrypted
      after failed decrypt, thus tablespace could be marked just corrupted.

Tests modified
      innodb-page_compression_[zip, lz4, lzma, lzo, bzip2, snappy]
            to use innodb-page-compression.inc
      innodb-page-compression.inc add innochecksum and intentional tablespace
            corruption tests.
      innodb-force-corrupt, innodb_bug14147491 add new error
            messages to mtr suppression and new error codes.

New tests
      encryption/innodb-corrupted.test test intentionally corrupted
            tablespaces containing encryption and compression.
      doublewrite-compressed test doublewrite recovery for page
            compressed tables
      innodb-import import files from both big_endian and little_endian
            machine

1 files changed, 2 insertions, 0 deletions

diff --git a/storage/xtradb/include/fsp0pagecompress.ic b/storage/xtradb/include/fsp0pagecompress.ic
index 14f968e319e..0e9a574eef6 100644
--- a/storage/xtradb/include/fsp0pagecompress.ic
+++ b/storage/xtradb/include/fsp0pagecompress.ic
@@ -25,6 +25,7 @@ Created 11/12/2013 Jan Lindström jan.lindstrom@mariadb.com
 
 ***********************************************************************/
 
+#ifndef UNIV_INNOCHECKSUM
 /********************************************************************//**
 Determine the tablespace is page compression level from dict_table_t::flags.
 @return	page compression level or 0 if not compressed*/
@@ -84,6 +85,7 @@ fil_page_is_compressed_encrypted(
 {
 	return(mach_read_from_2(buf+FIL_PAGE_TYPE) == FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED);
 }
+#endif /*!UNIV_INNOCHECKSUM */
 
 /****************************************************************//**
 Get the name of the compression algorithm used for page