MDEV-13103: InnoDB crash recovery fails to decompress a page in buf_dblwr_process()bb-10.1-MDEV-13103

There were several problems. Firstly, page decompression code did not handle
possible decompression errors correctly. Secondly, not all compression methods
tolerate corrupted input (e.g. lz4 did not tolerate input that was compressed
using snappy method). Finally, if page is actually also encrypted we can't
decompress page. Solutions: Add proper error handling to decompression code
and add post compression checksum to page. As whole page including page checksum
is compressed we can reuse the original checksum field for post compression
checksum. With post compression checksum we can detect most of the corruptions.
If no corruption is detected hopefully decompression code can detect
remaining problems.

Doublewrite buffer page recovery for page compressed pages require
that post compression checksum matches. For pages from old releases
supporting page compression checksum must be BUF_NO_CHECKSUM_MAGIC.

Upgrade from older versions is supported as post compression
checksum check accepts the BUF_NO_CHECKSUM_MAGIC that they stored
in checksum filed.

Downgrade to older versions is not supported (assuming that there
were some changes to compressed tables) as page compression code
would not tolerate any other checksum except BUF_NO_CHECKSUM_MAGIC.

innochecksum.cc is_page_corrupted:
    If page is compressed verify post compression checksum

buf_page_decrypt_after_read
     Return DB_PAGE_CORRUPTED if page is found to be corrupted
     after post compression checksum check.

buf_page_io_complete
     If page is found corrupted after buf_page_decrypt_after_read
     there is no need to continue page check.

buf_page_decrypt_after_read
     Verify post compression checksum before decompression and
     if it does not match mark page corrupted. Note that old
     compressed pages do not really have post compression
     checksum so they are treated as not corrupted and then
     we need to hope that decompression code can handle the
     possible corruptions by returning error.

buf_calc_compressed_crc32
     New function to calculate post compression checksum
     so that necessary compression metadata fields are
     included.

buf_dblwr_decompress
     New function that handles post compression checksum check
     and page decompression if it is ok.

buf_dblwr_process
     Verify post compression checksum before trying to decompress
     page.

fil_space_verify_crypt_checksum
     Remove incorrect code as compressed and encrypted pages
     do have post encryption checksum.

fil_compress_page
     Calculate and store post compression checksum to FIL_SPACE_OR_CHKSUM
     field as original value is stored on compressed image.

fil_decompress_page
     Add error handling if decompression fails.

fil_verify_compression_checksum
     New function to verify post compression checksum.
     Compressed tablespaces before this change have BUF_NO_CHECKSUM_MAGIC
     in checksum field and they must be treated as not corrupted.

convert_error_code_to_mysql
      Handle also page corruptions DB_PAGE_CORRUPTED as HA_ERR_CRASHED.
      Note that there are cases when we do not know for certain
      is page corrupted, corrupted and compressed, or still encrypted
      after failed decrypt, thus tablespace could be marked just corrupted.

Tests modified
      innodb-page_compression_[zip, lz4, lzma, lzo, bzip2, snappy]
            to use innodb-page-compression.inc
      innodb-page-compression.inc add innochecksum and intentional tablespace
            corruption tests.
      innodb-force-corrupt, innodb_bug14147491 add new error
            messages to mtr suppression and new error codes.

New tests
      encryption/innodb-corrupted.test test intentionally corrupted
            tablespaces containing encryption and compression.
      doublewrite-compressed test doublewrite recovery for page
            compressed tables
      innodb-import-101 import files from both big_endian and little_endian
            machine

This is 10.1 version use null merge to 10.2 as it has its own version.

1 files changed, 217 insertions, 73 deletions

diff --git a/mysql-test/suite/innodb/include/innodb-page-compression.inc b/mysql-test/suite/innodb/include/innodb-page-compression.inc
index 3acbeaf0988..8a0036d61e0 100644
--- a/mysql-test/suite/innodb/include/innodb-page-compression.inc
+++ b/mysql-test/suite/innodb/include/innodb-page-compression.inc
@@ -1,53 +1,63 @@
+if (!$INNOCHECKSUM) {
+  --echo Need innochecksum binary
+  --die Need innochecksum binary
+}
+
 --disable_warnings
 set global innodb_file_format = `Barracuda`;
 set global innodb_file_per_table = on;
 --enable_warnings
 
-create table innodb_normal (c1 int not null auto_increment primary key, b char(200)) engine=innodb;
-create table innodb_page_compressed1 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=1;
-create table innodb_page_compressed2 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=2;
-create table innodb_page_compressed3 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=3;
-create table innodb_page_compressed4 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=4;
-create table innodb_page_compressed5 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=5;
-create table innodb_page_compressed6 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=6;
-create table innodb_page_compressed7 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=7;
-create table innodb_page_compressed8 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=8;
-create table innodb_page_compressed9 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=9;
+call mtr.add_suppression("InnoDB: Database page corruption on disk or a failed file read of tablespace test/t[0-9]+ page \\[page id: space=[0-9]+, page number=[0-9]+\\]. You may have to recover from a backup.");
+call mtr.add_suppression("InnoDB: Page \\[page id: space=[0-9]+, page number= [0-9]+\\] in file ./test/t[0-9]+.ibd may be corrupted. Post compression checksum [0-9]+ stored [0-9]+ compression_method [ZLIB|SNAPPY|LZ4|LZO|LZMA|BZIP2]");
+call mtr.add_suppression("InnoDB: Background Page read failed to read or decrypt \\[page id: space=[0-9]+, page number=[0-9]+\\]");
+call mtr.add_suppression("mysqld: Index for table 't[0-9]+' is corrupt; try to repair it");
+
+create table t0 (c1 int not null auto_increment primary key, b char(200)) engine=innodb;
+create table t1 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=1;
+create table t2 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=2;
+create table t3 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=3;
+create table t4 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=4;
+create table t5 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=5;
+create table t6 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=6;
+create table t7 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=7;
+create table t8 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=8;
+create table t9 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=9;
 
 --disable_query_log
 begin;
-let $i = 2000;
+let $i = 100;
 while ($i)
 {
-    insert into innodb_normal(b) values(REPEAT('Aa',50));
-    insert into innodb_normal(b) values(REPEAT('a',100));
-    insert into innodb_normal(b) values(REPEAT('b',100));
-    insert into innodb_normal(b) values(REPEAT('0',100));
-    insert into innodb_normal(b) values(REPEAT('1',100));
+    insert into t0(b) values(REPEAT('Aa',50));
+    insert into t0(b) values(REPEAT('a',100));
+    insert into t0(b) values(REPEAT('b',100));
+    insert into t0(b) values(REPEAT('0',100));
+    insert into t0(b) values(REPEAT('1',100));
     dec $i;
 }
 
-insert into innodb_page_compressed1 select * from innodb_normal;
-insert into innodb_page_compressed2 select * from innodb_normal;
-insert into innodb_page_compressed3 select * from innodb_normal;
-insert into innodb_page_compressed4 select * from innodb_normal;
-insert into innodb_page_compressed5 select * from innodb_normal;
-insert into innodb_page_compressed6 select * from innodb_normal;
-insert into innodb_page_compressed7 select * from innodb_normal;
-insert into innodb_page_compressed8 select * from innodb_normal;
-insert into innodb_page_compressed9 select * from innodb_normal;
+insert into t1 select * from t0;
+insert into t2 select * from t0;
+insert into t3 select * from t0;
+insert into t4 select * from t0;
+insert into t5 select * from t0;
+insert into t6 select * from t0;
+insert into t7 select * from t0;
+insert into t8 select * from t0;
+insert into t9 select * from t0;
 commit;
 --enable_query_log
 
-select count(*) from innodb_page_compressed1;
-select count(*) from innodb_page_compressed3;
-select count(*) from innodb_page_compressed4;
-select count(*) from innodb_page_compressed5;
-select count(*) from innodb_page_compressed6;
-select count(*) from innodb_page_compressed6;
-select count(*) from innodb_page_compressed7;
-select count(*) from innodb_page_compressed8;
-select count(*) from innodb_page_compressed9;
+select count(*) from t1;
+select count(*) from t3;
+select count(*) from t4;
+select count(*) from t5;
+select count(*) from t6;
+select count(*) from t6;
+select count(*) from t7;
+select count(*) from t8;
+select count(*) from t9;
 
 #
 # Wait until pages are really compressed
@@ -61,71 +71,205 @@ let $wait_condition= select variable_value > 0 from information_schema.global_st
 
 --source include/shutdown_mysqld.inc
 
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_normal.ibd
+--let t1_IBD = $MYSQLD_DATADIR/test/t0.ibd
 --let SEARCH_RANGE = 10000000
 --let SEARCH_PATTERN=AaAaAaAa
---echo # innodb_normal expected FOUND
+--echo # t0 expected FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed1.ibd
---echo # innodb_page_compressed1 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t1.ibd
+--echo # t1 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed2.ibd
---echo # innodb_page_compressed2 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t2.ibd
+--echo # t2 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed3.ibd
---echo # innodb_page_compressed3 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t3.ibd
+--echo # t3 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed4.ibd
---echo # innodb_page_compressed4 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t4.ibd
+--echo # t4 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed5.ibd
---echo # innodb_page_compressed5 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t5.ibd
+--echo # t5 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed6.ibd
---echo # innodb_page_compressed6 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t6.ibd
+--echo # t6 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed7.ibd
---echo # innodb_page_compressed7 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t7.ibd
+--echo # t7 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed8.ibd
---echo # innodb_page_compressed8 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t8.ibd
+--echo # t8 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
---let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed9.ibd
---echo # innodb_page_compressed9 page compressed expected NOT FOUND
+--let t1_IBD = $MYSQLD_DATADIR/test/t9.ibd
+--echo # t9 page compressed expected NOT FOUND
 -- let SEARCH_FILE=$t1_IBD
 -- source include/search_pattern_in_file.inc
 
+#
+# Run innochecksum to all tables, all tables should be ok
+#
+let t_IBD = $MYSQLD_DATADIR/test/t0.ibd;
+--exec $INNOCHECKSUM $t_IBD
+let $i=9;
+while $($i > 0) {
+--echo # Run innochecksum on t$i
+let t_IBD = $MYSQLD_DATADIR/test/t$i.ibd;
+--exec $INNOCHECKSUM $t_IBD
+dec $i;
+}
+
 -- source include/start_mysqld.inc
 
-select count(*) from innodb_page_compressed1;
-select count(*) from innodb_page_compressed3;
-select count(*) from innodb_page_compressed4;
-select count(*) from innodb_page_compressed5;
-select count(*) from innodb_page_compressed6;
-select count(*) from innodb_page_compressed6;
-select count(*) from innodb_page_compressed7;
-select count(*) from innodb_page_compressed8;
-select count(*) from innodb_page_compressed9;
+select count(*) from t0;
+select count(*) from t1;
+select count(*) from t3;
+select count(*) from t4;
+select count(*) from t5;
+select count(*) from t6;
+select count(*) from t7;
+select count(*) from t8;
+select count(*) from t9;
 
 let $wait_condition= select variable_value > 0 from information_schema.global_status where variable_name = 'INNODB_NUM_PAGES_PAGE_DECOMPRESSED';
 --source include/wait_condition.inc
 
-drop table innodb_normal;
-drop table innodb_page_compressed1;
-drop table innodb_page_compressed2;
-drop table innodb_page_compressed3;
-drop table innodb_page_compressed4;
-drop table innodb_page_compressed5;
-drop table innodb_page_compressed6;
-drop table innodb_page_compressed7;
-drop table innodb_page_compressed8;
-drop table innodb_page_compressed9;
+let INNODB_PAGE_SIZE=`select @@innodb_page_size`;
+let MYSQLD_DATADIR=`select @@datadir`;
+
+--echo # Restart server
+--source include/shutdown_mysqld.inc
+
+--echo # Corrupting tablespaces...
+
+#
+# Now we corrupt page compressed pages as follows:
+# (1) compression method
+# (2) payload size
+# (3) data
+# (4) checksum
+#
+# Page 0 is not compressed or encrypted
+#
+perl;
+sub ib_write_value {
+  my($fh, $file, $pos, $len, $val) = @_;
+  seek($fh, $pos, SEEK_SET) or die "$0: seek $file to pos $pos: $!";;
+  syswrite($fh, $val, $len) == $len or die "$0: write to $file val $val len $len: $!";;
+}
+sub ib_corrupt_table {
+  my($file, $pos, $val, $len) = @_;
+  open(my $fh, "+<", $file) or die "$0: open $file: $!";
+  binmode $fh;
+  ib_write_value($fh, $file, $pos, $len, $val);
+  close $fh or die "$0: close $file: $!";
+}
+sub ib_read_value {
+    my($fh, $file, $pos, $len) = @_;
+    seek($fh, $pos, SEEK_SET) or die "$0: seek $file to pos $pos: $!";
+    sysread($fh, $buf, $len) == $len or die "$0: read $file : $!";
+    return $buf;
+}
+
+my($pos) = $ENV{'INNODB_PAGE_SIZE'} * 3;
+my($file) = "$ENV{MYSQLD_DATADIR}/test/t1.ibd";
+open($fh, "+<", $file) or die "$0: open $file : $!";
+binmode $fh;
+
+# Find out the page type, compression method location is based on that
+my($ptype) = unpack("n", ib_read_value($fh, $file, $pos+24, 2));
+if ($ptype == 34354) {
+   $pos += 32; # FLUSH_LSN_OR_KEY_VERSION + 6
+} else {
+   $pos += 40; # FIL_PAGE_DATA + 2
+}
+
+# Corrupt compression method by decreasing it by one, if zero set 6
+# Note that compression method is not encrypted
+my($cmethod) = unpack("n", ib_read_value($fh, $file, $pos, 2));
+$cmethod--;
+if ($cmethod == 0) {
+  $cmethod = 6;
+}
+
+ib_write_value($fh, $file, $pos, 2, pack("n", $cmethod));
+close $fh or die $!;
+
+# (2) corrupt payload size by decreasing size by 50 and if 0 set it to 20
+# note that payload size is not encrypted
+my($pos) = $ENV{'INNODB_PAGE_SIZE'} * 3 + 38;
+my($file) = "$ENV{MYSQLD_DATADIR}/test/t2.ibd";
+open($fh, "+<", $file) or die "$0: open $file : $!";
+my($size) = unpack("n", ib_read_value($fh, $file, $pos, 2));
+$size -= 50;
+if ($size <= 0) {
+  $size = 20;
+}
+ib_write_value($fh, $file, $pos, 2, pack("n", $size));
+close $fh or die $!;
+
+# (3) Corrupt data
+
+ib_corrupt_table("$ENV{MYSQLD_DATADIR}/test/t3.ibd", $ENV{'INNODB_PAGE_SIZE'} * 3 + 42,
+    "deadaaaaffffbbbb",14);
+
+# (4) Corrupt checksum
+my($pos) = $ENV{'INNODB_PAGE_SIZE'} * 3;
+my($file) = "$ENV{MYSQLD_DATADIR}/test/t4.ibd";
+open($fh, "+<", $file) or die "$0: open $file : $!";
+
+# Find out the page type, checksum location is based on that
+my($ptype) = unpack("n", ib_read_value($fh, $file, $pos + 24, 2));
+if ($ptype == 37401) {
+   $pos += 30;
+}
+ib_write_value($fh, $file, $pos, 4, pack("N", 1020102010));
+close $fh or die $!;
+EOF
+
+--echo # Corruption done
+
+#
+# Run innochecksum to page compressed (and maybe encrypted) tables
+# now we should detect corruptions
+#
+let $i=4;
+while $($i > 0) {
+--echo # Run innochecksum on t$i
+let t_IBD = $MYSQLD_DATADIR/test/t$i.ibd;
+--error 1
+--exec $INNOCHECKSUM $t_IBD
+dec $i;
+}
+
+--echo # Start server again
+--source include/start_mysqld.inc
+
+#
+# Server should not crash on corrupted tables
+#
+--error ER_NOT_KEYFILE
+select * from t1;
+--error ER_NOT_KEYFILE
+select * from t2;
+--error ER_NOT_KEYFILE
+select * from t3;
+--error ER_NOT_KEYFILE
+select * from t4;
+select count(*) from t5;
+select count(*) from t6;
+select count(*) from t7;
+select count(*) from t8;
+select count(*) from t9;
+
+# We should be able to drop even corrupted tables
+
+drop table t0, t1, t2, t3, t4, t5, t6, t7, t8, t9;