MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional

In MySQL 5.7, it was noticed that files are not portable between
big-endian and little-endian processor architectures
(such as SPARC and x86), because the original implementation of
innodb_checksum_algorithm=crc32 was not byte order agnostic.

A byte order agnostic implementation of innodb_checksum_algorithm=crc32
was only added to MySQL 5.7, not backported to 5.6. Consequently,
MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C
implementation that works incorrectly on big-endian architectures,
and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C
implementation from MySQL 5.7.

MySQL 5.7 introduced a "legacy crc32" variant that is functionally
equivalent to the big-endian version of the original crc32 implementation.
Thanks to this variant, old data files can be transferred from big-endian
systems to newer versions.

Introducing new variants of checksum algorithms (without introducing
new names for them, or something on the pages themselves to identify
the algorithm) generally is a bad idea, because each checksum algorithm
is like a lottery ticket. The more algorithms you try, the more likely
it will be for the checksum to match on a corrupted page.

So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32,
and MariaDB 10.2.2 inherited this weakening.

We introduce a build option that together with MDEV-17957
makes innodb_checksum_algorithm=strict_crc32 strict again
by only allowing one variant of the checksum to match.

WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the
bug-compatible "legacy crc32" checksum. This is only enabled on
big-endian systems by default, to facilitate an upgrade from
MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32.

ut_crc32_byte_by_byte: Remove (unused function).

legacy_big_endian_checksum: Remove. This variable seems to have
unnecessarily complicated the logic. When the weakening is enabled,
we must always fall back to the buggy checksum.

buf_page_check_crc32(): A helper function to compute one or
two CRC-32C variants.

1 files changed, 15 insertions, 49 deletions

diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ut0crc32.cc
index cbb571e8f47..0c04cc2fc49 100644
--- a/storage/innobase/ut/ut0crc32.cc
+++ b/storage/innobase/ut/ut0crc32.cc
@@ -2,7 +2,7 @@
 
 Copyright (c) 2009, 2010 Facebook, Inc. All Rights Reserved.
 Copyright (c) 2011, 2015, Oracle and/or its affiliates. All Rights Reserved.
-Copyright (c) 2016, MariaDB Corporation.
+Copyright (c) 2016, 2018, MariaDB Corporation.
 
 This program is free software; you can redistribute it and/or modify it under
 the terms of the GNU General Public License as published by the Free Software
@@ -92,13 +92,11 @@ mysys/my_perf.c, contributed by Facebook under the following license.
 /** Pointer to CRC32 calculation function. */
 ut_crc32_func_t	ut_crc32;
 
+#ifdef INNODB_BUG_ENDIAN_CRC32
 /** Pointer to CRC32 calculation function, which uses big-endian byte order
 when converting byte strings to integers internally. */
 ut_crc32_func_t	ut_crc32_legacy_big_endian;
-
-/** Pointer to CRC32-byte-by-byte calculation function (byte order agnostic,
-but very slow). */
-ut_crc32_func_t	ut_crc32_byte_by_byte;
+#endif /* INNODB_BUG_ENDIAN_CRC32 */
 
 /** Text description of CRC32 implementation */
 const char*	ut_crc32_implementation;
@@ -278,6 +276,7 @@ ut_crc32_64_hw(
 	*len -= 8;
 }
 
+#ifdef INNODB_BUG_ENDIAN_CRC32
 /** Calculate CRC32 over 64-bit byte string using a hardware/CPU instruction.
 The byte string is converted to a 64-bit integer using big endian byte order.
 @param[in,out]	crc	crc32 checksum so far when this function is called,
@@ -308,6 +307,7 @@ ut_crc32_64_legacy_big_endian_hw(
 	*data += 8;
 	*len -= 8;
 }
+#endif /* INNODB_BUG_ENDIAN_CRC32 */
 
 /** Calculates CRC32 using hardware/CPU instructions.
 @param[in]	buf	data over which to calculate CRC32
@@ -396,6 +396,7 @@ ut_crc32_hw(
 	return(~crc);
 }
 
+# ifdef INNODB_BUG_ENDIAN_CRC32
 /** Calculates CRC32 using hardware/CPU instructions.
 This function uses big endian byte ordering when converting byte sequence to
 integers.
@@ -445,26 +446,7 @@ ut_crc32_legacy_big_endian_hw(
 
 	return(~crc);
 }
-
-/** Calculates CRC32 using hardware/CPU instructions.
-This function processes one byte at a time (very slow) and thus it does
-not depend on the byte order of the machine.
-@param[in]	buf	data over which to calculate CRC32
-@param[in]	len	data length
-@return CRC-32C (polynomial 0x11EDC6F41) */
-uint32_t
-ut_crc32_byte_by_byte_hw(
-	const byte*	buf,
-	ulint		len)
-{
-	uint32_t	crc = 0xFFFFFFFFU;
-
-	while (len > 0) {
-		ut_crc32_8_hw(&crc, &buf, &len);
-	}
-
-	return(~crc);
-}
+# endif /* INNODB_BUG_ENDIAN_CRC32 */
 #endif /* defined(__GNUC__) && defined(__x86_64__) || (_WIN64) */
 
 /* CRC32 software implementation. */
@@ -577,6 +559,7 @@ ut_crc32_64_sw(
 	*len -= 8;
 }
 
+#ifdef INNODB_BUG_ENDIAN_CRC32
 /** Calculate CRC32 over 64-bit byte string using a software implementation.
 The byte string is converted to a 64-bit integer using big endian byte order.
 @param[in,out]	crc	crc32 checksum so far when this function is called,
@@ -602,6 +585,7 @@ ut_crc32_64_legacy_big_endian_sw(
 	*data += 8;
 	*len -= 8;
 }
+#endif /* INNODB_BUG_ENDIAN_CRC32 */
 
 /** Calculates CRC32 in software, without using CPU instructions.
 @param[in]	buf	data over which to calculate CRC32
@@ -653,6 +637,7 @@ ut_crc32_sw(
 	return(~crc);
 }
 
+#ifdef INNODB_BUG_ENDIAN_CRC32
 /** Calculates CRC32 in software, without using CPU instructions.
 This function uses big endian byte ordering when converting byte sequence to
 integers.
@@ -704,28 +689,7 @@ ut_crc32_legacy_big_endian_sw(
 
 	return(~crc);
 }
-
-/** Calculates CRC32 in software, without using CPU instructions.
-This function processes one byte at a time (very slow) and thus it does
-not depend on the byte order of the machine.
-@param[in]	buf	data over which to calculate CRC32
-@param[in]	len	data length
-@return CRC-32C (polynomial 0x11EDC6F41) */
-uint32_t
-ut_crc32_byte_by_byte_sw(
-	const byte*	buf,
-	ulint		len)
-{
-	uint32_t	crc = 0xFFFFFFFFU;
-
-	ut_a(ut_crc32_slice8_table_initialized);
-
-	while (len > 0) {
-		ut_crc32_8_sw(&crc, &buf, &len);
-	}
-
-	return(~crc);
-}
+#endif /* INNODB_BUG_ENDIAN_CRC32 */
 
 /********************************************************************//**
 Initializes the data structures used by ut_crc32*(). Does not do any
@@ -736,8 +700,9 @@ ut_crc32_init()
 {
 	ut_crc32_slice8_table_init();
 	ut_crc32 = ut_crc32_sw;
+#ifdef INNODB_BUG_ENDIAN_CRC32
 	ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_sw;
-	ut_crc32_byte_by_byte = ut_crc32_byte_by_byte_sw;
+#endif /* INNODB_BUG_ENDIAN_CRC32 */
 	ut_crc32_implementation = "Using generic crc32 instructions";
 
 #if (defined(__GNUC__) && defined(__x86_64__)) || defined(_MSC_VER)
@@ -770,8 +735,9 @@ ut_crc32_init()
 
 	if (features_ecx & 1 << 20) {
 		ut_crc32 = ut_crc32_hw;
+#ifdef INNODB_BUG_ENDIAN_CRC32
 		ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_hw;
-		ut_crc32_byte_by_byte = ut_crc32_byte_by_byte_hw;
+#endif /* INNODB_BUG_ENDIAN_CRC32 */
 		ut_crc32_implementation = "Using SSE2 crc32 instructions";
 	}