Aria issues:

- Fix for LP#700623 "Aria recovery: ma_blockrec.c:3930: _ma_update_at_original_place: Assertion `block->org_bitmap_value == _ma_bitmap_get_page_bits(info, &info->s->bitmap, page)' failed"
  - Issue was that when deleting a tail page where all index entries where full, the page was marked wrongly in the bitmap.
- If debug_assert_if_crashed_table is set, we now crash when we find Aria corrupted.
- Write more information if we find something wrong with the bitmap.
- Fixed that REPAIR also can fix wrong create_rename_lsn issues (a very unlikely event)
- Define STATE_CRASHED_FLAGS as set of all CRASHED flags (to simplify code)


storage/maria/ha_maria.cc:
  Mark the normal page cache (not the page cache for the logs) so that we can request extra debugging for it.
  Copy the value of debug_assert_if_crashed_table to maria_assert_if_crashed_table so that we can request a crash at exactly the point where we find Aria corrupted.
  Use STATE_CRASHED_FLAGS
storage/maria/ma_bitmap.c:
  Made bits_to_txt extern so that we can use this in maria_chk
  Added extra information to the log files to be able to easier find bitmap failures in recovery. (When compiling with -DEXTRA_DEBUG_BITMAP)
  Added _ma_get_bitmap_description() to request a clear text description of the bitmap.
  Simplify _ma_check_bitmap_data(), as we know the bitmap pattern in the caller.
storage/maria/ma_blockrec.c:
  In delete_head_or_tail(), fixed a bug where we sent wrong information to _ma_bitmap_set() if the directory was full for a page that should be freed.
  This fixed LP#700623 (failure in bitmap found during recovery)
storage/maria/ma_blockrec.h:
  Added definitions for _ma_get_bitmap_description() and bits_to_txt
storage/maria/ma_check.c:
  Simplify call to _ma_check_bitmap_data().
  Write more information if we find something wrong with the bitmap.
  Moved getting clear text information about the bitmap to ma_bitmap.c::_ma_get_bitmap_description()
storage/maria/ma_checkpoint.c:
  More asserts
storage/maria/ma_create.c:
  Fix wrong create_rename_lsn during repair.
  (Create_rename_lsn can be too big if someone restores an old maria_log_file after an Aria file was created)
storage/maria/ma_delete.c:
  Call _ma_set_fatal_error() in case of crashed file
  Remove not needed test of save_errno == HA_ERR_KEY_NOT_FOUND. (Handled by other code
storage/maria/ma_extra.c:
  Call _ma_set_fatal_error() in case of crashed file
  Reset share->bitmap.changed_not_flushed to not cause new ASSERTS to trigger.
  Added _ma_file_callback_to_id() for writing share->id to log file in case of DEBUG logging.
storage/maria/ma_init.c:
  Destroy also translog if it's readonly (as when called by maria_read_log -d)
storage/maria/ma_key.c:
  Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_key_recover.c:
  STATE_CRASHED -> STATE_CRASHED_FLAGS
storage/maria/ma_keycache.c:
  Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_locking.c:
  Call _ma_set_fatal_error() in case of crashed file.
  Added _ma_set_fatal_error()
storage/maria/ma_open.c:
  Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_page.c:
  Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_pagecache.c:
  Added extra information to log file to simply debugging of bitmap errors.
storage/maria/ma_pagecache.h:
  Added extra_debug flag to allow marking of row and index cache for extra logging (for debugging).
storage/maria/ma_panic.c:
  Flush both data and index blocks in case of HA_PANIC_CLOSE
  Fixed wrong position of 'break'. (Not critical for MariaDB as MariaDB never uses this code)
storage/maria/ma_recovery_util.c:
  Avoid writing extra not needed \n to DBUG log.
storage/maria/ma_rkey.c:
  Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_search.c:
  Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_static.c:
  Define maria_assert_if_crashed_table
storage/maria/ma_update.c:
  Call _ma_set_fatal_error() in case of crashed file.
  The new code also avoids a problem where we before would print the error twice.
storage/maria/ma_write.c:
  Call _ma_set_fatal_error() in case of crashed file
storage/maria/maria_chk.c:
  STATE_CRASHED -> STATE_CRASHED_FLAGS
storage/maria/maria_def.h:
  Added STATE_CRASHED_PRINTED to avoid giving error message about crash twice.
  Added STATE_CRASHED_FLAGS to be able to easily detect and set all CRASHED related flags.
  Added prototypes for new functions.
storage/myisam/mi_panic.c:
  Fixed wrong position of 'break'. (Not critical for MariaDB as MariaDB never uses this code)

1 files changed, 100 insertions, 21 deletions

diff --git a/storage/maria/ma_bitmap.c b/storage/maria/ma_bitmap.c
index bc83f6ea0e9..2cbb66299a8 100644
--- a/storage/maria/ma_bitmap.c
+++ b/storage/maria/ma_bitmap.c
@@ -104,10 +104,11 @@
   - On checkpoint
   (Ie: When we do a checkpoint, we have to ensure that all bitmaps are
   put on disk even if they are not in the page cache).
-  - When explicitely requested (for example on backup or after recvoery,
+  - When explicitely requested (for example on backup or after recovery,
   to simplify things)
 
  The flow of writing a row is that:
+ - Mark the bitmap not flushable (_ma_bitmap_flushable(X, 1))
  - Lock the bitmap
  - Decide which data pages we will write to
  - Mark them full in the bitmap page so that other threads do not try to
@@ -119,6 +120,7 @@
    pages (that is, we marked pages full but when we are done we realize
    we didn't fill them)
  - Unlock the bitmap.
+ - Mark the bitmap flushable (_ma_bitmap_flushable(X, -1))
 */
 
 #include "maria_def.h"
@@ -127,6 +129,12 @@
 #define FULL_HEAD_PAGE 4
 #define FULL_TAIL_PAGE 7
 
+const char *bits_to_txt[]=
+{
+  "empty", "00-30% full", "30-60% full", "60-90% full", "full",
+  "tail 00-40 % full", "tail 40-80 % full", "tail/blob full"
+};
+
 /*#define WRONG_BITMAP_FLUSH 1*/ /*define only for provoking bugs*/
 #undef WRONG_BITMAP_FLUSH
 
@@ -273,6 +281,12 @@ my_bool _ma_bitmap_end(MARIA_SHARE *share)
   delete_dynamic(&share->bitmap.pinned_pages);
   my_free(share->bitmap.map, MYF(MY_ALLOW_ZERO_PTR));
   share->bitmap.map= 0;
+  /*
+    This is to not get an assert in checkpoint. The bitmap will be flushed
+    at once by  _ma_once_end_block_record() as part of the normal flush
+    of the kfile.
+  */
+  share->bitmap.changed_not_flushed= 0;
   return res;
 }
 
@@ -353,6 +367,24 @@ my_bool _ma_bitmap_flush_all(MARIA_SHARE *share)
   my_bool res= 0;
   MARIA_FILE_BITMAP *bitmap= &share->bitmap;
   DBUG_ENTER("_ma_bitmap_flush_all");
+
+#ifdef EXTRA_DEBUG_BITMAP
+  {
+    char buff[160];
+    uint len= my_sprintf(buff,
+                         (buff, "bitmap_flush:  fd: %d  id: %u  "
+                          "changed: %d  changed_not_flushed: %d  "
+                          "flush_all_requsted: %d",
+                          share->bitmap.file.file,
+                          share->id,
+                          bitmap->changed,
+                          bitmap->changed_not_flushed,
+                          bitmap->flush_all_requested));
+    (void) translog_log_debug_info(0, LOGREC_DEBUG_INFO_QUERY,
+                                   (uchar*) buff, len);
+  }
+#endif
+
   pthread_mutex_lock(&bitmap->bitmap_lock);
   if (bitmap->changed || bitmap->changed_not_flushed)
   {
@@ -364,6 +396,15 @@ my_bool _ma_bitmap_flush_all(MARIA_SHARE *share)
       pthread_cond_wait(&bitmap->bitmap_cond, &bitmap->bitmap_lock);
     }
 #endif
+#ifdef EXTRA_DEBUG_BITMAP
+    {
+      char tmp[MAX_BITMAP_INFO_LENGTH];      
+      _ma_get_bitmap_description(bitmap, bitmap->map, bitmap->page, tmp);
+      (void) translog_log_debug_info(0, LOGREC_DEBUG_INFO_QUERY,
+                                     (uchar*) tmp, strlen(tmp));
+    }
+#endif
+
     DBUG_ASSERT(bitmap->flush_all_requested == 1);
     /*
       Bitmap is in a flushable state: its contents in memory are reflected by
@@ -680,7 +721,7 @@ static inline uint pattern_to_size(MARIA_FILE_BITMAP *bitmap, uint pattern)
   Print bitmap for debugging
 
   SYNOPSIS
-  _ma_print_bitmap()
+  _ma_print_bitmap_changes()
   bitmap	Bitmap to print
 
   IMPLEMENTATION
@@ -691,12 +732,6 @@ static inline uint pattern_to_size(MARIA_FILE_BITMAP *bitmap, uint pattern)
 
 #ifndef DBUG_OFF
 
-const char *bits_to_txt[]=
-{
-  "empty", "00-30% full", "30-60% full", "60-90% full", "full",
-  "tail 00-40 % full", "tail 40-80 % full", "tail/blob full"
-};
-
 static void _ma_print_bitmap_changes(MARIA_FILE_BITMAP *bitmap)
 {
   uchar *pos, *end, *org_pos;
@@ -747,7 +782,6 @@ void _ma_print_bitmap(MARIA_FILE_BITMAP *bitmap, uchar *data,
   uchar *pos, *end;
   char llbuff[22];
 
-  end= bitmap->map + bitmap->used_size;
   DBUG_LOCK_FILE;
   fprintf(DBUG_FILE,"\nDump of bitmap page at %s\n", llstr(page, llbuff));
 
@@ -781,6 +815,56 @@ void _ma_print_bitmap(MARIA_FILE_BITMAP *bitmap, uchar *data,
 #endif /* DBUG_OFF */
 
 
+/*
+  Return content of bitmap as a printable string
+*/
+
+void _ma_get_bitmap_description(MARIA_FILE_BITMAP *bitmap,
+                                uchar *bitmap_data,
+                                pgcache_page_no_t page,
+                                char *out)
+{
+  uchar *pos, *end;
+  uint count=0, dot_printed= 0, len;
+  char buff[80], last[80];
+
+  page++;
+  last[0]=0;
+  for (pos= bitmap_data, end= pos+ bitmap->used_size ; pos < end ; pos+= 6)
+  {
+    ulonglong bits= uint6korr(pos);    /* 6 bytes = 6*8/3= 16 patterns */
+    uint i;
+
+    for (i= 0; i < 16 ; i++, bits>>= 3)
+    {
+      if (count > 60)
+      {
+        if (memcmp(buff, last, count))
+        {
+          memcpy(last, buff, count);
+          len= my_sprintf(out, (out, "%8lu: ", (ulong) page - count));
+          memcpy(out+len, buff, count);
+          out+= len + count + 1;
+          out[-1]= '\n';
+          dot_printed= 0;
+        }
+        else if (!(dot_printed++))
+        {
+          out= strmov(out, "...\n");
+        }
+        count= 0;
+      }
+      buff[count++]= '0' + (uint) (bits & 7);
+      page++;
+    }
+  }
+  len= my_sprintf(out, (out, "%8lu: ", (ulong) page - count));
+  memcpy(out+len, buff, count);
+  out[len + count]= '\n';
+  out[len + count + 1]= 0;
+}
+
+
 /***************************************************************************
   Reading & writing bitmap pages
 ***************************************************************************/
@@ -2383,16 +2467,14 @@ my_bool _ma_bitmap_release_unused(MARIA_HA *info, MARIA_BITMAP_BLOCKS *blocks)
         The page has all bits set; The following test is an optimization
         to not set the bits to the same value as before.
       */
+      DBUG_ASSERT(current_bitmap_value ==
+                  _ma_bitmap_get_page_bits(info, bitmap, block->page));
+
       if (bits != current_bitmap_value)
       {
         if (set_page_bits(info, bitmap, block->page, bits))
           goto err;
       }
-      else
-      {
-        DBUG_ASSERT(current_bitmap_value ==
-                    _ma_bitmap_get_page_bits(info, bitmap, block->page));
-      }
     }
     else if (!(block->used & BLOCKUSED_USED) &&
              _ma_bitmap_reset_full_page_bits(info, bitmap,
@@ -2521,17 +2603,15 @@ my_bool _ma_bitmap_set(MARIA_HA *info, pgcache_page_no_t page, my_bool head,
     page_type	    What kind of page this is
     page	    Adress to page
     empty_space     Empty space on page
-    bitmap_pattern  Store here the pattern that was in the bitmap for the
-		    page. This is always updated.
+    bitmap_pattern  Bitmap pattern for page (from bitmap)
 
   RETURN
     0  ok
     1  error
 */
 
-my_bool _ma_check_bitmap_data(MARIA_HA *info,
-                              enum en_page_type page_type, pgcache_page_no_t page,
-                              uint empty_space, uint *bitmap_pattern)
+my_bool _ma_check_bitmap_data(MARIA_HA *info, enum en_page_type page_type,
+                              uint empty_space, uint bitmap_pattern)
 {
   uint bits;
   switch (page_type) {
@@ -2552,8 +2632,7 @@ my_bool _ma_check_bitmap_data(MARIA_HA *info,
     bits= 0; /* to satisfy compiler */
     DBUG_ASSERT(0);
   }
-  return ((*bitmap_pattern= _ma_bitmap_get_page_bits(info, &info->s->bitmap,
-                                                     page)) != bits);
+  return (bitmap_pattern != bits);
 }