add a description of the recovery-process

(This used to be ctdb commit 194abb41e1a0036956a9401efcae8b14ed66c532)

3 files changed, 486 insertions, 0 deletions

diff --git a/ctdb/Makefile.in b/ctdb/Makefile.in
index 779bd8b4b01..72ca2d8afbf 100755
--- a/ctdb/Makefile.in
+++ b/ctdb/Makefile.in
@@ -187,6 +187,7 @@ install: all
 	${INSTALLCMD} -m 644 config/functions $(DESTDIR)$(etcdir)/ctdb
 	${INSTALLCMD} -m 755 config/statd-callout $(DESTDIR)$(etcdir)/ctdb
 	${INSTALLCMD} -m 644 config/events.d/README $(DESTDIR)/$(docdir)/ctdb/README.eventscripts
+	${INSTALLCMD} -m 644 doc/recovery-process.txt $(DESTDIR)/$(docdir)/ctdb/doc/recovery-process.txt
 	${INSTALLCMD} -m 755 config/events.d/00.ctdb $(DESTDIR)$(etcdir)/ctdb/events.d
 	${INSTALLCMD} -m 755 config/events.d/10.interface $(DESTDIR)$(etcdir)/ctdb/events.d
 	${INSTALLCMD} -m 755 config/events.d/20.multipathd $(DESTDIR)$(etcdir)/ctdb/events.d
diff --git a/ctdb/doc/recovery-process.txt b/ctdb/doc/recovery-process.txt
new file mode 100644
index 00000000000..d0294a30fb7
--- /dev/null
+++ b/ctdb/doc/recovery-process.txt
@@ -0,0 +1,484 @@
+Valid as of 1.0.66, may/will change in the future
+
+
+RECMASTER
+=========
+Recovery Master, this is one of the nodes in the cluster that has been designated to
+be the "recovery master".
+The recovery master is responsible for performing full checks of cluster and cluster node consistency and is also responsible for performing the actual database recovery procedure.
+
+Only one node at a time can be the recovery master.
+This is ensured by CTDB using a lock on a single file in the shared gpfs filesystem:
+  /etc/sysconfig/ctdb :
+  ...
+  # Options to ctdbd. This is read by /etc/init.d/ctdb
+  # you must specify the location of a shared lock file across all the
+  # nodes. This must be on shared storage
+  # there is no default here
+  CTDB_RECOVERY_LOCK=/gpfs/.ctdb/shared
+  ...
+
+In order to prevent that two nodes become recovery master at the same time (==split brain)
+CTDB here relies on GPFS that GPFS will guarantee coherent locking across the cluster.
+Thus CTDB relies on that GPFS MUST only allow one ctdb process on one node to take out and
+hold this lock.
+
+The recovery master is designated through an election process. 
+
+
+VNNMAP
+======
+The VNNMAP is a list of all nodes in the cluster that is currently part of the cluster
+and participates in hosting the cluster databases.
+All nodes that are CONNECTED but not BANNED be present in the VNNMAP.
+
+The VNNMAP is the list of LMASTERS for the cluster as reported by 'ctdb status' "
+  ...
+  Size:3
+  hash:0 lmaster:0
+  hash:1 lmaster:1
+  hash:2 lmaster:2
+  ...
+
+
+CLUSTER MONITORING
+==================
+All nodes in the cluster monitor its own health and its own consistency regards to the
+recovery master. How and what the nodes monitor for differs between the node which is
+the recovery master and normal nodes.
+This monitoring it to ensure that the cluster is healthy and consistent. 
+This is not related to monitoring of inidividual node health, a.k.a. eventscript monitroing.
+
+At the end of each step in the process are listed some of the most common and important
+error messages that can be generated during that step.
+
+
+NORMAL NODE CLUSTER MONITORING
+------------------------------
+Monitoring is performed in the dedicated recovery daemon process.
+The implementation can be found in server/ctdb_recoverd.c:monitor_cluster()
+This is an overview of the more important tasks during monitoring.
+These tests are to verify that the local node is consistent with the recovery master.
+
+Once every second the following monitoring loop is performed :
+
+1, Verify that the parent ctdb daemon on the local node is still running.
+   If it is not, the recovery daemon logs an error and terminates.
+   "CTDB daemon is no longer available. Shutting down recovery daemon"
+
+2, Check if any of the nodes has been recorded to have misbehaved too many times.
+   If so we ban the node and log a message :
+   "Node %u has caused %u failures in %.0f seconds - banning it for %u seconds"
+
+3, Check that there is a recovery master.
+   If not we initiate a clusterwide election process and log :
+   "Initial recovery master set - forcing election"
+   and we restart monitoring from 1.
+
+4, Verify that recovery daemon and the local ctdb daemon agreed on all the
+   node BANNING flags.
+   If the recovery daemon and the local ctdb daemon disagrees on these flags we update
+   the local ctdb daemon, logs one of two messages and restarts monitoring from 1 again.
+   "Local ctdb daemon on recmaster does not think this node is BANNED but the recovery master disagrees. Unbanning the node"
+   "Local ctdb daemon on non-recmaster does not think this node is BANNED but the recovery master disagrees. Re-banning the node"
+
+5, Verify that the node designated to be recovery master exists in the local list of all nodes.
+   If the recovery master is not in the list of all cluster nodes a new recovery master
+   election is triggered and monitoring restarts from 1.
+   "Recmaster node %u not in list. Force reelection"
+
+6, Check if the recovery master has become disconnected.
+   If is has, log an error message, force a new election and restart monitoring from 1.
+   "Recmaster node %u is disconnected. Force reelection"
+
+7, Read the node flags off the recovery master and verify that it has not become banned.
+   If is has, log an error message, force a new election and restart monitoring from 1.
+   "Recmaster node %u no longer available. Force reelection"
+
+8, Verify that the recmaster and the local node agrees on the flags (BANNED/DISABLED/...)
+   for the local node. 
+   If there is an inconsistency, push the flags for the local node out to all other nodes.
+   "Recmaster disagrees on our flags flags:0x%x recmaster_flags:0x%x  Broadcasting out flags."
+
+9, Verify that the local node hosts all public ip addresses it should host and that it does
+   NOT host any public addresses it should not host.
+   If there is an inconsistency we log an error, trigger a recovery to occur and restart
+   monitoring from 1 again.
+   "Public address '%s' is missing and we should serve this ip"
+   "We are still serving a public address '%s' that we should not be serving."
+
+These are all the checks we perform during monitoring for a normal node.
+These tests are performed on all nodes in the cluster which is why it is optimized to perform
+as few network calls to other nodes as possible.
+Each node only performs 1 call to the recovery master in each loop and to no other nodes.
+
+NORMAL NODE CLUSTER MONITORING
+------------------------------
+The recovery master performs a much more extensive test. In addition to tests 1-9 above
+the recovery master also performs the following tests:
+
+10, Read the list of nodes and flags from all other CONNECTED nodes in the cluster.
+    If there is a failure to read this list from one of the nodes, then log an 
+    error, mark this node as a candidate to become BANNED and restart monitoring from 1.
+    "Unable to get nodemap from remote node %u"
+
+11, Verify that the local recovery master and the remote node agrees on the flags
+    for the remote node. If there is a inconsistency for the BANNING flag,
+    log an error, trigger a new recmaster election and restart monitoring from 1.
+    "Remote node %u had different BANNED flags 0x%x, local had 0x%x - trigger a re-election"
+    "Remote node %u had flags 0x%x, local had 0x%x - updating local"
+
+12, Verify that the local recovery master and the remote node agrees on the flags
+    for the remote node. If one of the flags other than the BANNING flag was inconsistent,
+    just update the set of flags for the local recovery daemon, log an information message
+    and continue monitoring.
+    "Remote node %u had flags 0x%x, local had 0x%x - updating local"
+
+13, Read the list of public ip addresses from all of the CONNECTED nodes and merge into a
+    single clusterwide list.
+    If we fail to read the list of ips from a node, log an error and restart monitoring from 1.
+    "Failed to read public ips from node : %u"
+
+14, Verify that all other nodes agree that this node is the recovery master.
+    If one of the other nodes discgrees this is the recovery master, log an error,
+    force a new election and restart monitoring from 1.
+    "Node %d does not agree we are the recmaster. Need a new recmaster election"
+
+15, Check if the previous attempt to run a recovery failed, and if it did, try a new recovery.
+    After the recovery, restart monitoring from 1.
+    "Starting do_recovery"
+
+16, Verify that all CONNECTED nodes in the cluster are in recovery mode NORMAL.
+    If one of the nodes were in recovery mode ACTIVE, force a new recovery and restart
+    monitoring from 1.
+    "Node:%u was in recovery mode. Restart recovery process"
+
+17, Verify that the filehandle to the recovery lock file is valid.
+    If it is not, this may mean a split brain and is a critical error.
+    Try a new recovery and restart monitoring from 1.
+    "recovery master doesn't have the recovery lock"
+
+18, Verify that GPFS allows us to read from the recovery lock file.
+    If not there is a critical GPFS issue and we may have a split brain.
+    Try forcing a new recovery and restart monitoring from 1.
+    "failed read from recovery_lock_fd - %s"
+
+19, Read the list of all nodes and flags from all CONNECTED nodes in the cluster.
+    If fail to read the nodemap from one of the remote nodes, log an error and restart
+    monitoring from 1.
+    "Unable to get nodemap from remote node %u"
+
+20, If the nodemap differs between the local node and the remote node, log an error
+    and force a recovery.
+    This would happen if the /etc/ctdb/nodes file differs across nodes in the cluster.
+    It is unlikely that the recovery will rectify the situation.
+    This is a critical error, it is most likely the entire cluster will be unavailable
+    until the files are fixed or have became banned.
+    "Remote node:%u has different node count. %u vs %u of the local node"
+
+21, If a remote node disagrees on the content of the nodes list, try a recovery and restart
+    monitoring from 1.
+    It is unlikely that the recovery will rectify the situation.
+    This is a critical error, it is most likely the entire cluster will be unavailable
+    until the files are fixed or have became banned.
+    "Remote node:%u has different nodemap pnn for %d (%u vs %u)."
+
+22, If a remote node disgrees on the node flags in the list, try a recovery to re-sync
+    the flags and restart monitoring from 1.
+    "Remote node:%u has different nodemap flag for %d (0x%x vs 0x%x)"
+
+23, Verify that all active nodes are part of the VNNMAP.
+    If not, this would be a new node that has become CONNECTED but does not yet participate
+    in the cluster.
+    Perform a recovery to merge the new node to the cluster and restart monitoring from 1.
+    "The vnnmap count is different from the number of active nodes. %u vs %u"
+    or
+    "Node %u is active in the nodemap but did not exist in the vnnmap"
+
+24, Read the VNNMAP from all CONNECTED nodes.
+    Verify that all nodes have the same VNNMAP content and that all nodes are in the same
+    generation instance of the databases.
+    If not, force a recovery to re-synchronize the vnnmap and the databases across the cluster
+    and restart monitoring from 1.
+    "Remote node %u has different generation of vnnmap. %u vs %u (ours)"
+    "Remote node %u has different size of vnnmap. %u vs %u (ours)"
+    "Remote node %u has different vnnmap."
+
+25, If there has been changes to the cluster that requires a reallocation of public ip
+    addresses. On all nodes run the "startrecovery" event. Run "releaseip" and "takeip"
+    events to reassign the ips across the cluster and finally run the "recovered" event.
+
+Finished monitoring, continue monitoring from 1.
+
+
+CLUSTER RECOVERY
+================
+Recoveries are driven by the recovery daemon on the node that is currently the recovery
+master.
+Most of the logging that is performed during recovery is only logged on the node that
+is the recovery master.
+Make sure to find which node is the recovery master and check the log for that node.
+
+Example log entries that start in column 1 are expected to be present in the
+log. Example log entries that are indented 3 columns are optional and might
+only be present if an error occured.
+
+
+1, Log that recovery has been initiated.
+"Starting do_recovery"
+
+   It might log an informational message :
+"New recovery culprit %u".
+   This is only semi-accurate and might may not mean that there is any problem
+   at all with the node indicated.
+
+
+2, Check if a node has caused too many failed recoveries and if so ban it from
+   the cluster, giving the other nodes in the cluster a chance to recovery
+   operation.
+   "Node %u has caused %u recoveries in %.0f seconds - banning it for %u seconds"
+
+
+3, Verify that the recovery daemon can lock the recovery lock file.
+   At this stage this should be recovery master. 
+   If this operation fails it means we have a split brain and have to abort recovery.
+   "("ctdb_recovery_lock: Unable to open %s - (%s)"
+   "ctdb_recovery_lock: Failed to get recovery lock on '%s'"
+   "Unable to get recovery lock - aborting recovery"
+"ctdb_recovery_lock: Got recovery lock on '%s'"
+
+
+4, Log which node caused the recovery to be initiated.
+   This is a semi-accurate information message only.
+   This line does NOT mean that there has to be something wrong with the node listed.
+"Recovery initiated due to problem with node %u"
+
+
+5, Pull the names of all databases from all nodes and verify that these databases also
+   exists locally.
+   If a database is missing locally, just create it.
+   It is not an error if a database is missing locally. Databases are created on demand and
+   this could happen if it was one database which samba has never tried to access on the
+   local node.
+
+
+6, Check the list of databases on each remote node and create any databases that may be missing
+   on the remote node.
+"Recovery - created remote databases"
+
+
+7, Set recovery mode to ACTIVE on all remote nodes.
+
+
+8, Run the "startrecovery" eventscript on all nodes.
+
+   At this stage you will also get a few additional log entries, these are not
+   from the recovery daemon but from the main ctdb daemon due to running
+   the eventscript :
+"startrecovery eventscript has been invoked"
+"Monitoring has been disabled"
+"Executing event script ...
+...
+
+
+9, Create a new generation id and update the generation id and the VNNMAP on the local node
+   only.
+   This guarantees that the generation id will now be inconsistent across the cluster and
+   that if recovery fails a new recovery is attempted in the next iteration of the monitoring
+   loop.
+
+
+10, Start a TDB TRANSACTION on all nodes for all databases.
+   This is to ensure that if recovery is aborted or fails that we do not
+   modify any databases on only some of the nodes.
+"started transactions on all nodes"
+
+
+11, For each database, pull the content from all CONNECTED nodes and merge it into 
+    the TDB databases on the local node.
+    This merges the records from the remote nodes based on their serial numbers so we
+    only keep the most recent record found.
+"Recovery - pulled remote database 0x%x"
+
+
+12, For each database, perform a fast TDB WIPE operation to delete the entire TDB under the
+    transaction started above.
+
+
+13, For each database, drop all empty records.
+    Force the DMASTER field of all records to point to the recovery master.
+    Push the database out to all other nodes.
+
+    The PUSH process lists some additional log entries for each database of the
+    form :
+"Recovery - pulled remote database 0x..."
+"Recovery - pushed remote database 0x... of size ..."
+
+
+14, Commit all changes to all TDB databases.
+"Recovery - starting database commits"
+"Recovery - committed databases"
+
+
+15, Create a new VNNMAP of all CONNECTED nodes, create a new generation number
+    and piush this new VNNMAP out to all nodes.
+"Recovery - updated vnnmap"
+
+
+16, Update all nodes that the local node is the recovery master.
+"Recovery - updated recmaster"
+
+
+17, synchronize node flags across the cluster.
+"Recovery - updated flags"
+
+18, Change recovery mode back to NORMAL.
+"Recovery - disabled recovery mode"
+
+
+19, Re-allocate all public ip addresses across the cluster.
+"Deterministic IPs enabled. Resetting all ip allocations"
+
+    If the IP address allocation on the local node changes you might get
+    "Takeover of IP 10.0.0.201/24 on interface eth0"
+    "Release of IP 10.0.0.204/24 on interface eth0"
+
+"Recovery - takeip finished"
+
+
+20, Run the "recovered" eventscript on all nodes.
+"Recovery - finished the recovered event"
+
+    You will also get an entry from the local ctdb daemon itself that it has 
+    switched back to recovery mode NORMAL.
+"Recovery has finished"
+
+
+21, Broadcast a message to all samba daemons in the cluster that the databases have been
+    recovered. Samba will now do some additional checking/cleanup of the content in the stored
+    records.
+
+"Recovery complete"
+
+
+22. Finished. At this stage a 10 second timeout (ctdb listvars : rerecoverytimeout) is 
+    initiated. The cluster will not allow a new recovery to be performed until this timeout
+    has expired.
+
+"New recoveries supressed for the rerecovery timeout"
+"Rerecovery timeout elapsed. Recovery reactivated."
+
+
+
+
+
+
+
+Example : RECOVERY LOG ON RECMASTER
+====================================
+2008/12/01 09:57:28.110732 [ 4933]: 10.0.0.21:4379: node 10.0.0.24:4379 is dead:
+ 2 connected
+2008/12/01 09:57:28.110838 [ 4933]: Tearing down connection to dead node :3
+2008/12/01 09:57:28.967297 [ 4935]: server/ctdb_recoverd.c:2682 The vnnmap count
+ is different from the number of active nodes. 4 vs 3
+2008/12/01 09:57:28.967297 [ 4935]: server/ctdb_recoverd.c:1327 Starting do_reco
+very
+2008/12/01 09:57:28.967297 [ 4935]: ctdb_recovery_lock: Got recovery lock on '/g
+pfs/.ctdb/shared'
+2008/12/01 09:57:28.967297 [ 4935]: server/ctdb_recoverd.c:1355 Recovery initiat
+ed due to problem with node 0
+2008/12/01 09:57:28.967297 [ 4935]: server/ctdb_recoverd.c:1381 Recovery - creat
+ed remote databases
+2008/12/01 09:57:28.973543 [ 4933]: server/ctdb_recover.c:589 Recovery mode set 
+to ACTIVE
+2008/12/01 09:57:28.974823 [ 4933]: server/ctdb_recover.c:904 startrecovery even
+tscript has been invoked
+2008/12/01 09:57:29.187264 [ 4935]: server/ctdb_recoverd.c:1431 started transact
+ions on all nodes
+2008/12/01 09:57:29.187264 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0x42fe72c5
+2008/12/01 09:57:29.187264 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0x42fe72c5 of size 0
+2008/12/01 09:57:29.187264 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0x1421fb78
+2008/12/01 09:57:29.197262 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0x1421fb78 of size 0
+2008/12/01 09:57:29.197262 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0xc0bdde6a
+2008/12/01 09:57:29.197262 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0xc0bdde6a of size 0
+2008/12/01 09:57:29.197262 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0x17055d90
+2008/12/01 09:57:29.207261 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0x17055d90 of size 8
+2008/12/01 09:57:29.207261 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0x7bbbd26c
+2008/12/01 09:57:29.207261 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0x7bbbd26c of size 1
+2008/12/01 09:57:29.207261 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0xf2a58948
+2008/12/01 09:57:29.217259 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0xf2a58948 of size 51
+2008/12/01 09:57:29.217259 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0x92380e87
+2008/12/01 09:57:29.217259 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0x92380e87 of size 17
+2008/12/01 09:57:29.227258 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0x63501287
+2008/12/01 09:57:29.227258 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0x63501287 of size 1
+2008/12/01 09:57:29.227258 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0xe98e08b6
+2008/12/01 09:57:29.227258 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0xe98e08b6 of size 4
+2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0x2672a57f
+2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0x2672a57f of size 28
+2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulle
+d remote database 0xb775fff6
+2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushe
+d remote database 0xb775fff6 of size 6
+2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1440 Recovery - start
+ing database commits
+2008/12/01 09:57:29.297247 [ 4935]: server/ctdb_recoverd.c:1452 Recovery - commi
+tted databases
+2008/12/01 09:57:29.297247 [ 4935]: server/ctdb_recoverd.c:1502 Recovery - updat
+ed vnnmap
+2008/12/01 09:57:29.297247 [ 4935]: server/ctdb_recoverd.c:1511 Recovery - updat
+ed recmaster
+2008/12/01 09:57:29.297247 [ 4935]: server/ctdb_recoverd.c:1522 Recovery - updat
+ed flags
+2008/12/01 09:57:29.305235 [ 4933]: server/ctdb_recover.c:589 Recovery mode set 
+to NORMAL
+2008/12/01 09:57:29.307245 [ 4935]: server/ctdb_recoverd.c:1531 Recovery - disab
+led recovery mode
+2008/12/01 09:57:29.307245 [ 4935]: Deterministic IPs enabled. Resetting all ip 
+allocations
+2008/12/01 09:57:29.311071 [ 4933]: takeoverip called for an ip '10.0.0.201' tha
+t is not a public address
+2008/12/01 09:57:29.311186 [ 4933]: takeoverip called for an ip '10.0.0.202' tha
+t is not a public address
+2008/12/01 09:57:29.311204 [ 4933]: takeoverip called for an ip '10.0.0.203' tha
+t is not a public address
+2008/12/01 09:57:29.311299 [ 4933]: takeoverip called for an ip '10.0.0.204' tha
+t is not a public address
+2008/12/01 09:57:29.537210 [ 4935]: server/ctdb_recoverd.c:1542 Recovery - takei
+p finished
+2008/12/01 09:57:29.545404 [ 4933]: Recovery has finished
+2008/12/01 09:57:29.807169 [ 4935]: server/ctdb_recoverd.c:1551 Recovery - finis
+hed the recovered event
+2008/12/01 09:57:29.807169 [ 4935]: server/ctdb_recoverd.c:1557 Recovery complet
+e
+2008/12/01 09:57:29.807169 [ 4935]: server/ctdb_recoverd.c:1565 New recoveries s
+upressed for the rerecovery timeout
+2008/12/01 09:57:39.815648 [ 4935]: server/ctdb_recoverd.c:1567 Rerecovery timeo
+ut elapsed. Recovery reactivated.
+
+
+
+
+
+
+
+
diff --git a/ctdb/packaging/RPM/ctdb.spec b/ctdb/packaging/RPM/ctdb.spec
index d8bd76bcb30..b8ec19ffcd3 100644
--- a/ctdb/packaging/RPM/ctdb.spec
+++ b/ctdb/packaging/RPM/ctdb.spec
@@ -94,6 +94,7 @@ fi
 %attr(755,root,root) %{initdir}/ctdb
 
 %{_docdir}/ctdb/README.eventscripts
+%{_docdir}/ctdb/recovery-process.txt
 %{_sysconfdir}/ctdb/events.d/00.ctdb
 %{_sysconfdir}/ctdb/events.d/10.interface
 %{_sysconfdir}/ctdb/events.d/20.multipathd