QPID-6154: [Java Broker] HA - Handle rollback of node when use of weak durability has allowed nodes to diverge

git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@1633407 13f79535-47bb-0310-9956-ffa450edef68

2 files changed, 65 insertions, 39 deletions

diff --git a/qpid/doc/book/src/java-broker/Java-Broker-Appendix-Operational-Logging-Messages.xml b/qpid/doc/book/src/java-broker/Java-Broker-Appendix-Operational-Logging-Messages.xml
index 7d3d62b075..8b84605683 100644
--- a/qpid/doc/book/src/java-broker/Java-Broker-Appendix-Operational-Logging-Messages.xml
+++ b/qpid/doc/book/src/java-broker/Java-Broker-Appendix-Operational-Logging-Messages.xml
@@ -346,21 +346,27 @@
         </row>
         <row id="Java-Broker-Appendix-Operation-Logging-Message-BRK-1014">
           <entry morerows="1">BRK-1014</entry>
-          <entry>Message flow to disk active :  Message memory use <replaceable>size of all messages</replaceable> exceeds threshold <replaceable>threshold size</replaceable></entry>
+          <entry>Message flow to disk active : Message memory use <replaceable>size of all
+              messages</replaceable> exceeds threshold <replaceable>threshold
+            size</replaceable></entry>
         </row>
         <row>
           <entry>
-            <para>Indicates that the heap memory space occupied by messages has exceeded the threshold so the flow to disk feature has been activated.</para>
+            <para>Indicates that the heap memory space occupied by messages has exceeded the
+              threshold so the flow to disk feature has been activated.</para>
           </entry>
         </row>
-        
+
         <row id="Java-Broker-Appendix-Operation-Logging-Message-BRK-1015">
           <entry morerows="1">BRK-1015</entry>
-          <entry>Message flow to disk inactive :  Message memory use <replaceable>size of all messages</replaceable> within threshold <replaceable>threshold size</replaceable></entry>
+          <entry>Message flow to disk inactive : Message memory use <replaceable>size of all
+              messages</replaceable> within threshold <replaceable>threshold
+            size</replaceable></entry>
         </row>
         <row>
           <entry>
-            <para>Indicates that the heap memory space occupied by messages has fallen below the threshold so the flow to disk feature has been deactivated.</para>
+            <para>Indicates that the heap memory space occupied by messages has fallen below the
+              threshold so the flow to disk feature has been deactivated.</para>
           </entry>
         </row>
       </tbody>
@@ -1196,9 +1202,9 @@
         </row>
         <row>
           <entry>
-            <para>The node is no longer reachable. This may be as a result of the node being
-              stopped or a network partition may be preventing it from being connected. The node is
-              still a member of the group.</para>
+            <para>The node is no longer reachable. This may be as a result of the node being stopped
+              or a network partition may be preventing it from being connected. The node is still a
+              member of the group.</para>
           </entry>
         </row>
         <row id="Java-Broker-Appendix-Operation-Logging-Message-HA-1007">
@@ -1273,6 +1279,24 @@
               node groups only.</para>
           </entry>
         </row>
+        <row id="Java-Broker-Appendix-Operation-Logging-Message-HA-1014">
+          <entry morerows="1">HA-1014</entry>
+          <entry>HA-1014 : Diverged transactions discarded</entry>
+        </row>
+        <row>
+          <entry>
+            <para>This node is in the process of rejoining the group but has discovered that some
+              of its transactions differ from those of the current master. The node will
+              automatically roll-back (i.e. discard) the diverging transactions in order to be
+              allowed to rejoin the group. This situation can only usually occur as a result of use
+              of the weak durability options. These allow the group to operate with fewer than
+              quorum nodes and therefore allow the inconsistencies to develop. </para>
+            <para>On encountering this condition, it is <emphasis>strongly</emphasis> recommendend
+              to run an application level reconcilation to determine the data that has been
+              lost.</para>
+          </entry>
+        </row>
+
       </tbody>
     </tgroup>
   </table>
diff --git a/qpid/doc/book/src/java-broker/Java-Broker-High-Availability.xml b/qpid/doc/book/src/java-broker/Java-Broker-High-Availability.xml
index 25c56a8d14..8282beaa5e 100644
--- a/qpid/doc/book/src/java-broker/Java-Broker-High-Availability.xml
+++ b/qpid/doc/book/src/java-broker/Java-Broker-High-Availability.xml
@@ -120,9 +120,9 @@
             complete, all nodes will be considered equal.</para>
         </listitem>
         <listitem>
+          <para>Click the <literal>Add</literal> button on the Virtualhost Panel on the Broker
+            tab.</para>
           <para>
-            <para>Click the <literal>Add</literal> button on the Virtualhost Panel on the Broker
-              tab.</para>
             <orderedlist>
               <listitem>
                 <para>Give the Virtualhost node a unique name e.g. <literal>weather1</literal>. The
@@ -157,25 +157,23 @@
                   master.</para>
               </listitem>
             </orderedlist>
-            <para>
-              <figure>
-                <title>Creating 1st node in a group</title>
-                <mediaobject>
-                  <imageobject>
-                    <imagedata fileref="images/HA-Create-1.png" format="PNG" scalefit="1"/>
-                  </imageobject>
-                  <textobject>
-                    <phrase>Creating 1st node in a group</phrase>
-                  </textobject>
-                </mediaobject>
-              </figure>
-            </para>
+            <figure>
+              <title>Creating 1st node in a group</title>
+              <mediaobject>
+                <imageobject>
+                  <imagedata fileref="images/HA-Create-1.png" format="PNG" scalefit="1"/>
+                </imageobject>
+                <textobject>
+                  <phrase>Creating 1st node in a group</phrase>
+                </textobject>
+              </mediaobject>
+            </figure>
           </para>
         </listitem>
         <listitem>
+          <para>Now move to the second Broker to be the group. Click the <literal>Add</literal>
+            button on the Virtualhost Panel on the Broker tab of the second Broker.</para>
           <para>
-            <para>Now move to the second Broker to be the group. Click the <literal>Add</literal>
-              button on the Virtualhost Panel on the Broker tab of the second Broker.</para>
             <orderedlist>
               <listitem>
                 <para>Give the Virtualhost node a unique name e.g.
@@ -202,19 +200,17 @@
                 <para>Repeat these steps until you have added all the nodes to the group.</para>
               </listitem>
             </orderedlist>
-            <para>
-              <figure>
-                <title>Adding subsequent nodes to the group</title>
-                <mediaobject>
-                  <imageobject>
-                    <imagedata fileref="images/HA-Create-2.png" format="PNG" scalefit="1"/>
-                  </imageobject>
-                  <textobject>
-                    <phrase>Adding subsequent nodes to the group</phrase>
-                  </textobject>
-                </mediaobject>
-              </figure>
-            </para>
+            <figure>
+              <title>Adding subsequent nodes to the group</title>
+              <mediaobject>
+                <imageobject>
+                  <imagedata fileref="images/HA-Create-2.png" format="PNG" scalefit="1"/>
+                </imageobject>
+                <textobject>
+                  <phrase>Adding subsequent nodes to the group</phrase>
+                </textobject>
+              </mediaobject>
+            </figure>
           </para>
         </listitem>
 
@@ -351,6 +347,8 @@
           running behind for some reason (perhaps a full-GC). If a Master failure were to occur at
           that moment, the replicas would elect Replica-1 even though Replica-2 had the most recent
           transaction.</para>
+        <para>Transaction loss is reported by message <link
+            linkend="Java-Broker-Appendix-Operation-Logging-Message-HA-1014">HA-1014</link>.</para>
       </important>
     </section>
     <section id="Java-Broker-High-Availability-Behaviour-MinimumNumberOfNodes">
@@ -378,6 +376,8 @@
             >split-brain</ulink> in the event of a network partition. If used to temporarily restore
           service from fewer than quorum nodes, it is <emphasis>imperative</emphasis> to revert it
           to the Default value as the failed nodes are restored.</para>
+        <para>Transaction loss is reported by message <link
+          linkend="Java-Broker-Appendix-Operation-Logging-Message-HA-1014">HA-1014</link>.</para>
       </important>
     </section>
     <section id="Java-Broker-High-Availability-Behaviour-DesignatedPrimary">
@@ -397,7 +397,9 @@
         <para>It is imperative not to allow designated primary to be set on both nodes at once. To
           do so will mean, in the event of a network partition, a <ulink
             url="http://en.wikipedia.org/wiki/Split-brain_(computing)">split-brain</ulink> will
-          occur.</para>
+            occur.</para>
+        <para>Transaction loss is reported by message <link
+          linkend="Java-Broker-Appendix-Operation-Logging-Message-HA-1014">HA-1014</link>.</para>
       </important>
     </section>
   </section>