Add ZK session-aware elections

This creates a session-aware election class which will set a flag
that indicates it has lost the underlying lock.  We can check this
flag when iterating to make sure that we don't continue to attempt
to operate when we have lost the lock underlying an election.

Some drivers had connection lost handling for the EventReceiverElection
at the driver level.  Those are updated to use the handling at the
election level for consistency as well as brevity.

Change-Id: I776f88d015acdfbf1487a85d8473cd174917e90f

1 files changed, 5 insertions, 2 deletions

diff --git a/zuul/nodepool.py b/zuul/nodepool.py
index 86ceaa51f..689c55bd7 100644
--- a/zuul/nodepool.py
+++ b/zuul/nodepool.py
@@ -121,8 +121,11 @@ class Nodepool(object):
             # completed event is added to the zk queue.
             try:
                 if self.election_won:
-                    self.emitStats(request)
-                    self._sendNodesProvisionedEvent(request)
+                    if self.election.is_still_valid():
+                        self.emitStats(request)
+                        self._sendNodesProvisionedEvent(request)
+                    else:
+                        self.stop_watcher_event.set()
             except Exception:
                 # If there are any errors moving the event, re-run the
                 # election.