Add controller, worker classes and a new "build-distributed" command.

This commit introduces four new classes:

BuildController:
  * takes an app instance and a tempdir
  * allows to add BuildWorker objects
  * provides a build() method that takes a set of blobs and
    a build order that is then built by assigning work to the
    build workers as needed
  * the build() method takes care of polling the workers for
    their state, moving them between busy and idle states
    reliably, collect and print their output in a non-confusing
    order, and makes sure to wait for all workers to finish
    before processing the next group in the build order.
  * at this point, when waiting for one or more workers to
    become idle to assign them another blob to build, the
    controller always picks the worker that has been idling
    for the longest period of time. this can be changed later.

BuildWorker:
  * base class for all worker classes
  * takes a name and an app instance
  * has a idle_since datetime property
  * provides a build() method that takes a Blob object and builds
    it in whatever way the subclasses implement it
  * provides a check_complete(timeout) method that checks whether
    the worker has finished building the blob yet or not

LocalBuildWorker:
  * worker class for local builds that don't go through SSH
  * it uses morphlib.execute.Execute to run morph in a child process
    in build()
  * at the moment, this class executes "./morph" instead of "morph"
    as it assumes the user to run morph from its source tree. obviously,
    this will have to be fixed later.

RemoteBuildWorker:
  * doesn't implement anything yet, will be used for distributing
    work to other machines running morph via SSH

Notes:
  * At the moment, there is a degree of undesired redundancy when
    building a stratum in a worker, as this will cause the worker to
    rebuild all its dependencies. This will have to be fixed as it is
    avoidable and wastes a lot of time and processing power.

1 files changed, 119 insertions, 0 deletions

diff --git a/morphlib/buildcontroller.py b/morphlib/buildcontroller.py
new file mode 100644
index 00000000..a53bea06
--- /dev/null
+++ b/morphlib/buildcontroller.py
@@ -0,0 +1,119 @@
+# Copyright (C) 2012  Codethink Limited
+# 
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+import collections
+import time
+
+
+class BuildController(object):
+
+    def __init__(self, app, tempdir):
+        self.settings = app.settings
+        self.real_msg = app.msg
+        self.tempdir = tempdir
+        self.indent = 1
+
+        self.workers = set()
+        self.busy_workers = set()
+        self.idle_workers = set()
+
+        self.blobs = set()
+        self.build_order = collections.deque()
+
+    def indent_more(self):
+        self.indent += 1
+
+    def indent_less(self):
+        self.indent -= 1
+
+    def msg(self, text):
+        spaces = '  ' * self.indent
+        self.real_msg('%s%s' % (spaces, text))
+
+    def add_worker(self, worker):
+        self.workers.add(worker)
+        self.mark_idle(worker)
+
+    def wait_for_workers(self, need_idle=False, timeout=100):
+        # first, check if any of the busy workers are finished
+        while all(not x.check_complete(timeout) for x in self.busy_workers):
+            # wait and repeat if they are all busy and we have no idle workers
+            if need_idle and len(self.idle_workers) == 0:
+                self.msg('Waiting for idle workers...')
+                time.sleep(0.250)
+            else:
+                break
+
+        # get a list of all finished busy workers
+        finished = [x for x in self.busy_workers if x.check_complete(0)]
+
+        # log the result of all workers that we are moving from busy to idle
+        for worker in finished:
+            self.msg('Built %s using worker %s' % (worker.blob, worker))
+            for line in worker.output.split('\n'):
+                self.msg('> %s' % line)
+
+        # mark all finished workers as being idle
+        for worker in finished:
+            self.mark_idle(worker)
+
+    def wait_for_worker(self):
+        # wait for at least one worker to be idle
+        self.wait_for_workers(need_idle = True)
+
+        # sort idle workers by their idle timestamps (ascending)
+        idle_workers = sorted(self.idle_workers, key=lambda x: x.idle_since)
+
+        # return the worker that has been idling for the longest period of time
+        return idle_workers[0]
+
+    def build(self, blobs, build_order):
+        self.blobs = blobs
+        self.build_order = build_order
+
+        result = []
+
+        while len(build_order) > 0:
+            group = build_order.popleft()
+            group_str = ', '.join([x.morph.filename for x in group])
+            self.msg('Building parallel group %s' % group_str)
+            self.indent_more()
+
+            while len(group) > 0:
+                blob = group.pop()
+
+                worker = self.wait_for_worker()
+                self.msg('Distributing %s to worker %s' % (blob, worker))
+                self.mark_busy(worker)
+                worker.build(blob)
+
+            self.wait_for_workers(need_idle = False, timeout = None)
+
+            self.indent_less()
+
+        return result
+
+    def mark_idle(self, worker):
+        if worker not in self.idle_workers:
+            self.idle_workers.add(worker)
+        if worker in self.busy_workers:
+            self.busy_workers.remove(worker)
+
+    def mark_busy(self, worker):
+        if worker not in self.busy_workers:
+            self.busy_workers.add(worker)
+        if worker in self.idle_workers:
+            self.idle_workers.remove(worker)