Merge branch 'baserock/richardipsum/merge_liw_docs'

Reviewed by:
	Sam Thursfield
	Richard Maw
	Richard Ipsum

4 files changed, 324 insertions, 170 deletions

diff --git a/ARCH b/ARCH
index 271b2bc..a815c99 100644
--- a/ARCH
+++ b/ARCH
@@ -5,56 +5,53 @@ Introduction
 ============
 
 This is an architecture document for Lorry Controller. It is aimed at
-those who develop the software.
+those who develop the software, or develop against its HTTP API. See
+the file `README` for general information about Lorry Controller.
 
-Lorry is a tool in Baserock for mirroring code from whatever format
-upstream provides it into git repositories, converting them to git as
-needed. Lorry Controller is service, running on a Trove, which runs
-Lorry against all configured upstreams, including other Troves.
-
-Lorry Controller reads a configuration from a git repository. That
-configuration includes specifications of which upstreams to
-mirror/convert. This includes what upstream Troves to mirror. Lorry
-Controller instructs Lorry to push to a Trove's git repositories.
-
-Lorry specifications, and upstream Trove specifications, may include
-scheduling information, which the Lorry Controller uses to decide when
-to execute which specification.
 
 Requirements
 ============
 
 Some concepts/terminology:
 
-* CONFGIT is the git repository the Lorry Controller instance uses for
-  its configuration.
-* Lorry specification: which upstream version control repository or
-  tarball to mirror.
-* Trove specification: which upstream Trove to mirror. This gets
+* CONFGIT is the git repository Lorry Controller uses for its
+  configuration.
+
+* Lorry specification: the configuration to Lorry to mirror an
+  upstream version control repository or tarball. Note that a `.lorry`
+  file may contain several specifications.
+
+* Trove specification: which remote Trove to mirror. This gets
   broken into generated Lorry specifications, one per git repository
-  on the upstream Trove. There can be many Trove specifications to
+  on the other Trove. There can be many Trove specifications to
   mirror many Troves.
-* job: An instance of executing a Lorry specification. Each job has an
-  identifier and associated data (such as the output provided by the
-  running job, and whether it succeeded).
+
 * run queue: all the Lorry specifications (from CONFGIT or generated
-  from the Troe specifications) a Lorry Controller knows about; this
+  from the Trove specifications) a Lorry Controller knows about; this
   is the set of things that get scheduled. The queue has a linear
   order (first job in the queue is the next job to execute).
+
+* job: An instance of executing a Lorry specification. Each job has an
+  identifier and associated data (such as the output provided by the
+  running job, and whether it succeeded).
+
 * admin: a person who can control or reconfigure a Lorry Controller
-  instance.
+  instance. All users of the HTTP API are admins, for example.
 
-Original set of requirement, which have been broken down and detailed
+Original set of requirements, which have been broken down and detailed
 up below:
 
 *   Lorry Controller should be capable of being reconfigured at runtime
     to allow new tasks to be added and old tasks to be removed.
     (RC/ADD, RC/RM, RC/START)
+
 *   Lorry Controller should not allow all tasks to become stuck if one
     task is taking a long time. (RR/MULTI)
+
 *   Lorry Controller should not allow stuck tasks to remain stuck
     forever. (Configurable timeout? monitoring of disk usage or CPU to
     see if work is being done?) (RR/TIMEOUT)
+
 *   Lorry Controller should be able to be controlled at runtime to allow:
     - Querying of the current task set (RQ/SPECS, RQ/SPEC)
     - Querying of currently running tasks (RQ/RUNNING)
@@ -84,13 +81,14 @@ used elsewhere to refer to the exact requirement):
     * (RC/START) A Lorry Controller reads CONFGIT when it starts,
       updating its run queue if anything has changed.
 * (RT) Lorry Controller can controlled at runtime.
-    * (RT/KILL) An admin can get their Lorry Controller to stop a running job.
-    * (RT/TOP) An admin can get their Lorry Controller to move a Lorry spec to
-      the beginning of the run queue.
+    * (RT/KILL) An admin can get their Lorry Controller to stop a
+      running job.
+    * (RT/TOP) An admin can get their Lorry Controller to move a Lorry
+      spec to the beginning of the run queue.
     * (RT/BOT) An admin can get their Lorry Controller to move a Lorry
       spec to the end of the run queue.
-    * (RT/QSTOP) An admin can stop their Lorry Controller from scheduling any new
-      jobs.
+    * (RT/QSTOP) An admin can stop their Lorry Controller from
+      scheduling any new jobs.
     * (RT/QSTART) An admin can get their Lorry Controller to start
       scheduling jobs again.
 * (RQ) Lorry Controller can be queried at runtime.
@@ -143,15 +141,16 @@ Constraints
 
 Python is not good at multiple threads (partly due to the global
 interpreter lock), and mixing threads and executing subprocesses is
-quite tricky to get right in general. Thus, this design avoids using
-threads.
+quite tricky to get right in general. Thus, this design splits the
+software into a threaded web application (using the bottle.py
+framework) and one or more single-threaded worker processes to execute
+Lorry.
 
 Entities
 --------
 
-* An admin is a human being that communicates with the Lorry
-  Controller using an HTTP API. They might do it using a command line
-  client.
+* An admin is a human being or some software using the HTTP API to
+  communicate with the Lorry Controller.
 * Lorry Controller runs Lorry appropriately, and consists of several
   components described below.
 * The local Trove is where Lorry Controller tells its Lorry to push
@@ -163,28 +162,32 @@ Components of Lorry Controller
 ------------------------------
 
 * CONFGIT is a git repository for Lorry Controller configuration,
-  which the Lorry Controller can access and pull from. Pushing is not
-  required and should be prevented by Gitano. CONFGIT is hosted on the
-  local Trove.
+  which the Lorry Controller (see WEBAPP below) can access and pull
+  from. Pushing is not required and should be prevented by Gitano.
+  CONFGIT is hosted on the local Trove.
+
 * STATEDB is persistent storage for the Lorry Controller's state: what
   Lorry specs it knows about (provided by the admin, or generated from
   a Trove spec by Lorry Controller itself), their ordering, jobs that
-  have been run or are being run, information about the jobs, etc.
-  The idea is that the Lorry Controller process can terminate (cleanly
-  or by crashing), and be restarted, and continue approximately where
-  it was. Also, a persistent storage is useful if there are multiple
-  processes involved due to how bottle.py and WSGI work. STATEDB is
-  implemented using sqlite3.
+  have been run or are being run, information about the jobs, etc. The
+  idea is that the Lorry Controller process can terminate (cleanly or
+  by crashing), and be restarted, and continue approximately from
+  where it was. Also, a persistent storage is useful if there are
+  multiple processes involved due to how bottle.py and WSGI work.
+  STATEDB is implemented using sqlite3.
+
 * WEBAPP is the controlling part of Lorry Controller, which maintains
   the run queue, and provides an HTTP API for monitoring and
-  controller Lorry Controller. WEBAPP is implemented as a bottle.py
-  application.
+  controlling Lorry Controller. WEBAPP is implemented as a bottle.py
+  application. bottle.py runs the WEBAPP code in multiple threads to
+  improve concurrency.
+
 * MINION runs jobs (external processes) on behalf of WEBAPP. It
   communicates with WEBAPP over HTTP, and requests a job to run,
-  starts it, and while it waits, sends partial output to the WEBAPP,
-  and asks the WEBAPP whether the job should be aborted or not. MINION
-  may eventually run on a different host than WEBAPP, for added
-  scalability.
+  starts it, and while it waits, sends partial output to the WEBAPP
+  every few seconds, and asks the WEBAPP whether the job should be
+  aborted or not. MINION may eventually run on a different host than
+  WEBAPP, for added scalability.
 
 Components external to Lorry Controller
 ---------------------------------------
@@ -192,9 +195,10 @@ Components external to Lorry Controller
 * A web server. This runs the Lorry Controller WEBAPP, using WSGI so
   that multiple instances (processes) can run at once, and thus serve
   many clients.
-* bottle.py is a Python microframework for web applications. We
-  already have it in Baserock, where we use it for morph-cache-server,
-  and it seems to be acceptable.
+
+* bottle.py is a Python microframework for web applications. It sits
+  between the web server itself and the WEBAPP code.
+
 * systemd is the operating system component that starts services and
   processes.
 
@@ -203,17 +207,21 @@ How the components work together
 
 * Each WEBAPP instance is started by the web server, when a request
   comes in. The web server is started by a systemd unit.
+
 * Each MINION instance is started by a systemd unit. Each MINION
   handles one job at a time, and doesn't block other MINIONs from
   running other jobs. The admins decide how many MINIONs run at once,
   depending on hardware resources and other considerations. (RR/MULTI)
+
 * An admin communicates with the WEBAPP only, by making HTTP requests.
   Each request is either a query (GET) or a command (POST). Queries
   report state as stored in STATEDB. Commands cause the WEBAPP
   instance to do something and alter STATEDB accordingly.
+
 * When an admin makes changes to CONFGIT, and pushes them to the local
   Trove, the Trove's git post-update hook makes an HTTP request to
   WEBAPP to update STATEDB from CONFGIT. (RC/ADD, RC/RM)
+
 * Each MINION likewise communicates only with the WEBAPP using HTTP
   requests. MINION requests a job to run (which triggers WEBAPP's job
   scheduling), and then reports results to the WEBAPP (which causes
@@ -221,8 +229,10 @@ How the components work together
   continue running the job or not (RT/KILL). There is no separate
   scheduling process: all scheduling happens when there is a MINION
   available.
+
 * At system start up, a systemd unit makes an HTTP request to WEBAPP
   to make it refresh STATEDB from CONFGIT. (RC/START)
+
 * A timer unit for systemd makes an HTTP request to get WEBAPP to
   refresh the static HTML status page. (MON/STATIC)
 
@@ -236,51 +246,72 @@ The WEBAPP
 
 The WEBAPP provides an HTTP API as described below.
 
-Requests for admins:
+Run queue management:
 
-* `GET /1.0/status` causes WEBAPP to return a JSON object that
-  describes the state of Lorry Controller. This information is meant
-  to be programmatically useable and may or may not be the same as in
-  the HTML page.
 * `POST /1.0/stop-queue` causes WEBAPP to stop scheduling new jobs to
   run. Any currently running jobs are not affected. (RT/QSTOP)
+
 * `POST /1.0/start-queue` causes WEBAPP to start scheduling jobs
   again. (RT/QSTART)
 
 * `GET /1.0/list-queue` causes WEBAPP to return a JSON list of ids of
   all Lorry specifications in the run queue, in the order they are in
   the run queue. (RQ/SPECS)
-* `GET /1.0/lorry/<lorryspecid>` causes WEBAPP to return a JSON map
-  (dict) with all the information about the specified Lorry
-  specification. (RQ/SPEC)
+
 * `POST /1.0/move-to-top` with `path=lorryspecid` as the body, where
   `lorryspecid` is the id (path) of a Lorry specification in the run
   queue, causes WEBAPP to move the specified spec to the head of the
   run queue, and store this in STATEDB. It doesn't affect currently
   running jobs. (RT/TOP)
+
 * `POST /1.0/move-to-bottom` with `path=lorryspecid` in the body is
   like `/move-to-top`, but moves the job to the end of the run queue.
   (RT/BOT)
 
+Running job management:
+
 * `GET /1.0/list-running-jobs` causes WEBAPP to return a JSON list of
   ids of all currently running jobs. (RQ/RUNNING)
+
 * `GET /1.0/job/<jobid>` causes WEBAPP to return a JSON map (dict)
   with all the information about the specified job.
+
 * `POST /1.0/stop-job` with `job_id=jobid` where `jobid` is an id of a
   running job, causes WEBAPP to record in STATEDB that the job is to
   be killed, and waits for it to be killed. (Killing to be done when
   MINION gets around to it.) This request returns as soon as the
   STATEDB change is done.
+
 * `GET /1.0/list-all-jobs` causes WEBAPP to return a JSON list of ids
   of all jobs, running or finished, that it knows about. (RQ/ALLJOBS)
+
 * `POST /1.0/remove-job` with `job_id=jobid` in the body, removes a
   stopped job from the state database.
+
 * `POST /1.0/remove-ghost-jobs` looks for any running jobs in STATEDB
   that haven't been updated (with `job-update`, see below) in a long
   time (see `--ghost-timeout`), and marks them as terminated. This is
   used to catch situations when a MINION fails to tell the WEBAPP that
   a job has terminated.
 
+Other status queries:
+
+* `GET /1.0/status` causes WEBAPP to return a JSON object that
+  describes the state of Lorry Controller. This information is meant
+  to be programmatically useable and may or may not be the same as in
+  the HTML page.
+
+* `GET /1.0/status-html` causes WEBAPP to return an HTML page that
+  describes the state of Lorry Controller. This also updates an
+  on-disk copy of the HTML page, which the web server is configured to
+  serve using a normal HTTP request. This is the primary interface for
+  human admins to look at the state of Lorry Controller. (MON/STATIC)
+
+* `GET /1.0/lorry/<lorryspecid>` causes WEBAPP to return a JSON map
+  (dict) with all the information about the specified Lorry
+  specification. (RQ/SPEC)
+
+
 Requests for MINION:
 
 * `GET /1.0/give-me-job` is used by MINION to get a new job to run.
@@ -290,6 +321,7 @@ Requests for MINION:
   to do, and MINION will then sleep for a while before it tries again.
   WEBAPP updates STATEDB to record that the job is allocated to a
   MINION.
+
 * `POST /1.0/job-update` is used by MINION to push updates about the
   job it is running to WEBAPP. The body sets fields `exit` (exit code
   of program, or `no` if not set), `stdout` (some output from the
@@ -308,17 +340,21 @@ Other requests:
 * `POST /1.0/read-configuration` causes WEBAPP to update its copy of
   CONFGIT and update STATEDB based on the new configuration, if it has
   changed. Returns OK/ERROR status. (RC/ADD, RC/RM, RC/START)
+  
+  This is called by systemd units at system startup and periodically
+  (perhaps once a minute) otherwise. It can also be triggered by an
+  admin (there is a button on the `/1.0/status-html` web page).
+
 * `POST /1.0/ls-troves` causes WEBAPP to refresh its list of
   repositories in each remote Trove, if the current list is too old
   (see the `ls-interval` setting for each remote trove in
   `lorry-controller.conf`). This gets called from a systemd timer unit
   at a suitable interval.
+
 * `POST /1.0/force-ls-troves` causes the repository refresh to happen
-  for all remote Troves, regardless of whether it is due or not.
-* `GET /1.0/status-html` causes WEBAPP to return an HTML page that
-  describes the state of Lorry Controller. This also updates an
-  on-disk copy of the HTML page, which the web server is configured to
-  serve using a normal HTTP request. (MON/STATIC)
+  for all remote Troves, regardless of whether it is due or not. This
+  can be called manually by an admin.
+
 
 The MINION
 ----------
@@ -356,3 +392,87 @@ WEBAPP knows about. It has the following columns:
   Lorry when a job is run.
 * `generated` is set to 0 or 1, depending on if the Lorry came from an
   actual `.lorry` file or was generated by Lorry Controller.
+
+
+Code structure
+==============
+
+The Lorry Controller code base is laid out as follows:
+
+* `lorry-controller-webapp` is the main program of WEBAPP. It sets up
+  the bottle.py framework. All the implementations for the various
+  HTTP requests are in classes in the `lorrycontroller` Python
+  package, as subclasses of the `LorryControllerRoute` class. The main
+  program uses introspection ("magic") to find the subclasses
+  automatically and sets up the bottle.py routes correctly. This makes
+  it possible to spread the code into simple classes; bottle's normal
+  way (with the `@app.route` decorator) seemed to make that harder and
+  require everything in the same class.
+
+* `lorrycontroller` is a Python package with the HTTP request
+  handlers, management of STATEDB, plus some helpful utilities.
+
+* `lorry-controller-minion` is the entirety of the MINION, except that
+  it uses the `lorrycontroller.setup_proxy` function.
+  The MINION is kept very simple on purpose: all the interesting logic
+  is in the WEBAPP instead.
+
+* `static` has static content to be served over HTTP. Primarily, the
+  CSS file for the HTML interfaces. When LC is integrated within the
+  Trove, the web server gets configured to serve these files directly.
+  The `static` directory will be accessible over plain HTTP on port
+  80, and on port 12765 via the WEBAPP, to allow HTML pages to refer
+  to it via a simple path.
+
+* `templates` contains bottle.py HTML templates for various pages.
+
+* `etc` contains files to be installed in `/etc` when LC is installed
+  on a Baserock system. Primarily this is the web server (lighttpd)
+  configuration to invoke WEBAPP.
+
+* `units` contains various systemd units that start services and run
+  time-based jobs.
+
+* `yarns.webapp` contains an integration test suite for WEBAPP.
+  This is run by the `./check` script. The `./test-wait-for-port`
+  script is used by the yarns.
+
+Example
+-------
+
+As an example, to modify how the `/1.0/status-html` request works, you
+would look at its implementation in `lorrycontroller/status.py`, and
+perhaps also the HTML templates in `templates/*.tpl`.
+
+STATEDB
+-------
+
+The persistent state of WEBAPP is stored in an Sqlite3 database. All
+access to STATEDB within WEBAPP is via the
+`lorrycontroller/statedb.py` code module. That means there are no SQL
+statements outside `statedb.py` at all, nor is it OK to add any. If
+the interface provided by the `StateDB` class isn't sufficient, then
+modify the class suitably, but do not add any new SQL outside it.
+
+All access from outside of WEBAPP happens via WEBAPP's HTTP API.
+Only the WEBAPP is allowed to touch STATEDB in any way.
+
+The bottle.py framework runs multiple threads of WEBAPP code. The
+threads communicate only via STATEDB. There is no shared state in
+memory. SQL's locking is used for mutual exclusion.
+
+The `StateDB` class acts as a context manager for Python's `with`
+statements to provide locking. To access STATEDB with locking, use
+code such as this:
+
+    with self.open_statedb() as statedb:
+        troves = statedb.get_troves()
+        for trove in troves:
+            statedb.remove_trove(troves)
+
+The code executed by the `with` statement is run under lock, and the
+lock gets released automatically even if there is an exception.
+
+(You could manage locks manually. It's a good way to build character
+and learn why using the context manager is really simple and leads to
+more correct code.)
diff --git a/README b/README
index e869a76..00da0bc 100644
--- a/README
+++ b/README
@@ -1,103 +1,169 @@
 README for lorry-controller
 ===========================
 
-Lorry Controller mirrors Troves and mirrors or converts upstream
-projects into git repositories on the local Trove. Lorry Controller
-reads a configuration file (see below) and runs the Lorry program
-against the intended targets at suitable intervals.
+Overview
+--------
 
-Lorry Controller configuration
+Lorry Controller, or LC for short, manages the importing of source
+code from external sources into git repositories on a Trove. A Trove
+is a component in the Baserock system for hosting source code, and LC
+runs inside a Trove.
+
+LC uses the Lorry tool to do the actual import. Lorry can read code
+from several different version control systems, and convert them to
+git. External repositories can be specfied individually, as Lorry
+`.lorry` specification files. In addition, LC can be told to mirror
+all the git repositories on another Trove.
+
+LC runs Lorry for the right external repositories, and takes care of
+running a suitable number of Lorry instances concurrently, and
+recovering from any problems. LC has a web based administration
+interface, and an HTTP API for reporting and controlling its state.
+
+This README file documents the LC configuration file and general use.
+For the architecture of LC and the HTTP API, see the `ARCH` file.
+
+Lorry Controller configuration: overview
 ------------------------------
 
-Lorry Controller reads a configuration file of the following format.
-
-* The file uses JSON syntax.
-* The file is a list.
-* Each item in the list is a mapping specifying a Trove or a set of
-  `.lorry` files.
-* A Trove spec specifies another Trove to mirror completely.
-* A Lorry spec specifies a set of Lorry specification (`.lorry` files)
-  for individual project repositories to convert or mirror to git on
-  the local Trove.
-
-Each spec (mapping) has a number of key/value pairs. The following are
-shared between Trove and Lorry specs:
-
-* `type` is the type of the spec; value MUST be either `trove` or
-  `lorries`.
-* `interval` specifies how often Lorry Controller should mirror the
-  repositories in the spec. See below for INTERVAL.
-* `protocol`: specifies how Lorry Controller (and Lorry) should talk
-  to remove Troves. Allowed values are `ssh`, `https`, `http`. This
-  field is mandatory.
-* `auth`: Specifies how to authenticate to the remote Trove over
-  https. The is an optional field. If present, it should be a
-  dictionary with the fields `username` and `password`.
-* Additionally, the following seem to be supported by an old version
-  of Lorry Controller, but are ignored by the new Lorry Controller:
-  `uuid`, `serial`, `create`, `destroy`, `stagger`, `tarball`.
-
-Trove specs have the following keys:
-
-* `trovehost` is the other Trove to mirror; a domain name or IP
-  address. It is mandatory.
-* `ls-interval` determines how often should Lorry Controller query the
-  other Trove for a list of repositories it may mirror. See below for
-  INTERVAL. `ls-interval` is mandatory.
-* `prefixmap` maps repository path prefixes from the other Trove to
-  the local Trove. It is mandatory in a Trove spec. If the remote
-  prefix is `foo`, and the local prefix is `bar`, then remote
-  repository `foo/baserock/yeehaa` gets mirrored to local repository
-  `bar/baserock/yeehaa`. If the remote Trove has a repository that
-  does not match a prefix, that repository gets ignored.
-* `ignore` is a list of git repositories from the other Trove that
-  should NOT be mirrored. Each list element is a path to the git
-  repository (not including leading slash). `ignore` is optional.
+Lorry Controller has two levels of configuration. The first level is
+command line options and configuration files. This level specifies
+things such as log levels, network addresses to listen on, and such.
+Most importantly, this level specifies the location of the second
+level. For information about these options, run
+`lorry-controller-webapp --help` to get a list of them.
 
-An INTERVAL value (for `interval` or `ls-interval`) is number and a
-unit to indicate a time interval. Allowed units are minutes (`m`),
-hours (`h`), and days (`d`), expressed as single-letter codes in upper
-or lower case.
+The second level is a git repository that specifies which external
+repositories and other Troves to import into the Trove LC runs on.
+This git repository is referred to as CONFGIT in documentation, and is
+specified with the the `--confgit-url` command line option, or the
+`confgit-url` key in the configuration file. The configuration file
+could contain this, for example:
 
-Lorry specs have the following keys:
+    [config]
+    confgit-url = ssh://git@localhost/baserock/local-config/lorries
 
-* `prefix` is a path prefix to be prepended to all repositories
-  created from the `.lorry` files from this spec. It is mandatory.
-* `globs` is a list of globs (as strings) for `.lorry` files to use.
-  The glob is matched in the directory containing the configuration
-  file in which this spec is. It is OK for the globs to not match
-  anything. A `globs` entry is mandatory, however.
+The system integration of a Trove automatically includes a
+configuration file that contains a configuration such as the above.
+The URL contains the name of the Trove, so it needs to be customised
+for each Trove, but as long as you're only using LC as part of a
+Baserock Trove, it's all taken care of for you automatically.
 
-A fairly minimal example for mirroring `git.baserock.org` and using
-local `.lorry` files.
+
+The CONFGIT repository
+----------------------
+
+The CONFGIT repository must contain at least the file
+`lorry-controller.conf`. It may also contain other files, including
+`.lorry` files for Lorry, but all other files are ignored unless
+referenced by `lorry-controller.conf`.
+
+
+
+The `lorry-controller.conf` file
+--------------------------------
+
+`lorry-controller.conf` is a JSON file containing a list of maps. Each
+map specifies another Trove, or one set of `.lorry` files. Here's an
+example that tells LC to mirror the `git.baserock.org` Trove and
+anything in the `open-source-lorries/*.lorry` files (if any exist).
 
     [
         {
-            "type": "trove",
-            "trovehost": "git.baserock.org",
-            "ls-interval": "4d",
-            "interval": "2h",
+            "ignore": [
+                "baserock/lorries"
+            ], 
+            "interval": "2H", 
+            "ls-interval": "4H", 
             "prefixmap": {
-                "baserock": "baserock",
+                "baserock": "baserock", 
                 "delta": "delta"
-            }
+            }, 
+            "protocol": "http", 
+            "tarball": "always", 
+            "trovehost": "git.baserock.org", 
+            "type": "trove"
         },
         {
             "type": "lorries",
-            "interval": "1h",
+            "interval": "6H",
+            "create": "always",
+            "destroy": "never",
             "prefix": "delta",
+            "tarball": "always",
             "globs": [
                 "open-source-lorries/*.lorry"
             ]
         }
     ]
 
+A Trove specification (map) uses the following mandatory keys:
+
+* `type: trove` -- specify it's a Trove specifiation.
+
+* `trovehost` -- the other Trove to mirror; a domain name or IP
+  address.
+
+* `protocol` -- specify how Lorry Controller (and Lorry) should talk
+  to other Troves. Allowed values are `ssh`, `https`, `http`.
+
+* `prefixmap` -- map repository path prefixes from the other Trove to
+  the local Trove. If the remote prefix is `foo`, and the local prefix
+  is `bar`, then remote repository `foo/baserock/yeehaa` gets mirrored
+  to local repository `bar/baserock/yeehaa`. If the remote Trove has a
+  repository that does not match a prefix, that repository gets
+  ignored.
+
+* `ls-interval` -- determine how often should Lorry Controller query
+  the other Trove for a list of repositories it may mirror. See below
+  for how the value is interpreted. The default is 24 hours.
+
+* `interval` -- specify how often Lorry Controller should mirror the
+  repositories in the spec. See below for INTERVAL. The default
+  interval is 24 hours.
+
+Additionally, the following optional keys are allowed in Trove
+specifications:
+
+* `ignore` -- a list of git repositories from the other Trove that
+  should NOT be mirrored. Each list element is a path to the git
+  repository (not including leading slash).
+
+* `auth` -- specify how to authenticate to the remote Trove over https
+  (only). It should be a dictionary with the fields `username` and
+  `password`.
+
+A Lorry specification (map) uses the following keys, all of them
+mandatory:
+
+* `type: lorries` -- specify it's a Trove specifiation.
+
+* `interval` -- identical in meaning to the `interval` in a
+  Trove specification.
+
+* `prefix` -- a path prefix to be prepended to all repositories
+  created from the `.lorry` files from this spec.
+
+* `globs` -- a list of globs (as strings) for `.lorry` files to use.
+  The glob is matched in the directory containing the configuration
+  file in which this spec is. It is OK for the globs to not match
+  anything.
+
+For backwards compatibility with another implementation of Lorry
+Controller, other fields in either type of specification are allowed
+and silently ignored.
+
+An INTERVAL value (for `interval` or `ls-interval`) is a number and a
+unit to indicate a time interval. Allowed units are minutes (`m`),
+hours (`h`), and days (`d`), expressed as single-letter codes in upper
+or lower case.
+
 The syntax of `.lorry` files is specified by the Lorry program; see
 its documentation for details.
 
 
-HTTP proxy configuration
-------------------------
+HTTP proxy configuration: `proxy.conf`
+--------------------------------------
 
 Lorry Controller will look for a file called `proxy.conf` in the same
 directory as the `lorry-controller.conf` configuration file.
diff --git a/exterminate-jobs-whose-minion-is-lost b/exterminate-jobs-whose-minion-is-lost
deleted file mode 100644
index 4e9ff01..0000000
--- a/exterminate-jobs-whose-minion-is-lost
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/bin/sh
-#
-# Sometimes a MINION dies and never tells WEBAPP that a job is
-# finished. This makes the list of jobs grow to max size, which
-# stops WEBAPP from scheduling more jobs to run. This scripts
-# EXTERMINATEs such jobs. Do NOT run it casually, you MUST check
-# that it is OK to EXTERMINATE the job first. Davros has spoken.
-#
-# This is a STOPGAP until Lorry Controller deals with this
-# automatically.
-
-# Not using -e so that curl failing won't stop things. curl may
-# fail if the job isn't in the expected state. We want to
-# EXTERMINATE it anyway.
-set -u
-
-for jobid in "$@"
-do
-	curl -s -X POST --data "job_id=$jobid" http://localhost:12765/1.0/stop-job
-	curl -s -X POST --data "job_id=$jobid&exit=125&stdout=&stderr=" http://localhost:12765/1.0/job-update
-	curl -s -X POST --data "job_id=$jobid" http://localhost:12765/1.0/remove-job
-done > /dev/null
diff --git a/lorry-controller.morph b/lorry-controller.morph
deleted file mode 100644
index a16f9d9..0000000
--- a/lorry-controller.morph
+++ /dev/null
@@ -1,10 +0,0 @@
-build-system: python-distutils
-kind: chunk
-name: lorry-controller
-post-install-commands:
-- install -d 0755 "$DESTDIR/etc/lighttpd"
-- install -m 0644 -D etc/lighttpd/*.conf "$DESTDIR/etc/lighttpd/."
-- |
-  TGT="$DESTDIR/usr/lib/systemd/system"
-  install -d "$TGT/multi-user.target.wants"
-  install -m 0644 units/*.service units/*.timer "$TGT/."