Creation of Baserock definitions format specification repo

This contains:

- textual description of the Baserock definitions format, and the list
  of changes since version 0, taken from: git://baserock.branchable.com/

- migrations and schemas taken from
  git://git.baserock.org/baserock/baserock/definitions.

22 files changed, 2869 insertions, 0 deletions

diff --git a/changelog.mdwn b/changelog.mdwn
new file mode 100644
index 0000000..82dbb52
--- /dev/null
+++ b/changelog.mdwn
@@ -0,0 +1,93 @@
+[[!meta title="Baserock definitions format, version history"]]
+
+This page describes the history of the Baserock definitions format.
+
+FIXME: A changelog should no longer be needed now that the spec is
+defined in a version control repository.
+
+See also: [[current]] version, and [[planned]] versions.
+
+# Version 7
+
+[Version 7](http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2015-July/013220.html) adds a DEFAULTS file that specifies predefined build systems and predefined splitting rules. Previously it was up to the build tool to define these.
+
+Note that versions of Morph before commit [d5c616d8287](http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/commit/?id=d5c616d8287d94514d1323c7b5e6d6d2936a942d) do not implement version 7 correctly.
+
+# Version 6
+
+In definitions version 6, build system autodetection no longer happens. This means
+that any chunk that wants to use one of the predefined build systems (those built
+into Morph) must say so explicitly, using the 'build-system' field.
+
+The build-system field for a chunk can now be specified within a stratum that
+contains it. Previously you needed to add a .morph file for a chunk in order
+to specify its build-system, but we want to avoid needing a .morph file for
+components that follow standard patterns.
+
+Previously, if build-system wasn't given, Morph would scan the contents of the chunk's
+Git repo and try to autodetect which build system was used. This could be slow, could
+fail in confusing ways, and meant that to fully parse definitions you needed access to
+some or all of the repos they referenced.
+
+The chosen build-system affects which predefined command sequences are set for a chunk.
+It is valid to omit the field if a chunk has its own build commands defined in a .morph
+file. When listing the chunks included in a stratum, either 'morph' or 'build-system'
+*must* be specified, but not both (to avoid the possibility of conflicting values).
+
+# Version 5
+
+Morph commit [7f2ccd3d3095a79ad0f3cd0215d062415b20e083](http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/commit/?id=7f2ccd3d3095a79ad0f3cd0215d062415b20e083) introduces version 5 of the definitions format.
+
+Morph supports defining `strip-commands` in chunk definitions to strip debug symbols from binaries in `DESTDIR`, and default strip commands have been set to strip all elf binaries that are executable or named like a shared library.
+
+This also sets `PYTHONPATH` such that write extensions may use libraries from definitions.git. Write extensions which require this functionality may declare that they are version 5 to prevent versions of morph which don't support this functionality, to produce a version mismatch error rather than deployments failing to locate libraries.
+
+# Version 4
+
+Morph commit [c373f5a403b0ec](http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/commit/?h=c373f5a403b0ec84834d2f04fd1efac3792a7d35) introduces version 4 of the definitions format. In older versions of Morph the install-files.configure extension would crash if it tried to overwrite a symlink. This bug is fixed in the version of Morph that can build definitions version 4.
+
+If you need to overwrite a symlink at deploytime using install-files.configure, please use VERSION to 4 or above in your definitions.git repo so older versions of Morph gracefully refuse to deploy, instead of crashing.
+
+We have now moved all .configure and .write extensions into the definitions.git repository. Changes like this will not require a version numbering change in future.
+
+# Version 3
+
+Since morph commit [154a760fb88](http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/commit/?h=154a760fb884cee14c2604b8bfbe52b0e7c0d4b1) morph supports a new architecture (armv5)
+
+This is effectively a change in the definitions format, as old morph doesn't recognize this architecture and will fail if a system with this architecture is added to definitions
+
+Version 3 also allows the new `install-essential-files.configure` extension to be used.
+
+
+# Version 2
+
+1. definitions repo commit [db1fe6e41](http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/commit/?h=db1fe6e41bebf7da71d11fe9bc492ede1821f57b)
+1. Version 2 makes paths to non-existent chunk morphs invalid, prior to version 2 morph would simply ignore paths to non-existent chunk morphs and either use the morph in the chunk repo (if there was one) or run build system detection.
+
+   With version 2, if morph encounters a path to a chunk morph that doesn't exist it will error with an error message such as:
+
+   "ERROR: Couldn't find morphology: strata/cats/xattr.morph referenced in strata/swift.morph"
+
+
+# Version 1
+
+The 'build-depends' parameter was made optional. It was previously
+mandatory to specify 'build-depends' for a chunk, even if it was an
+empty list.
+
+
+# Version 0
+
+As of [this commit](http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/commit/?id=c2d6bf758845076eca6eb71af09df77165993270) morph was updated to check for the presence of a VERSION file, and to parse it for version:
+
+Morph will fail if:
+
+- VERSION file exists
+- and it is a yaml file
+- and it is a dict
+- and has the key 'version'
+- and the contents of the key 'version' is an int
+- and that int is in the list of non_supported_versions (empty
+  at the moment)
+
+Otherwise, morph will assume Version 0 until we add future functionality. Thus Version 0 is default behaviour.
diff --git a/migrations/000-version-info.py b/migrations/000-version-info.py
new file mode 100755
index 0000000..2bff51f
--- /dev/null
+++ b/migrations/000-version-info.py
@@ -0,0 +1,49 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Migration to Baserock Definitions format version 0.
+
+The format of version 0 is not formally specified, except by the Morph
+codebase. It marks the starting point of the work to formalise the Baserock
+Definitions format.
+
+'''
+
+
+import os
+import sys
+
+import migrations
+
+
+TO_VERSION = 0
+
+
+try:
+    if os.path.exists('./VERSION'):
+        # This will raise an exception if the VERSION file is invalid, which
+        # might be useful.
+        migrations.check_definitions_version(TO_VERSION)
+
+        sys.stdout.write("Nothing to do.\n")
+        sys.exit(0)
+    else:
+        sys.stdout.write("No VERSION file found, creating one.\n")
+        migrations.set_definitions_version(TO_VERSION)
+        sys.exit(0)
+except RuntimeError as e:
+    sys.stderr.write("Error: %s\n" % e.message)
+    sys.exit(1)
diff --git a/migrations/001-empty-build-depends.py b/migrations/001-empty-build-depends.py
new file mode 100755
index 0000000..5d4296d
--- /dev/null
+++ b/migrations/001-empty-build-depends.py
@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Migration to Baserock Definitions format version 1.
+
+In version 1, the 'build-depends' parameter was made optional. It was
+previously mandatory to specify 'build-depends' for a chunk, even if it was an
+empty list.
+
+'''
+
+
+import sys
+import warnings
+
+import migrations
+
+
+TO_VERSION = 1
+
+
+def check_empty_build_depends(contents, filename):
+    assert contents['kind'] == 'stratum'
+
+    valid = True
+    for chunk_ref in contents.get('chunks', []):
+        if 'build-depends' not in chunk_ref:
+            chunk_ref_name = chunk_ref.get('name', chunk_ref.get('morph'))
+            warnings.warn(
+                "%s:%s has no build-depends field, which "
+                "is invalid in definitions version 0." %
+                (contents['name'], chunk_ref_name))
+            valid = False
+
+    return valid
+
+
+def remove_empty_build_depends(contents, filename):
+    assert contents['kind'] == 'stratum'
+
+    changed = False
+    for chunk_ref in contents.get('chunks', []):
+        if 'build-depends' in chunk_ref:
+            if len(chunk_ref['build-depends']) == 0:
+                del chunk_ref['build-depends']
+                changed = True
+
+    return changed
+
+
+try:
+    if migrations.check_definitions_version(TO_VERSION - 1):
+        success = migrations.process_definitions(
+            path='.', kinds=['stratum'],
+            validate_cb=check_empty_build_depends,
+            modify_cb=remove_empty_build_depends)
+        if success:
+            migrations.set_definitions_version(TO_VERSION)
+            sys.stdout.write("Migration completed successfully.\n")
+            sys.exit(0)
+        else:
+            sys.stderr.write("Migration failed due to warnings.\n")
+            sys.exit(1)
+    else:
+        sys.stdout.write("Nothing to do.\n")
+        sys.exit(0)
+except RuntimeError as e:
+    sys.stderr.write("Error: %s\n" % e.message)
+    sys.exit(1)
diff --git a/migrations/002-missing-chunk-morphs.py b/migrations/002-missing-chunk-morphs.py
new file mode 100755
index 0000000..2c93804
--- /dev/null
+++ b/migrations/002-missing-chunk-morphs.py
@@ -0,0 +1,73 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Migration to Baserock Definitions format version 2.
+
+In version 2, the processing of the 'morph:' field within stratum .morph files
+became more strict.  This migration checks whether definitions are valid
+according to version 2 of the format.
+
+'''
+
+
+import os
+import sys
+import warnings
+
+import migrations
+
+
+TO_VERSION = 2
+
+
+def check_missing_chunk_morphs(contents, filename):
+    assert contents['kind'] == 'stratum'
+
+    valid = True
+
+    for chunk_ref in contents.get('chunks', []):
+        if 'morph' in chunk_ref:
+            chunk_path = os.path.join('.', chunk_ref['morph'])
+            if not os.path.exists(chunk_path):
+                # There's no way we can really fix this, so
+                # just warn and say the migration failed.
+                warnings.warn(
+                    "%s points to non-existant file %s" %
+                    (contents['name'], chunk_ref['morph']))
+                valid = False
+
+    return valid
+
+
+try:
+    if migrations.check_definitions_version(TO_VERSION - 1):
+        safe_to_migrate = migrations.process_definitions(
+            kinds=['stratum'], validate_cb=check_missing_chunk_morphs)
+
+        if not safe_to_migrate:
+            sys.stderr.write(
+                "Migration failed due to one or more warnings.\n")
+            sys.exit(1)
+        else:
+            migrations.set_definitions_version(TO_VERSION)
+            sys.stdout.write("Migration completed successfully.\n")
+            sys.exit(0)
+    else:
+        sys.stdout.write("Nothing to do.\n")
+        sys.exit(0)
+except RuntimeError as e:
+    sys.stderr.write("Error: %s\n" % e.message)
+    sys.exit(1)
diff --git a/migrations/003-arch-armv5.py b/migrations/003-arch-armv5.py
new file mode 100755
index 0000000..58eb79d
--- /dev/null
+++ b/migrations/003-arch-armv5.py
@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Migration to Baserock Definitions format version 3.
+
+In version 3, there were two additions:
+
+    - the 'armv5' architecture
+    - the install-essential-files.configure configuration extension
+
+This migration checks that neither of these are in use in the input (version 2)
+definitions. Which isn't particularly useful.
+
+'''
+
+
+import sys
+import warnings
+
+import migrations
+
+
+TO_VERSION = 3
+
+
+def check_arch(contents, filename):
+    assert contents['kind'] == 'system'
+
+    valid = True
+
+    if contents['arch'] == 'armv5':
+        warnings.warn(
+            "%s uses armv5 architecture that is not understood until version "
+            "3." % filename)
+        valid = False
+
+    return valid
+
+
+def check_configuration_extensions(contents, filename):
+    assert contents['kind'] == 'system'
+
+    valid = True
+
+    for extension in contents.get('configuration-extensions', []):
+        if extension == 'install-essential-files':
+            warnings.warn(
+                "%s uses install-essential-files.configure extension, which "
+                "was not present in morph.git until commit 423dc974a61f1c0 "
+                "(tag baserock-definitions-v3)." % filename)
+            valid = False
+
+    return valid
+
+
+try:
+    if migrations.check_definitions_version(TO_VERSION - 1):
+        safe_to_migrate = migrations.process_definitions(
+            kinds=['system'], validate_cb=check_arch)
+        safe_to_migrate = migrations.process_definitions(
+            kinds=['system'], validate_cb=check_configuration_extensions)
+
+        if not safe_to_migrate:
+            sys.stderr.write(
+                "Migration failed due to one or more warnings.\n")
+            sys.exit(1)
+        else:
+            migrations.set_definitions_version(TO_VERSION)
+            sys.stdout.write("Migration completed successfully.\n")
+            sys.exit(0)
+    else:
+        sys.stdout.write("Nothing to do.\n")
+        sys.exit(0)
+except RuntimeError as e:
+    sys.stderr.write("Error: %s\n" % e.message)
+    sys.exit(1)
diff --git a/migrations/004-install-files-overwrite-symlink.py b/migrations/004-install-files-overwrite-symlink.py
new file mode 100755
index 0000000..6853dcb
--- /dev/null
+++ b/migrations/004-install-files-overwrite-symlink.py
@@ -0,0 +1,59 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+'''Migration to Baserock Definitions format version 4.
+
+This change to the format was made to work around a bug in a deployment
+extension present in morph.git.
+
+Automated migration is not really possible for this change, and unless you
+are experiencing the install-files.configure extension crashing, you can ignore
+it completely.
+
+We have now moved all .configure and .write extensions into the definitions.git
+repository. Changes like this no longer require a marking a new version of the
+Baserock definitions format in order to prevent build tools crashing.
+
+Morph commit c373f5a403b0ec introduces version 4 of the definitions format. In
+older versions of Morph the install-files.configure extension would crash if it
+tried to overwrite a symlink. This bug is fixed in the version of Morph that
+can build definitions version 4.
+
+If you need to overwrite a symlink at deploytime using install-files.configure,
+please use VERSION to 4 or above in your definitions.git repo so older versions
+of Morph gracefully refuse to deploy, instead of crashing.
+
+'''
+
+
+import sys
+
+import migrations
+
+
+TO_VERSION = 4
+
+
+try:
+    if migrations.check_definitions_version(TO_VERSION - 1):
+        migrations.set_definitions_version(TO_VERSION)
+        sys.stdout.write("Migration completed successfully.\n")
+        sys.exit(0)
+    else:
+        sys.stdout.write("Nothing to do.\n")
+        sys.exit(0)
+except RuntimeError as e:
+    sys.stderr.write("Error: %s\n" % e.message)
+    sys.exit(1)
diff --git a/migrations/005-strip-commands.py b/migrations/005-strip-commands.py
new file mode 100755
index 0000000..da3de94
--- /dev/null
+++ b/migrations/005-strip-commands.py
@@ -0,0 +1,78 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Migration to Baserock Definitions format version 5.
+
+Version 5 of the definitions format adds a 'strip-commands' field that can
+be set in chunk definitions.
+
+Version 5 also allows deployment extensions to live in definitions.git instead
+of morph.git. This greatly reduces the interface surface of the Baserock
+definitions format specification, because we no longer have to mark a new
+version of the definitions format each time an extension in morph.git is added,
+removed, or changes its API in any way.
+
+In commit 6f4929946 of git://git.baserock.org/baserock/baserock/definitions.git
+the deployment extensions were moved into an extensions/ subdirectory, and the
+system and cluster .morph files that referred to them were all updated to
+prepend 'extension/' to the filenames. This migration doesn't (re)do that
+change.
+
+'''
+
+
+import sys
+import warnings
+
+import migrations
+
+
+TO_VERSION = 5
+
+
+def check_strip_commands(contents, filename):
+    assert contents['kind'] == 'chunk'
+
+    valid = True
+
+    if 'strip-commands' in contents:
+        warnings.warn(
+            "%s has strip-commands, which are not valid until version 5" %
+            filename)
+        valid = False
+
+    return valid
+
+
+try:
+    if migrations.check_definitions_version(TO_VERSION - 1):
+        safe_to_migrate = migrations.process_definitions(
+            kinds=['chunk'], validate_cb=check_strip_commands)
+
+        if not safe_to_migrate:
+            sys.stderr.write(
+                "Migration failed due to one or more warnings.\n")
+            sys.exit(1)
+        else:
+            migrations.set_definitions_version(TO_VERSION)
+            sys.stdout.write("Migration completed successfully.\n")
+            sys.exit(0)
+    else:
+        sys.stdout.write("Nothing to do.\n")
+        sys.exit(0)
+except RuntimeError as e:
+    sys.stderr.write("Error: %s\n" % e.message)
+    sys.exit(1)
diff --git a/migrations/006-specify-build-system.py b/migrations/006-specify-build-system.py
new file mode 100755
index 0000000..b66736c
--- /dev/null
+++ b/migrations/006-specify-build-system.py
@@ -0,0 +1,354 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+# THIS MIGRATION REQUIRES NETWORK ACCESS TO A BASEROCK GIT CACHE SERVER! If
+# you do not have your own Trove, or don't know what a Trove is, it should
+# work as-is, provided you have internet access that allows access to
+# http://git.baserock.org:8080/.
+#
+# If you do have your own Trove, change the value of TROVE_HOST below to
+# point to it.
+#
+# This migration uses the same autodetection mechanism that Morph and YBD use
+# at build time, in order to fill in the 'build-system' field in strata where
+# it is now needed.
+
+
+'''Migration to Baserock Definitions format version 6.
+
+In definitions version 6, build system autodetection no longer happens. This
+means that any chunk that wants to use one of the predefined build systems
+(those built into Morph) must say so explicitly, using the 'build-system'
+field.
+
+The build-system field for a chunk can now be specified within a stratum that
+contains it. Previously you needed to add a .morph file for a chunk in order to
+specify its build-system, but we want to avoid needing a .morph file for
+components that follow standard patterns.
+
+Previously, if build-system wasn't given, Morph would scan the contents of the
+chunk's Git repo and try to autodetect which build system was used. This could
+be slow, could fail in confusing ways, and meant that to fully parse
+definitions you needed access to some or all of the repos they referenced.
+
+The chosen build-system affects which predefined command sequences are set for
+a chunk. It is valid to omit the field if a chunk has its own build commands
+defined in a .morph file. When listing the chunks included in a stratum, either
+'morph' or 'build-system' must be specified, but not both (to avoid the
+possibility of conflicting values).
+
+'''
+
+
+import requests
+import yaml
+
+import logging
+import os
+import sys
+import warnings
+
+import migrations
+
+
+TROVE_HOST = 'git.baserock.org'
+
+REPO_ALIASES = {
+    'baserock:': 'git://%s/baserock/' % TROVE_HOST,
+    'freedesktop:': 'git://anongit.freedesktop.org/',
+    'github:': 'git://github.com/',
+    'gnome:': 'git://git.gnome.org/',
+    'upstream:': 'git://%s/delta/' % TROVE_HOST,
+}
+
+GIT_CACHE_SERVER_URL = 'http://%s:8080/' % TROVE_HOST
+
+FAIL_ON_REMOTE_CACHE_ERRORS = False
+
+
+TO_VERSION = 6
+
+
+# From ybd.git file repos.py at commit eb3bf397ba729387f0d4145a8df8d3c1f9eb707f
+
+def get_repo_url(repo):
+    for alias, url in REPO_ALIASES.items():
+        repo = repo.replace(alias, url)
+    if repo.endswith('.git'):
+        repo = repo[:-4]
+    return repo
+
+
+# Based on morph.git file buildsystem.py at commit a7748f9cdaaf4112c30d7c1.
+#
+# I have copied and pasted this code here, as it should not be needed anywhere
+# once everyone has migrated to definitions version 6.
+
+class BuildSystem(object):
+    def used_by_project(self, file_list):
+        '''Does a project use this build system?
+
+        ``exists`` is a function that returns a boolean telling if a
+        filename, relative to the project source directory, exists or not.
+
+        '''
+        raise NotImplementedError()  # pragma: no cover
+
+
+class ManualBuildSystem(BuildSystem):
+
+    '''A manual build system where the morphology must specify all commands.'''
+
+    name = 'manual'
+
+    def used_by_project(self, file_list):
+        return False
+
+
+class DummyBuildSystem(BuildSystem):
+
+    '''A dummy build system, useful for debugging morphologies.'''
+
+    name = 'dummy'
+
+    def used_by_project(self, file_list):
+        return False
+
+
+class AutotoolsBuildSystem(BuildSystem):
+
+    '''The automake/autoconf/libtool holy trinity.'''
+
+    name = 'autotools'
+
+    def used_by_project(self, file_list):
+        indicators = [
+            'autogen',
+            'autogen.sh',
+            'configure',
+            'configure.ac',
+            'configure.in',
+            'configure.in.in',
+        ]
+
+        return any(x in file_list for x in indicators)
+
+
+class PythonDistutilsBuildSystem(BuildSystem):
+
+    '''The Python distutils build systems.'''
+
+    name = 'python-distutils'
+
+    def used_by_project(self, file_list):
+        indicators = [
+            'setup.py',
+        ]
+
+        return any(x in file_list for x in indicators)
+
+
+class CPANBuildSystem(BuildSystem):
+
+    '''The Perl cpan build system.'''
+
+    name = 'cpan'
+
+    def used_by_project(self, file_list):
+        indicators = [
+            'Makefile.PL',
+        ]
+
+        return any(x in file_list for x in indicators)
+
+
+class CMakeBuildSystem(BuildSystem):
+
+    '''The cmake build system.'''
+
+    name = 'cmake'
+
+    def used_by_project(self, file_list):
+        indicators = [
+            'CMakeLists.txt',
+        ]
+
+        return any(x in file_list for x in indicators)
+
+
+class QMakeBuildSystem(BuildSystem):
+
+    '''The Qt build system.'''
+
+    name = 'qmake'
+
+    def used_by_project(self, file_list):
+        indicator = '.pro'
+
+        for x in file_list:
+            if x.endswith(indicator):
+                return True
+
+        return False
+
+
+build_systems = [
+    ManualBuildSystem(),
+    AutotoolsBuildSystem(),
+    PythonDistutilsBuildSystem(),
+    CPANBuildSystem(),
+    CMakeBuildSystem(),
+    QMakeBuildSystem(),
+    DummyBuildSystem(),
+]
+
+
+def detect_build_system(file_list):
+    '''Automatically detect the build system, if possible.
+
+    If the build system cannot be detected automatically, return None.
+    For ``exists`` see the ``BuildSystem.exists`` method.
+
+    '''
+    for bs in build_systems:
+        if bs.used_by_project(file_list):
+            return bs
+    return None
+
+
+## End of code based on morph.git file buildsystem.py.
+
+def get_toplevel_file_list_from_repo(url, ref):
+    '''Try to list the set of files in the root directory of the repo at 'url'.
+
+    '''
+    try:
+        response = requests.get(
+            GIT_CACHE_SERVER_URL + '1.0/trees',
+            params={'repo': url, 'ref': ref},
+            headers={'Accept': 'application/json'},
+            timeout=9)
+        logging.debug("Got response: %s" % response)
+        try:
+            response.raise_for_status()
+            toplevel_tree = response.json()['tree']
+        except Exception as e:
+            raise RuntimeError(
+                "Unexpected response from server %s for repo %s: %s" %
+                (GIT_CACHE_SERVER_URL, url, e.message))
+        toplevel_filenames = toplevel_tree.keys()
+    except requests.exceptions.ConnectionError as e:
+        raise RuntimeError("Unable to connect to cache server %s while trying "
+                           "to query file list of repo %s. Error was: %s" %
+                           (GIT_CACHE_SERVER_URL, url, e.message))
+    return toplevel_filenames
+
+
+def validate_chunk_refs(contents, filename):
+    assert contents['kind'] == 'stratum'
+
+    valid = True
+    for chunk_ref in contents.get('chunks', []):
+        if chunk_ref.get('morph') is None:
+            # No chunk .morph file -- this stratum was relying on build-system
+            # autodetection here.
+
+            if 'repo' not in chunk_ref:
+                warnings.warn("%s: Chunk %s doesn't specify a source repo." %
+                              (filename, chunk_ref.get('name')))
+                valid = False
+
+            if 'ref' not in chunk_ref:
+                warnings.warn("%s: Chunk %s doesn't specify a source ref." %
+                              (filename, chunk_ref.get('name')))
+                valid = False
+    return valid
+
+
+def move_dict_entry_last(dict_object, key, error_if_missing=False):
+    '''Move an entry in a ordered dict to the end.'''
+
+    # This is a hack, I couldn't find a method on the 'CommentedMap' type dict
+    # that we receive from ruamel.yaml that would allow doing this neatly.
+    if key in dict_object:
+        value = dict_object[key]
+        del dict_object[key]
+        dict_object[key] = value
+    else:
+        if error_if_missing:
+            raise KeyError(key)
+
+
+def ensure_buildsystem_defined_where_needed(contents, filename):
+    assert contents['kind'] == 'stratum'
+
+    changed = False
+    for chunk_ref in contents.get('chunks', []):
+        if chunk_ref.get('morph') is None:
+            # No chunk .morph file -- this stratum was relying on build-system
+            # autodetection here.
+
+            chunk_git_url = get_repo_url(chunk_ref['repo'])
+            chunk_git_ref = chunk_ref['ref']
+
+            try:
+                toplevel_file_list = get_toplevel_file_list_from_repo(
+                    chunk_git_url, chunk_git_ref)
+            except Exception as e:
+                warnings.warn(str(e))
+                message = (
+                    "Unable to look up one or more repos on remote Git "
+                    "server %s. If you are using a Trove that is not %s, "
+                    "please edit the TROVE_HOST constant in this script "
+                    "and run it again." % (TROVE_HOST, TROVE_HOST))
+                if FAIL_ON_REMOTE_CACHE_ERRORS:
+                    raise RuntimeError(message)
+                else:
+                    warnings.warn(message)
+                    continue
+
+            logging.debug(
+                '%s: got file list %s', chunk_git_url, toplevel_file_list)
+            build_system = detect_build_system(toplevel_file_list)
+
+            chunk_ref['build-system'] = build_system.name
+            move_dict_entry_last(chunk_ref, 'build-depends')
+
+            changed = True
+
+    return changed
+
+
+try:
+    if migrations.check_definitions_version(TO_VERSION - 1):
+        success = migrations.process_definitions(
+            kinds=['stratum'],
+            validate_cb=validate_chunk_refs,
+            modify_cb=ensure_buildsystem_defined_where_needed)
+        if not success:
+            sys.stderr.write(
+                "Migration failed due to one or more warnings.\n")
+            sys.exit(1)
+        else:
+            migrations.set_definitions_version(TO_VERSION)
+            sys.stderr.write("Migration completed successfully.\n")
+            sys.exit(0)
+    else:
+        sys.stderr.write("Nothing to do.\n")
+        sys.exit(0)
+except RuntimeError as e:
+    sys.stderr.write("Error: %s\n" % e.message)
+    sys.exit(1)
diff --git a/migrations/007-defaults-in-definitions.py b/migrations/007-defaults-in-definitions.py
new file mode 100755
index 0000000..489baf9
--- /dev/null
+++ b/migrations/007-defaults-in-definitions.py
@@ -0,0 +1,67 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Migration to Baserock Definitions format version 7.
+
+Definitions version 7 adds a file named DEFAULTS which sets the default
+build commands and default split rules for the set of definitions in that
+repo.
+
+'''
+
+
+import os
+import shutil
+import sys
+import warnings
+
+import migrations
+
+
+TO_VERSION = 7
+
+
+
+try:
+    if migrations.check_definitions_version(TO_VERSION - 1):
+        if os.path.exists('DEFAULTS'):
+            warnings.warn(
+                "DEFAULTS file already exists in these definitions.")
+            valid = False
+        else:
+            shutil.copy(
+                'migrations/007-initial-defaults',
+                'DEFAULTS')
+            valid = True
+
+        if valid:
+            migrations.set_definitions_version(TO_VERSION)
+            sys.stdout.write("Migration completed successfully.\n")
+            sys.exit(0)
+        else:
+            sys.stderr.write(
+                "Migration failed due to one or more warnings.\n")
+            sys.exit(1)
+    else:
+        if not os.path.exists('DEFAULTS'):
+            warnings.warn(
+                "These definitions are marked as version 7 but there is no "
+                "DEFAULTS file.")
+        sys.stdout.write("Nothing to do.\n")
+        sys.exit(0)
+except RuntimeError as e:
+    sys.stderr.write("Error: %s\n" % e.message)
+    sys.exit(1)
diff --git a/migrations/007-initial-defaults b/migrations/007-initial-defaults
new file mode 100644
index 0000000..ab034a0
--- /dev/null
+++ b/migrations/007-initial-defaults
@@ -0,0 +1,199 @@
+# Baserock definitions defaults
+# =============================
+#
+# The DEFAULTS file is treated specially by Baserock build tools.
+#
+# For more information, see: <http://wiki.baserock.org/definitions/current>.
+
+
+# Predefined build commands
+# -------------------------
+#
+# Common patterns in build instructions can be defined here, which can save
+# users from having to write lots of similar-looking chunk .morph files.
+#
+# There are pre- and post- variants for each set of commands. These exist so
+# you can add more commands without having to copy the defaults. For example,
+# to create an extra symlink after running `make install`, you can use
+# post-install-commands. Since these exist as a way of extending the defaults,
+# you cannot set default values for the pre- and post- commands.
+#
+# The set of environment variables available when these commands are executed
+# is not formally specified right now, but you can assume PREFIX, DESTDIR and
+# MORPH_ARCH are all set.
+#
+build-systems:
+  manual:
+    # The special, default 'no-op' build system.
+    configure-commands: []
+    build-commands: []
+    install-commands: []
+    strip-commands: []
+
+  autotools:
+    # GNU Autoconf and GNU Automake, or anything which follow the same pattern.
+    #
+    # See also: https://github.com/cgwalters/build-api/blob/master/build-api.md
+    configure-commands:
+    - >-
+      export NOCONFIGURE=1;
+      if [ -e autogen ]; then ./autogen;
+      elif [ -e autogen.sh ]; then ./autogen.sh;
+      elif [ -e bootstrap ]; then ./bootstrap;
+      elif [ -e bootstrap.sh ]; then ./bootstrap.sh;
+      elif [ ! -e ./configure ]; then autoreconf -ivf;
+      fi
+    - ./configure --prefix="$PREFIX"
+    build-commands:
+    - make
+    install-commands:
+    - make DESTDIR="$DESTDIR" install
+    strip-commands:
+      # TODO: Make idempotent when files are hardlinks
+      # Strip all ELF binary files that are executable or named like a library.
+      # .so files for C, .cmxs for OCaml and .node for Node.
+      #
+      # The file name and permissions checks are done with the `find` command before
+      # the ELF header is checked with the shell command, because it is a lot cheaper
+      # to check the mode and file name first, because it is a metadata check, rather
+      # than a subprocess and a file read.
+      #
+      # `file` is not used, to keep the dependency requirements down.
+      - &generic-strip-command |
+        find "$DESTDIR" -type f \
+          '(' -perm -111 -o -name '*.so*' -o -name '*.cmxs' -o -name '*.node' ')' \
+          -exec sh -ec \
+          'read -n4 hdr <"$1" # check for elf header
+           if [ "$hdr" != "$(printf \\x7fELF)" ]; then
+               exit 0
+           fi
+           debugfile="$DESTDIR$PREFIX/lib/debug/$(basename "$1")"
+           mkdir -p "$(dirname "$debugfile")"
+           objcopy --only-keep-debug "$1" "$debugfile"
+           chmod 644 "$debugfile"
+           strip --remove-section=.comment --remove-section=.note --strip-unneeded "$1"
+           objcopy --add-gnu-debuglink "$debugfile" "$1"' - {} ';'
+
+  python-distutils:
+    # The Python distutils build systems.
+    configure-commands: []
+    build-commands:
+    - python setup.py build
+    install-commands:
+    - python setup.py install --prefix "$PREFIX" --root "$DESTDIR"
+    strip-commands:
+      - *generic-strip-command
+
+  cpan:
+    # The Perl ExtUtil::MakeMaker build system. This is called the 'cpan' build
+    # system for historical reasons.
+    #
+    # To install perl distributions into the correct location in our chroot
+    # we need to set PREFIX to <destdir>/<prefix> in the configure-commands.
+    #
+    # The mapping between PREFIX and the final installation
+    # directories is complex and depends upon the configuration of perl
+    # see,
+    # https://metacpan.org/pod/distribution/perl/INSTALL#Installation-Directories
+    # and ExtUtil::MakeMaker's documentation for more details.
+    configure-commands:
+      - perl Makefile.PL PREFIX=$DESTDIR$PREFIX
+    build-commands:
+    - make
+    install-commands:
+    - make install
+    strip-commands:
+      - *generic-strip-command
+
+  module-build:
+    # The Module::Build build system
+    #
+    # See the comment in ExtUtilsMakeMakerBuildSystem to see why --prefix is
+    # set to $DESTDIR$PREFIX here (--prefix in Module::Build has the same
+    # meaning as PREFIX in ExtUtils::MakeMaker).
+    configure-commands:
+      - perl Build.PL --prefix "$DESTDIR$PREFIX"
+    build-commands:
+      - ./Build
+    install-commands:
+      - ./Build install
+    strip-commands:
+      - *generic-strip-command
+
+  cmake:
+    # The CMake build system.
+    configure-commands:
+    - cmake -DCMAKE_INSTALL_PREFIX="$PREFIX"
+    build-commands:
+    - make
+    install-commands:
+    - make DESTDIR="$DESTDIR" install
+    strip-commands:
+      - *generic-strip-command
+
+  qmake:
+    # The Qt build system.
+    configure-commands:
+    - qmake -makefile
+    build-commands:
+    - make
+    install-commands:
+    - make INSTALL_ROOT="$DESTDIR" install
+    strip-commands:
+      - *generic-strip-command
+
+
+# Predefined artifact splitting rules
+# -----------------------------------
+#
+# Once a build has completed, you have some files that have been installed into
+# $DESTDIR. The splitting rules control how many 'artifact' tarballs are
+# generated as a result of the build, and which files from $DESTDIR end up in
+# which 'artifact'.
+#
+# The default split rules are defined here. These can be overriden in
+# individual chunk .morph files and stratum .morph files using the 'products'
+# field.
+#
+split-rules:
+  chunk:
+    - artifact: -bins
+      include:
+        - (usr/)?s?bin/.*
+    - artifact: -libs
+      include:
+        - (usr/)?lib(32|64)?/lib[^/]*\.so(\.\d+)*
+        - (usr/)libexec/.*
+    - artifact: -devel
+      include:
+        - (usr/)?include/.*
+        - (usr/)?lib(32|64)?/lib.*\.a
+        - (usr/)?lib(32|64)?/lib.*\.la
+        - (usr/)?(lib(32|64)?|share)/pkgconfig/.*\.pc
+    - artifact: -doc
+      include:
+        - (usr/)?share/doc/.*
+        - (usr/)?share/man/.*
+        - (usr/)?share/info/.*
+    - artifact: -locale
+      include:
+        - (usr/)?share/locale/.*
+        - (usr/)?share/i18n/.*
+        - (usr/)?share/zoneinfo/.*
+    - artifact: -misc
+      include:
+        - .*
+
+  stratum:
+    - artifact: -devel
+      include:
+        - .*-devel
+        - .*-debug
+        - .*-doc
+    - artifact: -runtime
+      include:
+        - .*-bins
+        - .*-libs
+        - .*-locale
+        - .*-misc
+        - .*
diff --git a/migrations/GUIDELINES b/migrations/GUIDELINES
new file mode 100644
index 0000000..3694e2c
--- /dev/null
+++ b/migrations/GUIDELINES
@@ -0,0 +1,35 @@
+Guidelines for writing migrations
+---------------------------------
+
+All changes to the definitions format must have a migration, but it is valid
+for the migration to do nothing except update the version number (see
+004-install-files-overwrite-symlink.py for an example of that).
+
+This small set of rules exists to ensure that the migrations are consistent and
+easy to understand. If you are writing a migration and these rules don't make
+any sense, we should probably change them. Please sign up to the
+baserock-dev@baserock.org mailing list to suggest the change.
+
+- Write migrations in Python. They must be valid Python 3. For now, since
+  only Python 2 is available in Baserock 'build' and 'devel' reference systems
+  up to the 15.25 release of Baserock, they must also be valid Python 2.
+
+- Don't use any external libraries.
+
+- Follow the existing file naming pattern, and the existing code convention.
+
+- Keep the migration code as simple as possible.
+
+- Avoid crashing on malformed input data, where practical. For example, use
+  contents.get('field') instead of contents['field'] to avoid crashing when
+  'field' is not present. The idea of this is to avoid a "cascade of errors"
+  problem when running the migrations on bad inputs. It is confusing when
+  migrations break on problems that are unrelated to the actual area where
+  they operate, even if they are theoretically "within their rights" to do so.
+
+- Migrate the definitions in line with current best practices. For example,
+  migrations/001-empty-build-depends.py doesn't need to remove empty
+  build-depends fields: they are still valid in version 1 of the format. But
+  best practice is now to remove them. Users who don't agree with this practice
+  can choose to not run that migration, which can be done with `chmod -x
+  migrations/xxx.py`.
diff --git a/migrations/indent b/migrations/indent
new file mode 100755
index 0000000..8d6f034
--- /dev/null
+++ b/migrations/indent
@@ -0,0 +1,36 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Automatically reformat a set of Baserock definition files.
+
+This tool expects to be able to use ruamel.yaml to load and write YAML.
+It will totally ruin things if used with PyYAML.
+
+It makes sense to run this script on your definitions, and check through
+and commit the result, before running any of the automated migrations. This
+way, you can be sure that the migrations will only change things that they need 
+to in the .morph files.
+
+'''
+
+
+import migrations
+
+
+def force_rewrite(contents, filename):
+    return True
+
+migrations.process_definitions(path='.', modify_cb=force_rewrite)
diff --git a/migrations/migrations.py b/migrations/migrations.py
new file mode 100644
index 0000000..22ed132
--- /dev/null
+++ b/migrations/migrations.py
@@ -0,0 +1,228 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Tools for migrating Baserock definitions from one format version to another.
+
+'''
+
+
+# ruamel.yaml is a fork of PyYAML which allows rewriting YAML files without
+# destroying all of the comments, ordering and formatting. The more
+# widely-used PyYAML library will produce output totally different to the
+# input file in most cases.
+#
+# See: <https://bitbucket.org/ruamel/yaml>
+import ruamel.yaml as yaml
+
+import logging
+import os
+import warnings
+
+
+# Uncomment this to cause all log messages to be written to stdout. By
+# default they are hidden, but if you are debugging something this might help!
+#
+# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
+
+
+def pretty_warnings(message, category, filename, lineno,
+                    file=None, line=None):
+    '''Format warning messages from warnings.warn().'''
+    return 'WARNING: %s\n' % (message)
+
+# Override the default warning formatter (which is ugly), and add a filter to
+# ensure duplicate warnings only get displayed once.
+warnings.simplefilter("once", append=True)
+warnings.formatwarning = pretty_warnings
+
+
+
+def parse_yaml_with_roundtrip_info(text):
+    return yaml.load(text, yaml.RoundTripLoader)
+
+def write_yaml_with_roundtrip_info(contents, stream, **kwargs):
+    yaml.dump(contents, stream, Dumper=yaml.RoundTripDumper, **kwargs)
+
+
+
+class VersionFileError(RuntimeError):
+    '''Represents errors in the version marker file (./VERSION).'''
+    pass
+
+
+class MigrationOutOfOrderError(RuntimeError):
+    '''Raised if a migration is run on too old a version of definitions.
+
+    It's not an error to run a migration on a version that is already migrated.
+
+    '''
+    pass
+
+
+def check_definitions_version(from_version, version_file='./VERSION',
+                              to_version=None):
+    '''Check if migration between 'from_version' and 'to_version' is needed.
+
+    Both 'from_version' and 'to_version' should be whole numbers. The
+    'to_version' defaults to from_version + 1.
+
+    This function reads the version marker file specified by 'version_file'.
+    Returns True if the version is between 'from_version' and 'to_version',
+    indicating that migration needs to be done. Returns False if the version is
+    already at or beyond 'to_version'. Raises MigrationOutOfOrderError if the
+    version is below 'from_version'.
+
+    If 'version_file' is missing or invalid, it raises VersionFileError. The
+    version file is expected to follow the following format:
+
+        version: 1
+
+    '''
+    to_version = to_version or (from_version + 1)
+    need_to_migrate = False
+
+    if os.path.exists(version_file):
+        logging.info("Found version information file: %s" % version_file)
+
+        with open(version_file) as f:
+            version_text = f.read()
+
+        if len(version_text) == 0:
+            raise VersionFileError(
+                "File %s exists but is empty." % version_file)
+
+        try:
+            version_info = yaml.safe_load(version_text)
+            current_version = version_info['version']
+
+            if current_version >= to_version:
+                logging.info(
+                    "Already at version %i." % current_version)
+            elif current_version < from_version:
+                raise MigrationOutOfOrderError(
+                    "This tool expects to migrate from version %i to version "
+                    "%i of the Baserock Definitions syntax. These definitions "
+                    "claim to be version %i." % (
+                        from_version, to_version, current_version))
+            else:
+                logging.info("Need to migrate from %i to %i.",
+                             current_version, to_version)
+                need_to_migrate = True
+        except (KeyError, TypeError, ValueError) as e:
+            logging.exception(e)
+            raise VersionFileError(
+                "Invalid version info: '%s'" % version_text)
+    else:
+        raise VersionFileError(
+            "No file %s was found. Please run the migration scripts in order,"
+            "starting from 000-version-info.py." % version_file)
+
+    return need_to_migrate
+
+
+def set_definitions_version(new_version, version_file='./VERSION'):
+    '''Update the version information stored in 'version_file'.
+
+    The new version must be a whole number. If 'version_file' doesn't exist,
+    it will be created.
+
+    '''
+    version_info = {'version': new_version}
+    with open(version_file, 'w') as f:
+        # If 'default_flow_style' is True (the default) then the output here
+        # will look like "{version: 0}" instead of "version: 0".
+        yaml.safe_dump(version_info, f, default_flow_style=False)
+
+
+def walk_definition_files(path='.', extensions=['.morph']):
+    '''Recursively yield all files under 'path' with the given extension(s).
+
+    This is safe to run in the top level of a Git repository, as anything under
+    '.git' will be ignored.
+
+    '''
+    for dirname, dirnames, filenames in os.walk('.'):
+        filenames.sort()
+        dirnames.sort()
+        if '.git' in dirnames:
+            dirnames.remove('.git')
+        for filename in filenames:
+            for extension in extensions:
+                if filename.endswith(extension):
+                    yield os.path.join(dirname, filename)
+
+
+ALL_KINDS = ['cluster', 'system', 'stratum', 'chunk']
+
+
+def process_definitions(path='.', kinds=ALL_KINDS, validate_cb=None,
+                        modify_cb=None):
+    '''Run callbacks for all Baserock definitions found in 'path'.
+
+    If 'validate_cb' is set, it will be called for each definition and can
+    return True or False to indicate whether that definition is valid according
+    a new version of the format. The process_definitions() function will return
+    True if all definitions were valid according to validate_cb(), and False
+    otherwise.
+
+    If 'modify_cb' is set, it will be called for each definition and can
+    modify the 'content' dict. It should return True if the dict was modified,
+    and in this case the definition file will be overwritten with the new
+    contents. The 'ruamel.yaml' library is used if it is available, which will
+    try to preserve comments, ordering and some formatting in the YAML
+    definition files.
+
+    If 'validate_cb' is set and returns False for a definition, 'modify_cb'
+    will not be called.
+
+    Both callbacks are passed two parameters: a dict containing the contents of
+    the definition file, and its filename. The filename is passed so you can
+    use it when reporting errors.
+
+    The 'kinds' setting can be used to ignore some definitions according to the
+    'kind' field.
+
+    '''
+    all_valid = True
+
+    for filename in walk_definition_files(path=path):
+        with open(filename) as f:
+            text = f.read()
+
+        if modify_cb is None:
+            contents = yaml.load(text)
+        else:
+            contents = parse_yaml_with_roundtrip_info(text)
+
+        if 'kind' in contents:
+            if contents['kind'] in kinds:
+                valid = True
+                changed = False
+
+                if validate_cb is not None:
+                    valid = validate_cb(contents, filename)
+                    all_valid &= valid
+
+                if valid and modify_cb is not None:
+                    changed = modify_cb(contents, filename)
+
+                if changed:
+                    with open(filename, 'w') as f:
+                        write_yaml_with_roundtrip_info(contents, f, width=80)
+        else:
+            warnings.warn("%s is invalid: no 'kind' field set." % filename)
+
+    return all_valid
diff --git a/migrations/run-all b/migrations/run-all
new file mode 100755
index 0000000..5a817ee
--- /dev/null
+++ b/migrations/run-all
@@ -0,0 +1,73 @@
+#!/usr/bin/env python
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+'''Run a set of migration scripts.
+
+This script does exactly what `PYTHONPATH=. run-parts --exit-on-error` would
+do. I avoided using 'run-parts' purely because the implementation in Fedora 22
+doesn't have an '--exit-on-error' option. The Busybox and Debian
+implementations do have that option.
+
+Please fix run-parts in https://git.fedorahosted.org/cgit/crontabs.git/tree/ 
+so we can simplify this script :-)
+
+'''
+
+
+import os
+import subprocess
+import sys
+
+
+if len(sys.argv) == 2:
+    migration_dir = sys.argv[1]
+elif len(sys.argv) == 1:
+    migration_dir = os.path.dirname(__file__)
+else:
+    sys.stderr.write("Usage: %s [MIGRATION_DIR]\n" % sys.argv[0])
+    sys.exit(1)
+
+
+def is_executable(fpath):
+    return os.path.isfile(fpath) and os.access(fpath, os.X_OK)
+
+env = os.environ
+if 'PYTHONPATH' in env:
+    env['PYTHONPATH'] = env['PYTHONPATH'] + ':' + migration_dir
+else:
+    env['PYTHONPATH'] = migration_dir
+
+try:
+    migrations_found = 0
+    for fname in sorted(os.listdir(migration_dir)):
+        migration_fpath = os.path.join(migration_dir, fname)
+        if is_executable(migration_fpath):
+            if not os.path.samefile(migration_fpath, __file__) and \
+                    fname != 'indent':
+                migrations_found += 1
+                sys.stdout.write(migration_fpath + ":\n")
+                subprocess.check_call(
+                    migration_fpath, env=env)
+
+    if migrations_found == 0:
+        sys.stderr.write("No migration files found in '%s'\n" % migration_dir)
+        sys.exit(1)
+    else:
+        sys.exit(0)
+
+except (subprocess.CalledProcessError, RuntimeError) as e:
+    sys.stderr.write(str(e) + '\n')
+    sys.exit(1)
diff --git a/schemas/007/baserock.owl b/schemas/007/baserock.owl
new file mode 100644
index 0000000..33113d8
--- /dev/null
+++ b/schemas/007/baserock.owl
@@ -0,0 +1,295 @@
+# Copyright (C) 2015  Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+# This schema describes the Baserock data model. It describes how the data
+# can be represented in computer memory, it doesn't specify anything about
+# how it should be stored on disk or transferred.
+#
+# Please see README.schemas, and http://wiki.baserock.org/definitions/current/
+# for more information.
+#
+# This is not a brilliant schema, in Linked Data terms, because it's using
+# Baserock-specific terminology and a rigid layout, instead of being a generic
+# vocabulary for describing how to assemble software. However, it maps closely
+# to the format that the existing Baserock reference system definitions use.
+# We hope to produce a simpler and more flexible 2.x version of this schema in
+# future.
+
+
+# Partial OWL and RDF Schema glossary:
+#
+# domain: the set of objects that a can have a property.
+# range: the set of possible values for a property.
+#
+# ObjectProperty:
+#   a property whose value must be another resource.
+# DatatypeProperty:
+#   a property whose value is one of the datatype defined by XML Schema ("xsd")
+# FunctionalProperty:
+#   a property which can only have one value per resource
+# InverseFunctionalProperty:
+#   a property which can only have one resource per value
+
+
+# Metadata
+
+@prefix : <http://baserock.org/definitions/example-schema#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+@prefix dc: <http://purl.org/dc/elements/1.1/> .
+
+: a owl:Ontology ;
+  dc:title "Schema for Baserock data model" ;
+  dc:creator "The Baserock Project" ;
+  # Versioned according to Semantic Versioning http://www.semver.org/
+  owl:versionInfo "0.9.0" .
+
+
+## Entities
+
+# Source definitions
+
+:Morphology a owl:Class ;
+   rdfs:comment "A definition file." .
+
+:Cluster a owl:Class ;
+   rdfs:subClassOf :Morphology ;
+   rdfs:comment "Instructions for deploying one or systems." ;
+   owl:disjointWith :System, :Stratum, :Chunk .
+
+:System a owl:Class ;
+   rdfs:subClassOf :Morphology ;
+   rdfs:comment "A collection of strata in a form that can be executed." ;
+   owl:disjointWith :Cluster, :Stratum, :Chunk .
+
+:Stratum a owl:Class ;
+   rdfs:subClassOf :Morphology ;
+   rdfs:comment "A set of related chunks." ;
+   owl:disjointWith :Cluster, :System, :Chunk .
+
+:Chunk a owl:Class ;
+   rdfs:subClassOf :Morphology ;
+   rdfs:comment "An individual component, that can be built from a Git repository." ;
+   owl:disjointWith :Cluster, :System, :Stratum .
+
+# Container contents
+
+:SystemDeployment a owl:Class ;
+   rdfs:comment "Information for deploying one system of a cluster." .
+
+:ChunkReference a owl:Class ;
+   rdfs:comment "An instance of a chunk within a stratum." .
+
+# Products (for artifact split rules).
+
+:Artifact a owl:Class ;
+   rdfs:comment "One part of the result of building a morphology. The output of a build is split according to patterns, so there are multiple artifacts produced for a given source." .
+
+:StratumArtifact a owl:Class ;
+   rdfs:subClassOf :Artifact ;
+   rdfs:comment "Part of the result of building a stratum." ;
+   owl:disjointWith :ChunkArtifact.
+
+:ChunkArtifact a owl:Class ;
+   rdfs:subClassOf :Artifact ;
+   rdfs:comment "Part of the result of building a chunk." ;
+   owl:disjointWith :StratumArtifact.
+
+
+## Shared properties
+
+# 'name' becomes part of the URL when we import definition .morph files as RDF,
+# so it doesn't need a property.
+
+# Use dc:description for 'description', see:
+# http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#terms-description
+
+
+## Cluster-specific properties
+
+:containsSystemDeployment a owl:ObjectProperty ;
+  rdfs:domain :Cluster ;
+  rdfs:range :SystemDeployment ;
+  owl:minCardinality 1 .
+
+
+## SystemDeployment-specific properties
+
+:deploysSystem a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :SystemDeployment ;
+  rdfs:range :System .
+
+:hasLabel a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :SystemDeployment ;
+  rdfs:range xsd:string .
+
+:hasType a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :SystemDeployment ;
+  rdfs:range xsd:string ;
+  rdfs:comment "The .write extension used to perform the deployment." .
+
+:hasLocation a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :SystemDeployment ;
+  rdfs:range xsd:string ;
+  rdfs:comment "Deployment location. This is passed to the selected .write extension" .
+
+# FIXME: this is actually a key-value mapping, how do you do that in RDF properly?
+:hasConfigurationSetting a owl:DatatypeProperty ;
+  rdfs:domain :SystemDeployment ;
+  rdfs:range xsd:string ;
+  rdfs:comment "Arbitrary key=value pair that is set in the environment of the .configure and .write extensions during deployment." .
+
+
+## System-specific properties
+
+:containsStratumArtifact a owl:ObjectProperty ;
+  rdfs:domain :System ;
+  rdfs:range :StratumArtifact ;
+  owl:minCardinality 1 .
+
+# A build tool will only understand a certain set of architectures, but I think
+# that should not be part of the data model.
+:hasArchitecture a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :System ;
+  rdfs:range xsd:string .
+
+:hasConfigurationExtension a owl:DatatypeProperty ;
+  rdfs:domain :System ;
+  rdfs:range xsd:string ;
+  rdfs:comment "A .configure extension to be run when with system is deployed." .
+
+
+## Stratum-specific properties
+
+:producesStratumArtifact a owl:ObjectProperty , rdf:Seq ;
+  rdfs:domain :Stratum ;
+  rdfs:range :StratumArtifact ;
+  rdfs:comment "iDescribes how this artifact is split once all chunks are built." .
+
+:hasStratumBuildDependency a owl:ObjectProperty ;
+  rdfs:domain :Stratum ;
+  rdfs:range :Stratum .
+
+:containsChunkReference a owl:ObjectProperty ;
+  rdfs:domain :Stratum ;
+  rdfs:range :ChunkReference ;
+  owl:minCardinality 1 .
+
+
+## ChunkReference-specific properties
+
+:refersToChunk a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :ChunkReference ;
+  rdfs:range :Chunk .
+
+:repo a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :ChunkReference ;
+  rdfs:range xsd:string ;
+  rdfs:comment "Git repo that this chunk's build instructions should be run in. Can be a keyed URL according to the repo-alias mapping, which isn't currently defined in this data model." .
+
+:ref a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :ChunkReference ;
+  rdfs:range xsd:string ;
+  rdfs:comment "Ref of the Git repo that should be checked out before building. Usually a SHA1." .
+
+:unpetrifyRef a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :ChunkReference ;
+  rdfs:range xsd:string ;
+  rdfs:comment "Name of the ref that should be checked out (useful when 'ref' is a SHA1 to keep track of the actual branch/tag name)." .
+
+:hasChunkBuildDependency a owl:ObjectProperty ;
+  rdfs:domain :ChunkReference ;
+  rdfs:range :Chunk .
+
+:prefix a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :ChunkReference ;
+  rdfs:range xsd:string ;
+  rdfs:comment "Value of the PREFIX environment variable in the build environment (defaults to /usr)" .
+
+# FIXME: it's possible to specify in OWL that 'bootstrap' and 'normal' are the
+# only valid values.
+:buildMode a owl:DatatypeProperty , owl:FunctionalProperty ;
+  rdfs:domain :ChunkReference ;
+  rdfs:range xsd:string .
+
+
+## Chunk-specific properties
+
+# build-system isn't included here, as it overlaps with the *-commands
+# fields in an unfortunate way.
+
+# Note that representing lists of things in RDF is a bit of a pain.
+# http://smiy.sourceforge.net/olo/spec/orderedlistontology.html might
+# be worth investigating.
+
+:producesChunkArtifact a owl:ObjectProperty , rdf:Seq ;
+  rdfs:domain :Chunk ;
+  rdfs:range :ChunkArtifact ;
+  rdfs:comment "Describes how this artifact is split once built." .
+
+:maxJobs a owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range xsd:integer ;
+  rdfs:comment "Number of parallel jobs that can be run for this chunk. Only useful if set to 1, to mark components which have Makefiles that do not work with `make -j N`.".
+
+:CommandSequence a owl:Class ;
+  rdfs:subClassOf rdf:Seq ;
+  rdfs:comment "An ordered sequence of commands." .
+
+:preConfigureCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+:configureCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+:postConfigureCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+:preBuildCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+:buildCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+:postBuildCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+:preInstallCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+:installCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+:postInstallCommands a owl:ObjectProperty , owl:FunctionalProperty ;
+  rdfs:domain :Chunk ;
+  rdfs:range :CommandSequence .
+
+
+# Artifact properties
+
+:includes a owl:DatatypeProperty , rdf:Seq ;
+  rdfs:domain :Artifact ;
+  rdfs:range xsd:string ;
+  rdfs:comment "A regular expression pattern. The results of a build are matched against this pattern and any that match will be included in that artifact." .
diff --git a/schemas/007/chunk.json-schema b/schemas/007/chunk.json-schema
new file mode 100644
index 0000000..8de2fb5
--- /dev/null
+++ b/schemas/007/chunk.json-schema
@@ -0,0 +1,116 @@
+$schema: http://json-schema.org/draft-04/schema#
+id: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas/chunk.json-schema
+
+description: |
+  This is a JSON-Schema description of a 'chunk' .morph file, which is part of
+  the Baserock definitions YAML representation format.
+
+  This JSON-Schema file is valid for VERSION 7 of the Baserock definitions
+  YAML serialisation format.
+
+  The Baserock definitions format is the recommended way of representing
+  Baserock definitions on disk. The actual data model is described separately.
+  See https://wiki.baserock.org/definitions for more information.
+
+  This schema is represented as YAML, so that it can be edited more easily.
+  You may need to convert to JSON if using a JSON-Schema tool that expects
+  its input to be an actual string containing data serialised as JSON.
+
+definitions:
+  # Corresponds to CommandSequence in Baserock data model.
+  command-sequence:
+    type: array
+    items: { type: string }
+
+  # Corresponds to DeviceNode in Baserock data model.
+  device-node:
+    type: object
+    required: [type, filename, gid, uid, major, minor, permissions]
+    properties:
+      type: { type: string }
+      filename: { type: string }
+      gid: { type: integer }
+      uid: { type: integer }
+      major: { type: integer }
+      minor: { type: integer }
+      permissions: { type: string }
+
+  # Defines one or more instances of ChunkArtifact from the Baserock data model.
+  split-rules:
+     type: array
+     items:
+       type: object
+
+       required: [artifact, include]
+       additionalProperties: false
+
+       properties:
+         artifact: {type: string}
+         include:
+           type: array
+           items:
+             type: string
+             format: regex
+
+  system-integration-commands:
+    type: object
+    patternProperties:
+      # The property name here should correspond to the name of the chunk's
+      # artifacts, probably the '-bins' artifact if you are using the normal
+      # splitting rules.
+      ^.*$:
+        type: object
+        patternProperties:
+          ^.*$:
+            $ref: "#/definitions/command-sequence"
+
+# Corresponds to Chunk in Baserock data model.
+type: object
+
+additionalProperties: false
+required: [ 'name', 'kind']
+
+properties:
+  name: { type: string }
+  kind: { enum: [ "chunk" ] }
+
+  description: { type: string }
+
+  build-system: { type: string }
+  max-jobs: { type: integer }
+
+  products:
+    $ref: "#/definitions/split-rules"
+
+  pre-configure-commands:
+    $ref: "#/definitions/command-sequence"
+  configure-commands:
+    $ref: "#/definitions/command-sequence"
+  post-configure-commands:
+    $ref: "#/definitions/command-sequence"
+  pre-build-commands:
+    $ref: "#/definitions/command-sequence"
+  build-commands:
+    $ref: "#/definitions/command-sequence"
+  post-build-commands:
+    $ref: "#/definitions/command-sequence"
+  pre-install-commands:
+    $ref: "#/definitions/command-sequence"
+  install-commands:
+    $ref: "#/definitions/command-sequence"
+  post-install-commands:
+    $ref: "#/definitions/command-sequence"
+  pre-strip-commands:
+    $ref: "#/definitions/command-sequence"
+  strip-commands:
+    $ref: "#/definitions/command-sequence"
+  post-strip-commands:
+    $ref: "#/definitions/command-sequence"
+
+  system-integration:
+    $ref: "#/definitions/system-integration-commands"
+
+  devices:
+    type: array
+    items:
+      $ref: "#/definitions/device-node"
diff --git a/schemas/007/cluster.json-schema b/schemas/007/cluster.json-schema
new file mode 100644
index 0000000..1a628c7
--- /dev/null
+++ b/schemas/007/cluster.json-schema
@@ -0,0 +1,64 @@
+$schema: http://json-schema.org/draft-04/schema#
+id: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas/cluster.json-schema
+
+description:
+  This is a JSON-Schema description of a 'cluster' .morph file, which is part
+  of the Baserock definitions YAML representation format.
+
+  This JSON-Schema file is valid for VERSION 7 of the Baserock definitions
+  YAML serialisation format.
+
+  The Baserock definitions format is the recommended way of representing
+  Baserock definitions on disk. The actual data model is described separately.
+  See https://wiki.baserock.org/definitions for more information.
+
+  This schema is represented as YAML, so that it can be edited more easily.
+  You may need to convert to JSON if using a JSON-Schema tool that expects
+  its input to be an actual string containing data serialised as JSON.
+
+definitions:
+  # A reference to a System, plus one or more SystemDeployment for that system.
+  system-deployment-set:
+    type: object
+
+    required: [ 'morph', 'deploy' ]
+    additionalProperties: false
+
+    properties:
+      morph: { type: string }
+      deploy-defaults:
+        type: object
+        additionalProperties: true
+      deploy:
+        patternProperties:
+          ^.*$: { $ref: "#/definitions/system-deployment" }
+      subsystems:
+        type: array
+        items: { $ref: "#/definitions/system-deployment-set" }
+
+  # Corresponds to SystemDeployment in Baserock data model.
+  system-deployment:
+    type: object
+
+    required: [ 'type', 'location' ]
+    additionalProperties: true
+
+    properties:
+      type: { type: string }
+      location: { type: string }
+
+
+# Corresponds to Cluster in Baserock data model.
+type: object
+
+#required: [ 'name', 'kind', 'systems' ]
+additionalProperties: false
+
+properties:
+  name: { type: string }
+  kind: { enum: [ "cluster" ] }
+  description: { type: string }
+
+  systems:
+    type: array
+    items: { $ref: "#/definitions/system-deployment-set" }
diff --git a/schemas/007/defaults.json-schema b/schemas/007/defaults.json-schema
new file mode 100644
index 0000000..2f71342
--- /dev/null
+++ b/schemas/007/defaults.json-schema
@@ -0,0 +1,66 @@
+$schema: http://json-schema.org/draft-04/schema#
+id: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas/defaults.json-schema
+
+description: |
+  This is a JSON-Schema description of the DEFAULTS file specified in the
+  Baserock definitions format. DEFAULTS is a YAML file that contains global
+  defaults for a set of Baserock definitions.
+
+  This JSON-Schema file is valid for VERSION 7 of the Baserock definitions
+  YAML serialisation format.
+
+  The Baserock definitions YAML serialisation format is the recommended way of
+  representing Baserock definitions on disk. The actual data model is described
+  separately. See <https://wiki.baserock.org/definitions> for more information.
+
+  This schema is represented as YAML, so that it can be edited more easily.
+  You may need to convert to JSON if using a JSON-Schema tool that expects
+  its input to be an actual string containing data serialised as JSON.
+
+definitions:
+  command-sequence:
+    type: array
+    items: {type: string}
+
+  build-system:
+    type: object
+    additionalProperties: false
+    properties:
+      build-commands: {$ref: '#/definitions/command-sequence'}
+      configure-commands: {$ref: '#/definitions/command-sequence'}
+      install-commands: {$ref: '#/definitions/command-sequence'}
+      strip-commands: {$ref: '#/definitions/command-sequence'}
+
+  split-rules:
+     type: array
+     items:
+       type: object
+
+       required: [artifact, include]
+       additionalProperties: false
+
+       properties:
+         artifact: {type: string}
+         include:
+           type: array
+           items:
+             type: string
+             format: regex
+
+type: object
+additionalProperties: false
+
+properties:
+  # Predefined build systems.
+  build-systems:
+    type: object
+    patternProperties:
+      ^.*$: {$ref: '#/definitions/build-system'}
+
+  # Predefined artifact splitting rules.
+  split-rules:
+    type: object
+    additionalProperties: false
+    properties:
+      chunk: {$ref: '#/definitions/split-rules'}
+      stratum: {$ref: '#/definitions/split-rules'}
diff --git a/schemas/007/stratum.json-schema b/schemas/007/stratum.json-schema
new file mode 100644
index 0000000..0330f37
--- /dev/null
+++ b/schemas/007/stratum.json-schema
@@ -0,0 +1,99 @@
+$schema: http://json-schema.org/draft-04/schema#
+id: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas/chunk.json-schema
+
+description: |
+  This is a JSON-Schema description of a 'stratum' .morph file, which is part
+  of the Baserock definitions YAML representation format.
+
+  This JSON-Schema file is valid for VERSION 7 of the Baserock definitions
+  YAML serialisation format.
+
+  The Baserock definitions format is the recommended way of representing
+  Baserock definitions on disk. The actual data model is described separately.
+  See https://wiki.baserock.org/definitions for more information.
+
+  This schema is represented as YAML, so that it can be edited more easily.
+  You may need to convert to JSON if using a JSON-Schema tool that expects
+  its input to be an actual string containing data serialised as JSON.
+
+definitions:
+  # Defines one or more instances of StratumArtifact from the Baserock data
+  # model.
+  split-rules:
+     type: array
+     items:
+       type: object
+
+       required: [artifact, include]
+       additionalProperties: false
+
+       properties:
+         artifact: {type: string}
+         include:
+           type: array
+           items:
+             type: string
+             format: regex
+
+  # Corresponds to ChunkReference in Baserock data model.
+  chunk-reference:
+    type: object
+
+    required: [ 'repo', 'ref' ]
+    additionalProperties: false
+
+    properties:
+      name: { type: string }
+      morph: { type: string }
+      repo: { type: string }
+      ref: { type: string }
+      unpetrify-ref: { type: string }
+      build-mode: { type: string }
+      prefix: { type: string }
+      build-system: { type: string }
+      build-depends:
+        type: array
+        items: { type: string }
+      artifacts:
+        type: object
+        # This defines which chunk artifacts go in which stratum artifact.
+        # E.g. "glibc-nss: build-essential-runtime" specifies the glibc-nss
+        # chunk artifact should go in the build-essential-runtime stratum
+        # artifact.
+        patternProperties:
+          ^.*$: { type: string }
+
+  # This doesn't need any special treatment in the Baserock data model because
+  # it's a link to another stratum definition, without any extra info.
+  stratum-reference:
+    type: object
+
+    required: [ 'morph' ]
+    additionalProperties: false
+
+    properties:
+      morph: { type: string }
+
+
+# Corresponds to Stratum in Baserock data model.
+type: object
+
+required: [ 'name', 'kind', 'chunks' ]
+additionalProperties: false
+
+properties:
+  name: { type: string }
+  kind: { enum: [ "stratum" ] }
+  description: { type: string }
+
+  build-depends:
+    type: array
+    items:
+      $ref: "#/definitions/stratum-reference"
+
+  products:
+    $ref: "#/definitions/split-rules"
+
+  chunks:
+    type: array
+    items: { $ref: "#/definitions/chunk-reference" }
diff --git a/schemas/007/system.json-schema b/schemas/007/system.json-schema
new file mode 100644
index 0000000..a2fc0c5
--- /dev/null
+++ b/schemas/007/system.json-schema
@@ -0,0 +1,59 @@
+$schema: http://json-schema.org/draft-04/schema#
+id: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas/system.json-schema
+
+description: |
+  This is a JSON-Schema description of a 'system' .morph file, which is part of
+  the Baserock definitions YAML representation format.
+
+  This JSON-Schema file is valid for VERSION 7 of the Baserock definitions
+  YAML serialisation format.
+
+  The Baserock definitions format is the recommended way of representing
+  Baserock definitions on disk. The actual data model is described separately.
+  See https://wiki.baserock.org/definitions for more information.
+
+  This schema is represented as YAML, so that it can be edited more easily.
+  You may need to convert to JSON if using a JSON-Schema tool that expects
+  its input to be an actual string containing data serialised as JSON.
+
+definitions:
+  # In the Baserock data model, this becomes a list of links to StratumArtifact
+  # entities.
+  stratum-reference:
+    type: object
+
+    required: [ 'morph' ]
+    additionalProperties: false
+
+    properties:
+      name: { type: string }
+      morph: { type: string }
+      artifacts:
+        type: array
+        items: { type: string }
+
+
+# Corresponds to System in Baserock data model.
+system:
+type: object
+
+required: [ 'name', 'kind', 'strata' ]
+additionalProperties: false
+
+properties:
+  name: { type: string }
+  kind: { enum: [ "system" ] }
+  description: { type: string }
+
+  # Morph and YBD will only accept certain values for 'arch' based on
+  # hardcoded conditions in those tools.
+  arch: { type: string }
+
+  strata:
+    type: array
+    items:
+      $ref: "#/definitions/stratum-reference"
+
+  configuration-extensions:
+    type: array
+    items: { type: string }
diff --git a/schemas/README.schemas b/schemas/README.schemas
new file mode 100644
index 0000000..a778918
--- /dev/null
+++ b/schemas/README.schemas
@@ -0,0 +1,137 @@
+Schemas for the Baserock definitions format
+===========================================
+
+The starting point for learning about the Baserock definitions format is the
+wiki page at <http://wiki.baserock.org/definitions/>.
+
+The schemas/ directory in the Baserock reference definitions.git repository is
+the canonical home for some schemas which describe the format in a
+machine-readable way.
+
+There are two parts to 'Baserock definitions'. The 'Baserock data model' is an
+abstract vocabulary for describing how to build, integrate and deploy software
+components. The 'Baserock definitions YAML representation format' is a
+serialisation format for the data model, which lets you write YAML files
+describing how to build, integrate and deploy software components.
+
+If you want to make the YAML files easier to deal with, you only need
+to care about the JSON-Schema schemas and anything that parses the YAML files.
+
+If you want to write a new tool to build, visualise, analyse or otherwise
+process Baserock definitions in some way, you can ignore the syntax altogether,
+use a pre-existing parser, and just think in terms of the data
+model.
+
+If you want to change the data model, you still have quite a difficult job,
+but at least it should be simple to write a translation layer on top of an
+existing parser so that you can interpret all the existing Baserock reference
+system definitions in terms of your new data model.
+
+
+The Baserock definitions YAML representation format
+---------------------------------------------------
+
+YAML itself is a syntax for representating data as text. The YAML specification
+is at <http://www.yaml.org/>.
+
+The data needs to be structured in a certain way for it to make sense as
+Baserock build/integration/deployment instructions. We have used JSON-Schema
+to describe the required layout of the data.
+
+The JSON-Schema standard is described at <http://json-schema.org/>. The
+JSON-Schema language was designed for use with JSON, which is another syntax
+for representing data as text, which happens to be a subset of YAML. We have
+found so far that JSON-Schema works well with YAML, at least when using the
+Python 'jsonschema' module.
+
+Definitions are represented by files with a '.morph' extension. There are four
+different kinds: 'chunk', 'stratum', 'system', and 'cluster'. Each of these is
+described with a different .json-schema file. It is possible to merge all these
+into one file, and use the 'oneOf' field to say that any .morph file should
+match exactly one of the layouts. The only issue with this approach is that
+the Python 'jsonschema' model will give you totally useless errors if anything
+is invalid (along the lines of "<dump of entire file> is not valid under any of
+the given schemas"). So for now they are separate.
+
+
+Tools for working with the Baserock YAML schemas
+------------------------------------------------
+
+You can use `scripts/yaml-jsonschema` to validate .morph files against the
+schemas. For example:
+
+    scripts/yaml-jsonschema schemas/cluster.json-schema clusters/*.morph
+
+
+The Baserock data model
+-----------------------
+
+The best way to represent information on disk may be a pretty inefficient way
+to represent that data in a computer's memory. Likewise, the way a program
+stores data internally may be totally impractical for people to edit directly.
+
+The file `baserock.owl` is an initial effort to describe the Baserock data
+model independently of any syntax or representation.
+
+We use the W3C standard Web Ontology Language (OWL), combined with the much
+simpler RDF Schema language. Together, this allows defining the vocabulary we
+can use to define build, integration and deployment instructions. There are
+various ways to represent OWL 'ontologies'; `baserock.owl` uses a
+representation format named Turtle, which is designed to be convenient for
+hand-editing.
+
+The current data model is very closely tied to the current syntax, but we are
+looking to change this and make it much more generic. This will involve
+removing the current 'Chunk', 'Stratum', 'System' and 'Cluster' classes, and
+adding something like 'thing with build instructions' and 'thing that contains
+other things' instead. Name suggestions are welcome :-)
+
+It's useful to consider existing OWL and RDF Schema vocabularies that are
+related to the Baserock data model. In future we can link the Baserock
+reference system definitions with related data published elsewhere on the Web.
+Here is an incomplete list:
+
+  - Description of a Project (DOAP): https://github.com/edumbill/doap
+  - Software Ontology: https://robertdavidstevens.wordpress.com/2014/06/19/the-software-ontology-swo/
+  - Software Packet Data Exchange (SPDX): https://spdx.org/about-spdx/what-is-spdx
+
+
+Tools for working with the Baserock data model schema
+-----------------------------------------------------
+
+It's difficult to find to a short, relevant 'getting started' guide. The
+website http://www.linkeddata.org/ has a lot of background that should be
+useful.
+
+The `rapper` commandline tool, which comes as part of the 'raptor2' C library,
+is helpful for converting from one syntax to another, and checking if
+`baserock.owl` is valid Turtle syntax. The 'raptor2' homepage is
+<http://www.librdf.org/>.
+
+To check the syntax of `baserock.owl` using `rapper`:
+
+    rapper -i turtle schemas/baserock.owl
+
+
+Omissions / TODO items
+----------------------
+
+- Device nodes: chunk .morph files can list a set of device nodes. In
+  `chunk.json-schema` this is recognised, but in `baserock.owl` it is missing.
+
+- 'Lorry' mirroring instructions. These contain information on where 'upstream'
+  source code is kept, which should be considered part of the data model. A
+  JSON schema may be better off in lorry.git or
+  baserock/local-config/lorries.git.
+
+- Metadata in built systems. This is currently not standardised at all.
+
+
+Comments
+--------
+
+As far as I know, Baserock is the first project to treat build, integration and
+deployment instructions as data rather than code. If you have questions about
+the schemas, the definitions format, or the overall approach, and they aren't
+answered here or in <http://wiki.baserock.org/definitions/>, then please ask on
+the baserock-dev@baserock.org mailing list.
diff --git a/spec.mdwn b/spec.mdwn
new file mode 100644
index 0000000..03fde9e
--- /dev/null
+++ b/spec.mdwn
@@ -0,0 +1,518 @@
+[[!meta title="Baserock definitions format"]]
+
+The Baserock definitions format
+===============================
+
+This page describes the Baserock definitions format (morph files). It is intended to be useful as
+an *informal* specification. It is not guaranteed to be accurate or exhaustive.
+
+If you are just getting started with Baserock, the [[quick-start]], [[devel-with]] and [[guides]] pages provide a more practical introduction.
+
+The allowed YAML constructs are described in json-schema format here: <http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas>.
+
+The data model is described using OWL here: <http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas/baserock.owl>.
+
+The source code of [[Morph]] and [[YBD]] might be more useful if you need a completely accurate description of how the current Baserock definition format is used in practice.
+
+[[!toc startlevel=2 levels=2]]
+
+Versioning
+----------
+
+The current version of the definitions format is version 7.
+
+See also: the [[planned]] versions, and the [[historical]] versions of the
+format.
+
+Please propose changes to the format described here on the
+[[baserock-dev@baserock.org mailing list|mailinglist]]. Ideally, provide a
+patch for this file against the <git://baserock.branchable.com/> repo, but just
+describing the change you want to make is also fine.
+
+Definitions repository
+----------------------
+
+The design of Baserock aims to encourage users to keep *all* the information
+needed to build and deploy their software in one Git repository. This repo is
+often referred to as the 'definitions.git' repo, although nothing forces you
+to call it that.
+
+Some of this information will be Baserock 'definitions files', which describe
+how to build or deploy some software. Baserock tooling should expect that all
+the definitions it needs to process live in one Git repo. The definitions.git
+repo can and should contain any other files needed for build and deployment as
+well, such as configuration data and documentation.
+
+The Baserock Project maintains a set of 'reference system definitions' at
+[git://git.baserock.org/baserock/baserock/definitions] (which can also be
+referred to as [baserock:baserock/definitions], when using the repo aliasing
+feature of [[Morph]]). That repo contains systems that can be built and
+deployed as-is, but it is important that users can fork this repo as well,
+and work on systems in their version using `git merge` or `git rebase` to keep
+up to date with changes from upstream.
+
+Baserock tooling should not mandate anything about the definitions repo that
+the user wants to process, other than the rules defined below.
+
+
+[git://git.baserock.org/baserock/baserock/definitions]: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git
+[baserock:baserock/definitions]: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git
+
+### Structure
+
+Tooling can enforce that the definitions.git repo is actually a Git repo, but
+it can equally just treat it as a tree of files and directories.
+
+The top directory of the repo must contain a file named `VERSION`, that is
+valid [YAML] and contains a dict with key "version" and a value that is an
+integer.
+
+The integer specifies the version of the definitions format that this repo
+uses. A tool should refuse to process a version that it doesn't support, to
+avoid unpredictable errors. See the "Versioning" heading above for more detail
+on versions.
+
+To find all the Baserock definition files in the repo, tooling can recursively
+scan the contents of the repo for files matching the glob pattern "\*.morph".
+
+Deployment tooling should look in the toplevel directory /only/ for files
+matching the following globs. (The purpose of these files is described in the
+"deployment" section below).
+
+  - `\*.check`
+  - `\*.check.help`
+  - `\*.configure`
+  - `\*.configure.help`
+  - `\*.write`
+  - `\*.write.help`
+
+Definitions file syntax
+-----------------------
+
+[YAML] is used for all Baserock definitions files.
+
+The toplevel entity in a definition is a dict, in all cases. Any syntax errors
+or type errors (such the toplevel entity being a number, or something) should
+be reported to the user.
+
+The [[Morph]] tool raises an error if any unknown dictionary keys are found in
+the definition, mainly so that it reports any spelling errors in key names.
+
+### Common fields
+
+For all definitions, use the following fields:
+
+* `name`: the name of the definition; it must currently match the filename
+  (without the `.morph` suffix); **required**
+* `kind`: the kind of thing being built; **required**
+* `description`: a comment to describe what the definition is for; optional
+
+### Build definitions: Chunks, Systems and Strata
+
+Within this document, consider 'building' to be the act of running a series of
+commands in a given 'environment', where the commands and how to build the
+environment are completely specified by the definitions and the build tool.
+
+For chunks, use the following fields:
+
+* `build-system`: if the program is built using a build system known to
+  `morph`, you can set this field and avoid having to set the various
+  `*-commands` fields; the commands that the build system specifies can
+  be overridden; the following build-systems are known:
+
+  - `autotools`
+  - `python-distutils`
+  - `cpan`
+  - `cmake`
+  - `qmake`
+
+  optional
+
+* `pre-configure-commands`: a list of shell commands to run at
+  the configuration phase of a build, before the list in `configure-commands`;
+  optional
+* `configure-commands`: a list of shell commands to run at the configuraiton
+  phase of a build; optional
+* `post-configure-commands`: a list of shell commands to run at
+  the configuration phase of a build, after the list in `configure-commands`;
+  optional
+
+* `pre-build-commands`: a list of shell commands to run at
+  the build phase of a build, before the list in `build-commands`;
+  optional
+* `build-commands`: a list of shell commands to run to build (compile) the
+  project; optional
+* `post-build-commands`: a list of shell commands to run at
+  the build phase of a build, after the list in `build-commands`;
+  optional
+
+* `pre-test-commands`: a list of shell commands to run at
+  the test phase of a build, before the list in `test-commands`;
+  optional
+* `test-commands`: a list of shell commands to run unit tests and other
+  non-interactive tests on the built but un-installed project; optional
+* `post-test-commands`: a list of shell commands to run at
+  the test phase of a build, after the list in `test-commands`;
+  optional
+
+* `pre-install-commands`: a list of shell commands to run at
+  the install phase of a build, before the list in `install-commands`;
+  optional
+* `install-commands`: a list of shell commands to ; optional
+* `post-install-commands`: a list of shell commands to run at
+  the install phase of a build, after the list in `strip-commands`;
+  optional
+
+* `pre-strip-commands`: a list of shell commands to run at
+  the strip phase of a build, before the list in `strip-commands`;
+  optional
+* `strip-commands`: a list of shell commands to strip debug symbols from binaries;
+  this should strip binaries in the directory named in the `DESTDIR` environment
+  variable, not the actual system; optional
+* `post-strip-commands`: a list of shell commands to run at
+  the strip phase of a build, after the list in `install-commands`;
+  optional
+
+* `max-jobs`: a string to be given to `make` as the argument to the `-j`
+  option to specify the maximum number of parallel jobs; the only sensible
+  value is `"1"` (including the quotes), to prevent parallel jobs to run
+  at all; parallel jobs are only used during the `build-commands` phase,
+  since the other phases are often not safe when run in parallel; `morph`
+  picks a default value based on the number of CPUs on the host system;
+  optional
+
+* `chunks`: a key/value map of lists of regular expressions;
+  the key is the name
+  of a binary chunk, the regexps match the pathnames that will be
+  included in that chunk; the patterns match the pathnames that get installed
+  by `install-commands` (the whole path below `DESTDIR`); every file must
+  be matched by at least one pattern; by default, a single chunk gets
+  created, named according to the definition, and containing all files;
+  optional
+
+For strata, use the following fields:
+
+* `build-depends`: a list of strings, each of which refers to another
+  stratum that the current stratum depends on. This list may be omitted
+  or empty if the stratum does not depend on anything else.
+* `chunks`: a list of key/value mappings, where each mapping corresponds
+  to a chunk to be included in the stratum; the mappings may use the
+  following keys:
+    - `name` is the chunk's name (may be different from the
+      morphology name),
+    - `repo` is the repository in which to find (defaults to chunk name),
+    - `ref` identifies the commit to use (typically a branch name, but
+       any tree-ish git accepts is ok)
+    - `morph` is a path, relative to the top of the definitions repo,
+      to a chunk .morph file.
+    - `build-system` specifies one of the predefined build systems. You
+      must specify ONE of `morph` or `build-system` for each chunk.
+  In addition to these keys, each of the sources can specify a list of
+  build dependencies using the `build-depends` field. To specify one or
+  more chunk dependencies, `build-depends` needs to be set to a list
+  that contains the names of chunks that the source depends on in the
+  same stratum. These names correspond to the values of the `name`
+  fields of the other chunks.
+
+  At the moment, the ordering is significant in chunk build-depends. This
+  is used during bootstrapping, when you want to override the first build of
+  a component with its second version in a staging area. This feature is kind
+  of a workaround for the lack of distinction between build and runtime
+  dependencies.
+
+For systems, use the following fields:
+
+* `strata`: a list of key/value mappings, similar to the 'chunks' field of a
+  stratum. Two fields are allowed (are both required?):
+    - `name`: name of the artifact when the stratum is build
+    - `morph`: path to a stratum .morph file relative to the top of the containing repo
+
+Example chunk (simplified commands):
+
+    name: eglibc
+    kind: chunk
+    configure-commands:
+    - mkdir o
+    - cd o && ../libc/configure --prefix=/usr
+    build-commands:
+    - cd o && make
+    install-commands:
+    - cd o && make install_root="$DESTDIR" install
+
+Example stratum:
+
+    name: foundation
+    kind: stratum
+    chunks:
+    - name: fhs-dirs
+      repo: upstream:fhs-dirs
+      ref: baserock/bootstrap
+      build-depends: []
+    - name: linux-api-headers
+      repo: upstream:linux
+      ref: baserock/morph
+      build-depends:
+      - fhs-dirs
+    - name: eglibc
+      repo: upstream:eglibc
+      ref: baserock/bootstrap
+      build-depends:
+      - linux-api-headers
+    - name: busybox
+      repo: upstream:busybox
+      ref: baserock/bootstrap
+      build-depends:
+      - fhs-dirs
+      - linux-api-headers
+
+Example system:
+
+    name: base
+    kind: system
+    strata:
+    - morph: foundation
+    - morph: linux-stratum
+
+### Deployment definitions: Clusters
+
+**NOTE**: The deployment mechanism specified here is quite abstract. Most of
+the code used to do real-world deployments is currently tied to [[Morph]] and
+kept in [morph.git].
+
+For 'deployment', [[Morph]] defines an API for running 'extensions'. The
+'cluster' and 'system' definitions together describe what extensions should be
+run, and what should be set in their environment, in order to deploy the
+system. See the "Deployment extension API" section below for how to find and
+execute the extensions.
+
+Within this document, consider "deployment" to be a process of first
+post-processing a filesystem tree with one or more 'configure extensions', then 
+performing an operation to convert and/or transfer the filesystem tree
+using a 'write extension'.
+
+A cluster morphology defines a list of systems to deploy, and for each system a
+list of ways to deploy them. Use the following field:
+
+* **systems**: a list of systems to deploy;
+    the value is a list of mappings, where each mapping has the
+    following keys:
+
+    * **morph**: the system morphology to use in the specified
+        commit.
+
+    * **deploy**: a mapping where each key identifies a
+        system and each system has at least the following keys:
+
+        * **type**: identifies the type of deployment e.g. (kvm,
+            pxeboot), and thus what '.write' extension should be used.
+        * **location**: where the deployed system should end up
+            at. The syntax depends on the '.write' extension chosen in the
+            'type' field.
+
+        Optionally, it can specify **upgrade-type** and
+        **upgrade-location** as well, which should be interpreted in the same
+        way.
+
+        The system dictionary can have any number of other entries. These
+        should be collected up and are passed to each '.configure' extension
+        and to the '.write' extension, through the environment. The extensions
+        can interpret any of them in any manner.
+
+    * **deploy-defaults**: allows multiple deployments of the same
+        system to share some settings, when they can. Default settings
+        will be overridden by those defined inside the deploy mapping.
+
+    * **subsystems**: structured in the same way as the 'systems' entry, this
+        allows deploying something *within* a system. The Baserock reference
+        definitions use this to provide an initramfs inside some of the
+        reference systems.
+
+Example:
+
+    name: cluster-foo
+    kind: cluster
+    systems:
+        - morph: devel-system-x86_64-generic.morph
+            deploy:
+                cluster-foo-x86_64-1:
+                    type: extensions/kvm
+                    location: kvm+ssh://user@host/x86_64-1/x86_64-1.img
+                    upgrade-type: extensions/ssh-rsync
+                    upgrade-location: root@localhost
+                    HOSTNAME: cluster-foo-x86_64-1
+                    DISK_SIZE: 4G
+                    RAM_SIZE: 4G
+                    VCPUS: 2
+        - morph: devel-system-armv7-highbank
+            deploy-defaults:
+                type: extensions/pxeboot
+                location: cluster-foo-pxeboot-server
+            deploy:
+                cluster-foo-armv7-1:
+                    HOSTNAME: cluster-foo-armv7-1
+                cluster-foo-armv7-2:
+                    HOSTNAME: cluster-foo-armv7-2
+
+### Repo URLs
+
+Git repository locations can (and should) be abbreviated using the 'repo-alias' feature of Baserock definitions. This is a kind of [Compact URI](http://www.w3.org/TR/2009/CR-curie-20090116/). It currently only affects the 'repo' fields in a stratum .morph file.
+
+For example, instead of writing this:
+
+```
+- name: fhs-dirs
+  repo: git://git.baserock.org/baserock/baserock/fhs-dirs.git
+  ref: master
+```
+
+You can write this:
+
+```
+- name: fhs-dirs
+  repo: baserock:baserock/fhs-dirs.git
+  ref: master
+```
+
+There are two repo aliases that *must* be defined:
+
+ - `baserock:` (defaulting to git://git.baserock.org/baserock/)
+ - `upstream:` (defaulting to git://git.baserock.org/delta/)
+
+Baserock tools should allow changing these values. The main benefit of this compact URI scheme is that definitions are not tied to a specific Git server, or protocol. You can build against a mirror of the original Git server, or change the protocol that is used, just by altering the repo-alias configuration.
+
+Build environment
+-----------------
+
+### Sandboxing
+
+Builds should be done an isolated 'staging area', with only the specified dependencies available to the build process. The simplest approach is to install the dependencies in an empty directory, then [chroot](https://en.wikipedia.org/wiki/Chroot) into it. The more sandboxing the build tool can do, the better, because it lowers the chance of unexpected and unreproducible errors in the build process. The [Sandboxlib](https://github.com/CodethinkLabs/sandboxlib) Python library may be useful.
+
+The exception to the above is if the 'build-mode' field for a chunk is set to 'bootstrap'. Chunks in bootstrap mode are treated specially and do have access to tools from the host system. 
+
+FIXME: more detail is needed here!
+
+### Environment variables
+
+The following environment variables can be used in chunk configure/build/install commands, and must be defined by the build tool.
+
+ - `MORPH_ARCH`: the Morph-specific architecture name; see <http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/tree/morphlib/util.py#n473> for a list of valid architectures
+ - `PREFIX`: the value of the 'prefix' field for this chunk (set in the stratum .morph file); default /usr
+ - `TARGET`: the [GNU architecture triplet](http://wiki.osdev.org/Target_Triplet) for the target architecture (for example, x86_64-baserock-linux)
+ - `TARGET_STAGE1`: the 'bootstrap' variant of the GNU architecture triplet. This must be different from $TARGET -- you can just change the vendor field to achieve that (e.g. x86_64-bootstrap-linux).
+
+FIXME: The `TARGET` and `TARGET_STAGE1` fields are specific to building GNU/Linux based systems, they shouldn't be mandated in the spec.
+
+Deployment extension API
+------------------------
+
+**NOTE**: This section is more of an explaination of how the `morph deploy`
+command works than a general-purpose spec, in places. We hope to fix that in
+future.
+
+As noted in the previous section: within this document, consider "deployment"
+to be a process of first post-processing a filesystem tree with one or more
+'configure extensions', then performing an operation to convert and/or transfer
+the filesystem tree using a 'write extension'.
+
+### Deployment sequence
+
+A Baserock deployment tool should, for each labelled deployment in the
+'cluster', follow this sequence.
+
+1. Run the .check extension for that deployment 'type', if one exists. Abort
+   now, if it fails.
+
+   The .check extensions exist only to work around the fact that (2), (3) and
+   (4) may be slow, because programs that wait several minutes just to raise an
+   error that could have been detected right away are very annoying.
+
+2. Create a writeable temporary directory containing the contents of the system
+   artifact.
+
+3. For each 'subsystem' specified, recursively run steps 1, 3, 4 and 5. The
+   intention is that the result of the subsystem deployments end up *inside*
+   the temporary directory created in step 2 for the 'outer' deployment.
+
+4. Run each .configure extension that was specified in the
+   `configuration-extensions` field of the system morphology. The order of
+   running these is, sadly, unspecified.
+
+5. Run the .write extension for that deployment 'type'.
+
+The deployment of that system and its subsystems is now considered complete,
+and temporary data can be removed.
+
+### Locating and running extensions
+
+Deployment extensions are executable files that live in the top level directory
+of the definitions.git repo. Configure extensions have the extension
+`.configure`, check extensions have the extension `.check`, and write
+extensions have the extension `.write`.
+
+[[Morph]] will look for extensions inside the 'morphlib' Python package, in the
+`exts/` dir, before looking in the definitions.git repo.
+
+An extension must be executable using the [POSIX `exec()` system call]. We
+encourage writing them as Python scripts and using a `#!/usr/bin/env python`
+[hashbang], but any executable code is permitted.
+
+The 'execution environment' for an extension is sadly unspecified at the moment.
+[[Morph]] runs extensions without any kid of chroot environment, but each one
+is run in a separate [mount namespace]. Running extensions chrooted into the
+system being deployed does not make sense, as it may not contain the right
+dependencies for the extensions to work, and it may expect to run on a
+different, incompatible architecture to the system running the deployment in
+any case.
+
+A .check extension must be passed one commandline argument: the value of the
+`location` or `upgrade-location` field.
+
+A .configure extension must be passed one commandline argument: the path to
+the unpacked filesystem tree, which it can modify in some way. It is expected
+that a .configure extension will do nothing unless it is enabled using a
+configuration variable, but it is up to the code in the .configure extension
+to do this, and this behaviour is not currently enforced.
+
+A .write extension must be passed two commandline arguments: the value of the
+`location` or `upgrade-location` field, then the path to the unpacked
+filesystem tree (which it could modify in some way).
+
+All key/value pairs from the 'cluster' definition for the given labelled
+deployment of a system must be passed to all extensions, as environment
+variables. The `type`, `location`, `upgrade-type` and `upgrade-location` fields
+do not need to be passed in.
+
+Extensions are expected to set exit code 0 on success, and a non-zero value on
+failure. If any extension returns non-zero, the deployment tool should abort
+the deployment.
+
+Extensions are expected to write status information to 'stdout', and any error
+messages to 'stderr'. This is for user feedback ONLY, deployment tools should
+not do anything with this data other than log and/or display it.
+
+[[Morph]] sets an extra environment variable `MORPH_LOG_FD` which is a file
+descriptor that the extension can write log messages to, in order for them to
+appear in morph.log but not on stdout or stderr.
+
+[POSIX `exec()` system call]: http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
+[hashbang]: https://en.wikipedia.org/wiki/Shebang_%28Unix%29
+[mount namespace]: https://stackoverflow.com/questions/22889241/linux-understanding-the-mount-namespace-clone-clone-newns-flag#22889401
+
+### Help for extensions
+
+The .configure and .write extensions may provide .help files as documentation.
+The .help file should be valid [YAML], containing a dict with the key `help`
+and a string value.
+
+For example, the `tar.write` extension could provide `tar.write.help` with the
+following content:
+
+    help: |
+        Deploy a system as a .tar file.
+
+### Common configuration parameters used by extensions
+
+Fill me in!
+
+[morph.git]: git://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph
+[YAML]: http://yaml.org/