Document the DEFAULTS file, and rewrite description of chunks

Change-Id: Ie4e693fe37252860110b4d5ff0b7d172c7dae415

1 files changed, 139 insertions, 87 deletions

diff --git a/spec.mdwn b/spec.mdwn
index c7574d7..30366dc 100644
--- a/spec.mdwn
+++ b/spec.mdwn
@@ -62,6 +62,10 @@ uses. A tool should refuse to process a version that it doesn't support, to
 avoid unpredictable errors. See the "Versioning" heading above for more detail
 on versions.
 
+The top directory of the repo can also contain a file named `DEFAULTS`. This
+holds repo-wide 'build-system' and 'split-rules' information. See the
+'Defaults' section below.
+
 To find all the Baserock definition files in the repo, tooling can recursively
 scan the contents of the repo for files matching the glob pattern "\*.morph".
 
@@ -101,82 +105,75 @@ The structure of a 'chunk' definition is described using [JSON-Schema] in the
 [spec.git] repo:
 <http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas/chunk.json-schema>
 
-The fields mean the following:
+The build sequence consists of four phases:
+
+1. configure
+2. build
+3. install
+4. strip
+
+You can define one or more commands for each phase, or none. Here is an example
+chunk:
+
+    name: glibc
+    kind: chunk
+    description: GNU C library (example)
+
+    configure-commands:
+    - mkdir o
+    - cd o && ../libc/configure --prefix=/usr
+
+    build-commands:
+    - cd o && make
+
+    install-commands:
+    - cd o && make install_root="$DESTDIR" install
+
+A chunk can optionally make use of a repo-wide build system defined in
+'DEFAULTS', by using the `build-system` field.  In this case, the
+command sequences defined in DEFAULTS are used.
+
+Any predefined command sequence can be overridden by specifying a new value.
+You can also extend a predefined build system with the `pre-` or `post-`
+fields. For example, `pre-configure-commands` are run directly before
+`configure-commands`, and all `post-configure-commands` are run directly after.
+
+For example:
 
-* `build-system`: if the program is built using a build system known to
-  `morph`, you can set this field and avoid having to set the various
-  `*-commands` fields; the commands that the build system specifies can
-  be overridden; the following build-systems are known:
-
-  - `autotools`
-  - `python-distutils`
-  - `cpan`
-  - `cmake`
-  - `qmake`
-
-  optional
-
-* `pre-configure-commands`: a list of shell commands to run at
-  the configuration phase of a build, before the list in `configure-commands`;
-  optional
-* `configure-commands`: a list of shell commands to run at the configuraiton
-  phase of a build; optional
-* `post-configure-commands`: a list of shell commands to run at
-  the configuration phase of a build, after the list in `configure-commands`;
-  optional
-
-* `pre-build-commands`: a list of shell commands to run at
-  the build phase of a build, before the list in `build-commands`;
-  optional
-* `build-commands`: a list of shell commands to run to build (compile) the
-  project; optional
-* `post-build-commands`: a list of shell commands to run at
-  the build phase of a build, after the list in `build-commands`;
-  optional
-
-* `pre-test-commands`: a list of shell commands to run at
-  the test phase of a build, before the list in `test-commands`;
-  optional
-* `test-commands`: a list of shell commands to run unit tests and other
-  non-interactive tests on the built but un-installed project; optional
-* `post-test-commands`: a list of shell commands to run at
-  the test phase of a build, after the list in `test-commands`;
-  optional
-
-* `pre-install-commands`: a list of shell commands to run at
-  the install phase of a build, before the list in `install-commands`;
-  optional
-* `install-commands`: a list of shell commands to ; optional
-* `post-install-commands`: a list of shell commands to run at
-  the install phase of a build, after the list in `strip-commands`;
-  optional
-
-* `pre-strip-commands`: a list of shell commands to run at
-  the strip phase of a build, before the list in `strip-commands`;
-  optional
-* `strip-commands`: a list of shell commands to strip debug symbols from binaries;
-  this should strip binaries in the directory named in the `DESTDIR` environment
-  variable, not the actual system; optional
-* `post-strip-commands`: a list of shell commands to run at
-  the strip phase of a build, after the list in `install-commands`;
-  optional
-
-* `max-jobs`: a string to be given to `make` as the argument to the `-j`
-  option to specify the maximum number of parallel jobs; the only sensible
-  value is `"1"` (including the quotes), to prevent parallel jobs to run
-  at all; parallel jobs are only used during the `build-commands` phase,
-  since the other phases are often not safe when run in parallel; `morph`
-  picks a default value based on the number of CPUs on the host system;
-  optional
-
-* `chunks`: a key/value map of lists of regular expressions;
-  the key is the name
-  of a binary chunk, the regexps match the pathnames that will be
-  included in that chunk; the patterns match the pathnames that get installed
-  by `install-commands` (the whole path below `DESTDIR`); every file must
-  be matched by at least one pattern; by default, a single chunk gets
-  created, named according to the definition, and containing all files;
-  optional
+    name: git
+    kind: chunk
+    description: Git version control tool (example)
+
+    build-system: autotools
+
+    # This command will run before the normal 'configure' command sequence.
+    pre-configure-commands:
+    - make configure
+
+    # This command overrides the normal 'build' command sequence.
+    build-commands:
+    - make all
+
+If a chunk doesn't need to override anything from 'DEFAULTS', you can avoid
+having a chunk .morph file altogether, and just set 'build-system' when
+referring to the chunk from the stratum.
+
+The `max-jobs` field can be used to pass a custom value for --jobs to Make,
+via the `MAKEFLAGS` environment variable. This is useful for Makefiles which
+don't work in parallel: you can set `max-jobs: 1` to work around the problem.
+Parallel jobs are only used during the `build-commands` phase, since the other
+phases are often not safe when run in parallel; `morph` picks a default value
+based on the number of CPUs on the host system.
+
+Built chunks are split up into multiple artifacts. The default splitting rules
+for a chunk are defined in 'DEFAULTS'. You can use the `chunks` field of a
+chunk to override these. The `chunks` field is a key/value map of lists of
+regular expressions; the key is the name of a binary chunk, the regexps match
+the pathnames that will be included in that chunk; the patterns match the
+pathnames that get installed by `install-commands` (the whole path below
+`DESTDIR`); every file must be matched by at least one pattern; by default, a
+single chunk gets created, named according to the definition, and containing
+all files.
 
 #### Strata
 
@@ -234,18 +231,6 @@ The fields mean the following:
     - `name`: name of the artifact when the stratum is build
     - `morph`: path to a stratum .morph file relative to the top of the containing repo
 
-#### Example chunk (simplified commands):
-
-    name: eglibc
-    kind: chunk
-    configure-commands:
-    - mkdir o
-    - cd o && ../libc/configure --prefix=/usr
-    build-commands:
-    - cd o && make
-    install-commands:
-    - cd o && make install_root="$DESTDIR" install
-
 #### Example stratum:
 
     name: foundation
@@ -387,6 +372,73 @@ There are two repo aliases that *must* be defined:
 
 Baserock tools should allow changing these values. The main benefit of this compact URI scheme is that definitions are not tied to a specific Git server, or protocol. You can build against a mirror of the original Git server, or change the protocol that is used, just by altering the repo-alias configuration.
 
+### Defaults
+
+A definitions repository can contain a file named DEFAULTS. This file sets up
+repo-wide configuration.
+
+The structure of the DEFAULTS file is described using [JSON-Schema] in
+the [spec.git] repo:
+<http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas/defaults.json-schema>
+
+#### Build systems
+
+You can define common command sequences using the 'build-systems' key. These
+can then be referenced in chunk (and stratum) definitions, to avoid repeating
+common patterns.
+
+Here is an example DEFAULTS file that defines the python-distutils build
+system:
+
+    build-systems:
+      python-distutils:
+        configure-commands: []
+        build-commands:
+        - python setup.py build
+        install-commands:
+        - python setup.py install --prefix "$PREFIX" --root "$DESTDIR"
+
+#### Splitting rules
+
+One 'source' in Baserock can produce multiple binary 'artifacts'. This allows
+you to separate programs, libraries, documentation, debugging information, and
+whatever else into different artifacts. You can then choose to exclude some of
+these artifacts from your final system.
+
+Split rules are defined separately for chunks, and for strata.
+
+Chunks define a list of artifacts, and a series of regular expression patterns
+that are matched against filenames to define what to include. The list of
+artifacts is evaluated *in order*, so if two artifacts match the same pattern,
+those files will go into whichever artifact is defined first.
+
+Stratum matches work similarly, except the stratum artifacts are a series of
+patterns that match the *names of chunk artifacts*.
+
+Here is an example DEFAULTS file that sets up some simple splitting rules.
+
+    split-rules:
+      chunk:
+      - artifact: -devel
+        include:
+        - (usr/)?include/.*
+        - (usr/)?share/man/.*
+      - artifact: -runtime
+        include:
+        - .*
+
+      stratum:
+      - artifact: -devel
+        include:
+        - .*-devel
+      - artifact: -runtime
+        include:
+        - .*-runtime
+
+You could use these rules to keep header files (/usr/include/\*) and manual
+pages (/usr/share/man/\*) out of your systems, by only including the -runtime
+stratum artifacts.
+
 Build environment
 -----------------