diff options
-rw-r--r-- | README | 343 |
1 files changed, 164 insertions, 179 deletions
@@ -3,43 +3,51 @@ README for morph > **NOTA BENE:** This document is very much work-in-progress, and anything > and everything may and will change at little or no notice. If you see -> problems, mail lars.wirzenius@codethink.co.uk (for now). +> problems, mail baserock-dev@baserock.org. `morph` builds binaries for [Baserock](http://www.baserock.org/), -an embedded Linux solution. Some important points: - -* everything is built from **source in git**, not release tarballs - and not from uncommitted changes -* a binary is called a **stratum**, and is a collection of software that forms - a whole, e.g., the core system for Baserock, the essential build tools - for Baserock, or the GNOME platform libraries - - later there will be support for **erratics**, for individual software - packages, which will be isolated from each other -* a stratum is atomic: it cannot be split into smaller parts -* parts of a stratum that correspond to individual upstream projects - (e.g., busybox for the core stratum, or a particular library in the GNOME - platform) can be built separately, but the result (a **chunk**) cannot - be installed on its own: it needs to be combined with other chunks to - form a complete stratum - -In other words: - -* an individual upstream project is built into a chunk -* chunks are combined into strata -* strata are installed completely, or not at all - -The build of a chunk or a stratum is controlled by a -**morphology**, which consists of: - -* type of result: chunk or stratum -* for a chunk, lists of commands for configuring, building, testing, and - installing the program -* for a stratum, one or more specifications of which chunks to build: - pairs of git repositories and commit references +an embedded Linux solution. Please see the website for overall information. + + +Usage +----- + +The Baserock builds are controlled by **morphology** files, +which are build recipes. See below for their syntax. Everything +in Baserock is built from git commits, and also the morphologies +must be committed in git before building. The `morph` tool is +used to actually run the build. The usual workflow is this: + +* put the morphology for an upstream project with its source code +* put other morphologies in the `morphs` (note plural) repository +* run `morph` to build stuff + +Eventually, `morph` will get a manual page. Meanwhile, a short usage: + + morph --keep-paths -v build morphs master base-system.morph + +Run `morph --help` to get a list of all options and short descriptions. + +You can write a configuration file to avoid having to write options on +the command line every time. Put it in `~/.obnam.conf` and make it look +something like this: + + [config] + cachedir = /home/liw/.cache/morph/bins + log = /home/liw/.cache/morph/morph.log + log-max = 200M + git-base-url = file:///home/liw/baserock/gits/ + +You should set at least the above settings. + + +Morphology file syntax +---------------------- JSON is used for the morphology syntax. For example, to build a chunk: { + "name": "foo", "kind": "chunk", "configure-commands": [ "./configure --prefix=$PREFIX" @@ -55,183 +63,160 @@ JSON is used for the morphology syntax. For example, to build a chunk: ] } -(Later, there will be defaults and things to make the morph files shorter.) +For all morphologies, use the following fields: + +* `name`: the name of the morphology; it must currently match the filename + (without the `.morph` suffix); **required** +* `kind`: the kind of thing being built; **required** + +For chunks, use the following fields: + +* `build-system`: if the program is built using a build system known to + `morph`, you can set this field and avoid having to set the various + `*-commands` fields; only `autotools` is currently known and that + requires the `configure` script to have been committed into git; + optional +* `configure-commands`: a list of shell commands to run at the configuraiton + phase of a build; optional +* `build-commands`: a list of shell commands to run to build (compile) the + project; optional +* `test-commands`: a list of shell commands to run unit tests and other + non-interactive tests on the built but un-installed project; optional +* `install-commands`: a list of shell commands to install the built project; + the install should go into the directory named in the `DESTDIR` environment + variable, not the actual system; optional +* `max-jobs`: a string to be given to `make` as the argument to the `-j` + option to specify the maximum number of parallel jobs; the only sensible + value is `"1"` (including the quotes), to prevent parallel jobs to run + at all; parallel jobs are only used during the `build-commands` phase, + since the other phases are often not safe when run in parallel; `morph` + picks a default value based on the number of CPUs on the host system; + optional +* `chunks`: a key/value map of lists of regular expressions; + the key is the name + of a binary chunk, the regexps match the pathnames that will be + included in that chunk; the patterns match the pathnames that get installed + by `install-commands` (the whole path below `DESTDIR`); every file must + be matched by at least one pattern; by default, a single chunk gets + created, named according to the morphology, and containing all files; + optional + +For strata, use the following fields: + +* `sources`: a list of key/value mappings, where each mapping corresponds + to a chunk to be included in the stratum; the mappings may use the + following keys: `name` is the chunk's name (may be different from the + morphology name), `repo` is the repository in which to find (defaults to + chunk name), `ref` identifies the commit to use (typically a branch + name, but any tree-ish git accepts is ok), and `morph` is the name + of the morphology to use; optional + +For systems, use the following fields: + +* `disk-size`: size of the disk image to be created; may use suffixes like + `M` and `G`; **required** +* `strata`: a list of stratum names to include in the system image; + **required** + +Example chunk (simplified commands): + + { + "name": "eglibc", + "kind": "chunk", + "configure-commands": [ + "mkdir o", + "cd o && ../libc/configure --prefix=/usr" + ], + "build-commands": [ + "cd o && make" + ], + "install-commands": [ + "cd o && make install_root=\"$DESTDIR\" install" + ] + } + +Example stratum: -To build a stratum: { + "name": "foundation", "kind": "stratum", "sources": [ { - "repo": "git://git.baserock.org/busybox/", - "ref": "DEADBEEF" + "name": "fhs-dirs", + "ref": "baserock/bootstrap" + }, + { + "name": "linux-api-headers", + "repo": "linux", + "ref": "baserock/morph" + }, + { + "name": "eglibc", + "ref": "baserock/bootstrap" }, { - "repo": "git://git.baserock.org/gzip/", - "ref": "CAFEBABE" + "name": "busybox", + "ref": "baserock/bootstrap" } ] } -To use morph, create the relevant morphology files (`*.morph`), -commit them to a git repository, and then run a command like the following: - - morph build file://$HOME/baserock/morphs master core.morph - -This will build the Baserock core stratum. It will recursively build -any chunks that also need to be built. The last three command line -arguments are: - -* URL to repository; this may be relative to `--git-base-url` or absolute. -* A git commit reference; here, the HEAD of the `master` branch is used. -* The filename in the git commit. - -Note that morph retrieves the morphology file from git. It does not use -the copy that is checked out. If you make any modifications, you **must** -commit them. +Example system: + { + "name": "base", + "kind": "system", + "disk-size": "1G", + "strata": [ + "foundation", + "linux-stratum" + ] + } -Morphology spec ---------------- +Note that unknown keys in morphologies are silently ignored. -A morphology is a JSON file with a single object (dict), with the -following keys: -* `kind`: either `chunk` or `stratum` -* `sources`: a dict, whose keys are names and the corresponding values - are dicts with the following keys (the top level names are - for documentation only): - - `repo`: URL to git repository - - the URL may be relative to the value given to the - `--git-base-url` option - - `ref`: a reference to a commit (either a commit id, or `HEAD`) -* `configure-commands`: a list of strings giving shell commands - that should be executed to configure the software being built - (can also be a single string instead of a list) -* `build-commands`: similarly, commands for building the software -* `test-commands`: similarly, commands for running automatic tests - on the built (but uninstalled) software -* `install-commands` similarly, commands for installing the software +Build environment +----------------- -Unknown keys are errors. Known keys with the wrong kind of values -result in an error. Keys that are valid only for chunks are errors -when given for strata, and vice versa. +When `morph` runs build commands, it clears the environment of all +variables and creates new ones. This is so that the build will run +more consistently across machines and developers. -Commands run during the building of a chunk are passed on to the shell -for execution, and may refer to the following extra environment variables: +See the `morphlib/builder.py` file, method `setup_env` for details. -* `WORKAREA`: the temporary directory where all actual building occurs - - commands must avoid touching anything in the actual source tree, - and must modify files only in the temporary directory -* `DESTDIR`: to be prefixed to install paths during install +Until Baserock is properly bootstrapped, it's awkward that the `PATH` +variable gets reset. To avoid that, use the `--keep-path` option. -Build process +Hacking morph ------------- -You give morph one or more morphologies (`*.morph`) in files, and -it builds them in order. Built chunks are stored in a central -cache directory (see `morph --cachedir`). Built strata are stored -in the current working directory. - -During the build of a chunk, morph goes through the following steps: - -* export the files from git to a temporary location, so the build happens - in a clean directory -* configure, build, and test the software -* install the software into a temporary directory -* create a chunk file of the contents of the temporary directory -* clean up by removing all temporary stuff - (a "make clean" target is **not** run) -* put chunk and build log and other deliverables in their places +Run the test suite with this command: -For strata, morph does this instead: + ./check -* build all chunks missing from the chunk directory -* create a temporary directory -* unpack all the chunks into the temporary directory -* create a stratum file from the temporary directory -* clean up -* put stratum file and build log and other deliverables in their places +Install CoverageTestRunner (from <http://liw.fi/coverage-test-runner/>), +and check out the `cmdtest` utility (from <http://liw.fi/cmdtest/>). -As a morph user, you should never need to build chunks manually. -Just create stratum morphologies, and build those: the chunks get -built automatically. +Run the checks before submitting a patch, please. -File formats ------------- -Both chunk and stratum files use the same file format, for simplicity. -The file is a tar file, to be unpacked at the filesystem root, and -all permission bits set exactly as they should be at the final install. -The metadata is stored in a directory `BASEROCK` at the root of the -directory tree, with the following files (where `foo` is the name -of the chunk or stratum): +Legalese +-------- -* `foo.json`: a JSON file with the metadata of the chunk or stratum -* in the future, there may be preinst, postinst, etc, scripts as well, - but as far as possible, we will try to do without +Copyright (C) 2011 Codethink Limited -No two chunks that are put into the same stratum may contain the -same files, and no two strata installed on the same Baserock system -can contain the same files. +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; version 2 of the License. -Note that this file format is preliminary, and may well change in -the future. It is chosen for simplicity of implementation, not -any other reason. - -Any tar variant that busybox tar and Python's tar library both can -unpack is acceptable. - - -Hacking morph -------------- +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. -You can run `morph` from the unpacked source tree: - - ./morph build foo.morph - -To run unit tests: - - nosetests - -Alternatively (and preferably), install CoverageTestRunner -(from <http://liw.fi/coverage-test-runner/>) and run this: - - python setup.py check - -To run black box tests, get cmdtest (from <http://liw.fi/cmdtest/>) -and run this: - - cmdtest tests -c ./morph --log cmdtest.log - -You should probably run the automatic tests before submitting a -bug report. A patch that includes a unit test or black box test is -most welcome. - - -Open questions and things to consider -------------------------------------- - -* When morph starts building missing chunks automatically, how will it - find the `*.morph` files? - - from the specified git? - - from some default git? - - from some other central location? - - from all of the above, in some order? - - maybe allow specifying the morph file in the "source" part of a - stratum's morphology? -* Build dependencies will need to be specified in some way. They should - be on whole strata, not individual software. -* There may be build dependencies between the projects that go into the - same chunk: app foo may build-depend on libfoobar in the same stratum. - We need to deal with this. Possibly put the things into `source` in - build order? -* Build dependency loops should be detected, and if found, they should - fail the build. -* We need ways to make use of git repositories that already local: - - if developer has cloned a repo, use that, instead of accessing the - central copy, for speed, and also so that we can build the developer's - changes without pushing them to a server - - also need a way to use mirrored repos, e.g., mirrored into - the Codethink office from a public server, for performance reasons +You should have received a copy of the GNU General Public License along +with this program; if not, write to the Free Software Foundation, Inc., +51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. |