path: root/README

README for morph
================

> **NOTA BENE:** This document is very much work-in-progress, and anything
> and everything may and will change at little or no notice. If you see
> problems, mail baserock-dev@baserock.org.

`morph` builds binaries for [Baserock](http://www.baserock.org/),
an embedded Linux solution. Please see the website for overall information.


Usage
-----

The Baserock builds are controlled by **morphology** files, 
which are build recipes. See below for their syntax. Everything
in Baserock is built from git commits, and also the morphologies
must be committed in git before building. The `morph` tool is
used to actually run the build. The usual workflow is this:

* put the morphology for an upstream project with its source code
* put other morphologies in the `morphs` (note plural) repository
* run `fakeroot morph` or `sudo morph` to build stuff

Eventually, `morph` will get a manual page. Meanwhile, a short usage:

    fakeroot morph --keep-path -v build morphs master foundation.morph
    sudo morph --keep-path -v build morphs master base-system.morph

(Run with `fakeroot`, when building anything but a system image; run
with `sudo` when building a system image.)

Run `morph --help` to get a list of all options and short descriptions.

You can write a configuration file to avoid having to write options on
the command line every time. Put it in `~/.morph.conf` and make it look
something like this:

    [config]
    cachedir = /home/username/baserock/cache
    log = /home/username/baserock/morph.log
    log-max = 200M
    git-base-url = file:///home/username/baserock/gits/

You should set at least the above settings.


Morphology file syntax
----------------------

JSON is used for the morphology syntax. For example, to build a chunk:

    {
        "name": "foo",
        "kind": "chunk",
        "configure-commands": [
            "./configure --prefix=$PREFIX"
        ],
        "build-commands": [
            "make"
        ],
        "test-commands": [
            "make check"
        ],
        "install-commands": [
            "make DESTDIR=$DESTDIR install"
        ]
    }

For all morphologies, use the following fields:

* `name`: the name of the morphology; it must currently match the filename
  (without the `.morph` suffix); **required**
* `kind`: the kind of thing being built; **required**

For chunks, use the following fields:

* `build-system`: if the program is built using a build system known to
  `morph`, you can set this field and avoid having to set the various
  `*-commands` fields; only `autotools` is currently known and that
  requires the `configure` script to have been committed into git; 
  optional
* `configure-commands`: a list of shell commands to run at the configuraiton
  phase of a build; optional
* `build-commands`: a list of shell commands to run to build (compile) the
  project; optional
* `test-commands`: a list of shell commands to run unit tests and other
  non-interactive tests on the built but un-installed project; optional
* `install-commands`: a list of shell commands to install the built project;
  the install should go into the directory named in the `DESTDIR` environment
  variable, not the actual system; optional
* `max-jobs`: a string to be given to `make` as the argument to the `-j`
  option to specify the maximum number of parallel jobs; the only sensible
  value is `"1"` (including the quotes), to prevent parallel jobs to run
  at all; parallel jobs are only used during the `build-commands` phase,
  since the other phases are often not safe when run in parallel; `morph`
  picks a default value based on the number of CPUs on the host system;
  optional
* `chunks`: a key/value map of lists of regular expressions; 
  the key is the name
  of a binary chunk, the regexps match the pathnames that will be
  included in that chunk; the patterns match the pathnames that get installed
  by `install-commands` (the whole path below `DESTDIR`); every file must
  be matched by at least one pattern; by default, a single chunk gets
  created, named according to the morphology, and containing all files;
  optional

For strata, use the following fields:

* `build-depends`: a list of strings, each of which refers to another
  stratum that the current stratum depends on. This list may be omitted
  or empty if the stratum does not depend on anything else.
* `sources`: a list of key/value mappings, where each mapping corresponds
  to a chunk to be included in the stratum; the mappings may use the
  following keys: `name` is the chunk's name (may be different from the
  morphology name), `repo` is the repository in which to find (defaults to
  chunk name), `ref` identifies the commit to use (typically a branch
  name, but any tree-ish git accepts is ok), and `morph` is the name
  of the morphology to use and is optional. In addition to these keys,
  each of the sources may specify a list of build dependencies using the
  `build-depends` field. This field may be omitted to make the source
  depend on all other chunks that are listed earlier in the `sources`
  list. The field may be an empty list to indicate that the chunk does 
  not depend on anything else in the same stratum. To specify one ore 
  more chunk dependencies, `build-depends` needs to be set to a list
  that contains the names of chunks that the source depends on in the 
  same stratum. These names correspond to the values of the `name` 
  fields of the other sources.

For systems, use the following fields:

* `disk-size`: size of the disk image to be created; may use suffixes like
  `M` and `G`; **required**
* `strata`: a list of stratum names to include in the system image;
  **required**
* `test-stories`: a list of filenames with "test stories" for morph to 
  run against the built image (see `morph test` subcommand); optional

Example chunk (simplified commands):

    {
        "name": "eglibc",
        "kind": "chunk",
        "configure-commands": [
            "mkdir o",
            "cd o && ../libc/configure --prefix=/usr"
        ],
        "build-commands": [
            "cd o && make"
        ],
        "install-commands": [
            "cd o && make install_root=\"$DESTDIR\" install"
        ]
    }

Example stratum:

    {
        "name": "foundation",
        "kind": "stratum",
        "sources": [
            {
                "name": "fhs-dirs",
                "ref": "baserock/bootstrap"
            },
            {
                "name": "linux-api-headers",
                "repo": "linux",
                "ref": "baserock/morph",
                "build-depends": []
            },
            {
                "name": "eglibc",
                "ref": "baserock/bootstrap",
                "build-depends": [
                  "linux-api-headers"
                ]
            },
            {
                "name": "busybox",
                "ref": "baserock/bootstrap",
                "build-depends": [
                  "fhs-dirs",
                  "linux-api-headers"
                ]
            }
        ]
    }

Example system:

    {
        "name": "base",
        "kind": "system",
        "disk-size": "1G",
        "strata": [
            "foundation",
            "linux-stratum"
        ],
        "test-stories": [
            "stories/login.py",
            "stories/hostname-is-set.py",
            "stories/basic-utilities-work.py",
            "stories/halt-works.py"
        ]
    }

Note that unknown keys in morphologies are silently ignored.


Test stories for system images
------------------------------

The `morph test` subcommand will run a series of automated tests against
an x86 system image under KVM using the serial console. Each test is
stored in a separate file, and consist of a sequence of send/expect pairs.
Example:

    story = [
        ('', 'login: '),
        ('root\r', '^~ # '),
    ]

Each file is a Python module, with a variable called `story`,
which is a list of tuples of two strings, or two strings and a
number. The first string is sent to the serial console. The second
string specifies a regular expression that is matched against the 
output from the serial console. The optional number is a timeout,
in seconds, for how long to wait for a match. The default is ten
seconds, and can be set with the `--test-timeout` option.

The test succeeds if all tuples are sent/matched successfully.


Build environment
-----------------

When `morph` runs build commands, it clears the environment of all
variables and creates new ones. This is so that the build will run
more consistently across machines and developers.

See the `morphlib/builder.py` file, method `setup_env` for details.

Until Baserock is properly bootstrapped, it's awkward that the `PATH`
variable gets reset. To avoid that, use the `--keep-path` option.


Hacking morph
-------------

Run the test suite with this command:

    ./check

Install CoverageTestRunner (from <http://liw.fi/coverage-test-runner/>),
and check out the `cmdtest` utility (from <http://liw.fi/cmdtest/>).

Run the checks before submitting a patch, please.


Legalese
--------

Copyright (C) 2011, 2012  Codethink Limited

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.