summaryrefslogtreecommitdiff
path: root/README
blob: 9ed92845535d01cf584921259ab78b2c50b04625 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
README for morph
================

> **NOTA BENE:** This document is very much work-in-progress, and anything
> and everything may and will change at little or no notice. If you see
> problems, mail baserock-dev@baserock.org.

`morph` builds binaries for [Baserock](http://www.baserock.org/),
an embedded Linux solution. Please see the website for overall information.


Usage
-----

The Baserock builds are controlled by **morphology** files, 
which are build recipes. See below for their syntax. Everything
in Baserock is built from git commits, and also the morphologies
must be committed in git before building. The `morph` tool is
used to actually run the build. The usual workflow is this:

* put the morphology for an upstream project with its source code
* put other morphologies in the `morphs` (note plural) repository
* run `fakeroot morph` or `sudo morph` to build stuff

Eventually, `morph` will get a manual page. Meanwhile, a short usage:

    fakeroot morph --keep-path -v build-morphology morphs master foundation
    sudo morph --keep-path -v build-morphology morphs master base-system

(Run with `fakeroot`, when building anything but a system image; run
with `sudo` when building a system image.)

Run `morph --help` to get a list of all options and short descriptions.

You can write a configuration file to avoid having to write options on
the command line every time. Put it in `~/.morph.conf` and make it look
something like this:

    [config]
    cachedir = /home/username/baserock/cache
    log = /home/username/baserock/morph.log
    log-max = 200M
    trove-host = git.baserock.org

All of the above settings apart from `log` are the defaults, so may be omitted.


Morphology file syntax
----------------------

JSON is used for the morphology syntax. For example, to build a chunk:

    {
        "name": "foo",
        "kind": "chunk",
        "configure-commands": [
            "./configure --prefix=\"$PREFIX\""
        ],
        "build-commands": [
            "make"
        ],
        "test-commands": [
            "make check"
        ],
        "install-commands": [
            "make DESTDIR=\"$DESTDIR\" install"
        ]
    }

For all morphologies, use the following fields:

* `name`: the name of the morphology; it must currently match the filename
  (without the `.morph` suffix); **required**
* `kind`: the kind of thing being built; **required**

For chunks, use the following fields:

* `build-system`: if the program is built using a build system known to
  `morph`, you can set this field and avoid having to set the various
  `*-commands` fields; only `autotools` is currently known; the commands
  that the build system specifies can be overridden;
  optional
* `configure-commands`: a list of shell commands to run at the configuraiton
  phase of a build; optional
* `build-commands`: a list of shell commands to run to build (compile) the
  project; optional
* `test-commands`: a list of shell commands to run unit tests and other
  non-interactive tests on the built but un-installed project; optional
* `install-commands`: a list of shell commands to install the built project;
  the install should go into the directory named in the `DESTDIR` environment
  variable, not the actual system; optional
* `max-jobs`: a string to be given to `make` as the argument to the `-j`
  option to specify the maximum number of parallel jobs; the only sensible
  value is `"1"` (including the quotes), to prevent parallel jobs to run
  at all; parallel jobs are only used during the `build-commands` phase,
  since the other phases are often not safe when run in parallel; `morph`
  picks a default value based on the number of CPUs on the host system;
  optional
* `chunks`: a key/value map of lists of regular expressions; 
  the key is the name
  of a binary chunk, the regexps match the pathnames that will be
  included in that chunk; the patterns match the pathnames that get installed
  by `install-commands` (the whole path below `DESTDIR`); every file must
  be matched by at least one pattern; by default, a single chunk gets
  created, named according to the morphology, and containing all files;
  optional

For strata, use the following fields:

* `build-depends`: a list of strings, each of which refers to another
  stratum that the current stratum depends on. This list may be omitted
  or empty if the stratum does not depend on anything else.
* `chunks`: a list of key/value mappings, where each mapping corresponds
  to a chunk to be included in the stratum; the mappings may use the
  following keys: `name` is the chunk's name (may be different from the
  morphology name), `repo` is the repository in which to find (defaults to
  chunk name), `ref` identifies the commit to use (typically a branch
  name, but any tree-ish git accepts is ok), and `morph` is the name
  of the morphology to use and is optional. In addition to these keys,
  each of the sources MUST specify a list of build dependencies using the
  `build-depends` field. This field may be omitted to make the source
  depend on all other chunks that are listed earlier in the `chunks`
  list. The field may be an empty list to indicate that the chunk does 
  not depend on anything else in the same stratum. To specify one ore 
  more chunk dependencies, `build-depends` needs to be set to a list
  that contains the names of chunks that the source depends on in the 
  same stratum. These names correspond to the values of the `name` 
  fields of the other chunks.

For systems, use the following fields:

* `disk-size`: size of the disk image to be created; may use suffixes like
  `M` and `G`; **required**
* `strata`: a list of triples specifying strata to be included in the system
  image. Similar to chunks, you should give `repo`, `ref` and `morph`
  (filename without the .morph extension) for each stratum. The value of
  the `morph` field will be taken as the artifact name; if this causes
  ambiguity then an `alias` may be specified as well. **required**

Example chunk (simplified commands):

    {
        "name": "eglibc",
        "kind": "chunk",
        "configure-commands": [
            "mkdir o",
            "cd o && ../libc/configure --prefix=/usr"
        ],
        "build-commands": [
            "cd o && make"
        ],
        "install-commands": [
            "cd o && make install_root=\"$DESTDIR\" install"
        ]
    }

Example stratum:

    {
        "name": "foundation",
        "kind": "stratum",
        "chunks": [
            {
                "name": "fhs-dirs",
                "repo": "upstream:fhs-dirs",
                "ref": "baserock/bootstrap",
                "build-depends": [],
            },
            {
                "name": "linux-api-headers",
                "repo": "upstream:linux",
                "ref": "baserock/morph",
                "build-depends": ["fhs-dirs"]
            },
            {
                "name": "eglibc",
                "repo": "upstream:eglibc",
                "ref": "baserock/bootstrap",
                "build-depends": [
                  "linux-api-headers"
                ]
            },
            {
                "name": "busybox",
                "repo": "upstream:busybox",
                "ref": "baserock/bootstrap",
                "build-depends": [
                  "fhs-dirs",
                  "linux-api-headers"
                ]
            }
        ]
    }

Example system:

    {
        "name": "base",
        "kind": "system",
        "disk-size": "1G",
        "strata": [
            {
                "morph": "foundation",
                "repo": "baserock:baserock/morphs",
                "ref": "master"
            },
            {
                "morph": "linux-stratum",
                "repo": "baserock:baserock/morphs",
                "ref": "master"
            }
        ]
    }

Note that currently, unknown keys in morphologies are silently ignored.


Build environment
-----------------

When `morph` runs build commands, it clears the environment of all
variables and creates new ones. This is so that the build will run
more consistently across machines and developers.

See the `morphlib/builder.py` file, method `setup_env` for details.

Until Baserock is properly bootstrapped, it's awkward that the `PATH`
variable gets reset. To avoid that, use the `--keep-path` option.


Hacking morph
-------------

Run the test suite with this command:

    ./check

Install CoverageTestRunner (from <http://liw.fi/coverage-test-runner/>),
and check out the `cmdtest` utility (from <http://liw.fi/cmdtest/>).

Run the checks before submitting a patch, please.


The Bootstrap Process
---------------------

The goal of the bootstrap process is to create an environment that has
only been built with morph. It is a long boring process that can take
a few hours.

It can be started by running `./run-bootstrap-in-chroot $workingdir`
with some environment variables set.

The variables are:
* `DEBIAN_MIRROR`; which must be a path suitable for passing to
  debootstrap; which at the time of writing allowed http://,
  file:// or ssh:// urls; **required**
* `GIT_TARBALLS`; which must be a url to a tarballs directory;
  morph will try to fetch tarballs from here before trying the
  git-base-url; **required**
* `snapshot`; which should be `false` (or any executable that
  exits unsuccessfully) if creating a snapshot after each stage
  of the bootstrap is not desirable; optional

If snapshot is true or omitted then after each stage of the bootstrap
a tarball of the working directory is created, so that if a later stage
fails it can start from the last successful stage.

The mirror variables allow the bootstrap to be run more quickly if the
required files are available locally.

The mirror variables being required and snapshotting defaulting
to true are artefacts of the bootstrap process' development, where
only re-running the step that failed and using local mirrors were
essential for making fixes.


Bootstrap stages
----------------

The stages required for the bootstrap are to create a debian squeeze
chroot; build a skeleton system using the linux from scratch specified
tarballs; chroot into this system and build a system using morph with
bootstrap mode; then build the system again but instead using morph's
--staging-chroot mode and with the linux from scratch system removed.

The squeeze chroot (stage 0) is used so that the bootstrap process does
not need to be supported on every available host system, but every host
system that can run debootstrap can be indirectly supported.

In this stage debootstrap creates a chroot with all the packages required
to build the linux from scratch system, from then onwards every stage is
built by running baserock-bootstrap and optionally making a snapshot.

Stage 1 builds everything in a subdirectory of the squeeze-chroot called
tree, and all the packages are built with --prefix=/tools, so the full
path for `cat` would be `$workingdir/squeeze-chroot/tree/tools/bin/cat`.

This is done so that we have a known good system to work from, that won't
become entangled with the final system, which installed with --prefix=/usr.

Stage 2 uses morph to build a system, while chrooted into
`$workingdir/squeeze-chroot/tree`, this naturally required PATH to include
/tools/bin.

Morph is used to build the `foundation.morph` and `devel-bootstrap.morph`
strata in the `morphs` repository. `devel-bootstrap.morph` may not use the
same morphs as `devel.morph` because the bootstrap requires some hacks that
aren't required when a fully bootstrapped system is available.

Morph also uses the `--bootstrap` flag to make the chunks be unpacked after
they have been built, this is required for build-dependencies to be met as
prerequisites are expected to be available on the system.

After morph has finished a reasonably complete system should be available,
so `$workingdir/squeeze-chroot/tree/tools` is removed to save space and
make it obvious if stage 2 is missing anything needed for stage 3.

Stage 3 builds `devel.morph` which does not need all the hacks required
in building during the bootstrap, so it should be much more like what
morph will finally build.

Rather than using `--bootstrap`, stage 3 uses `--staging-chroot`, which
builds and extracts into a chroot, rather than using anything on the host
system.

For this to be able to build anything `--staging-filler` is used, which
allows for specifying the environment that chunks are built in. Stage 3
uses the strata that were build in stage 2.


Legalese
--------

Copyright (C) 2011, 2012  Codethink Limited

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.