path: root/README.md

GitLab: [![build status](https://gitlab.com/baserock/ybd/badges/master/build.svg)](https://gitlab.com/baserock/ybd/commits/master)


# ybd

ybd is a tool for building integrated software stacks. It does four things:

- parse yaml files which describe integrated collections of software components
- gather source code (from git repos) for a user-specified target collection
- build the collection
- optionally deploy the collection (if the yaml files describe where/how to
  deploy)

Currently ybd understands the semantics of yaml
[definitions](http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/)
from the [Baserock](http://wiki.baserock.org) project.

The total codebase for ybd is only ~ 2200 lines of Python. even so ybd can
reliably build all systems in Baserock's definitions.git, i.e. all of the FOSS
components required for Linux appliances up to and including:

- minimal Linux systems
- self-hosting development systems
- GENIVI Baseline and GENIVI Demo Platform systems for x86 and ARMv7
- OpenStack appliances
- OpenWRT appliances

ybd is a simple start-point for building, deploying, learning and experimenting
with definitions, algorithms and functionality on Baserock projects. With a
little work it can be used to build other software stacks too.

ybd is under development. things will change :) If you're trying ybd for the
first time please start with the latest tag, not master.

ybd is designed for Linux, but can be used on other environments (eg MacOS)
using Vagrant and VirtualBox.

## Quick Start

On Fedora, Ubuntu, Debian (as root... hoping to fix this for non-root soon):

    git clone git://gitlab.com/baserock/ybd && cd ybd
    # checkout latest tag
    git checkout `git describe --tags $(git rev-list --tags --max-count=1)`
    sudo sh ./install_dependencies.sh
    cd .. && git clone git://git.baserock.org/baserock/baserock/definitions
    cd definitions

Or using Vagrant with VirtualBox:

    git clone git://gitlab.com/baserock/ybd && cd ybd
    # checkout latest tag
    git checkout `git describe --tags $(git rev-list --tags --max-count=1)`
    vagrant up
    vagrant ssh
    sudo -i
    cd /src/definitions

Once there you can run (as root)

    ../ybd/ybd.py $1 $2

where

- `$1` is relative path to a chunk, stratum, system or cluster definition file
- `$2` specifies the architecture you're building/deploying for.

Note that all of the current baserock definitions are for native builds, no
cross-compile. So if you're on a PC/Mac it may be that your only option for
`$2` is `x86_64`.

Some examples to try:

    # on an x86_64 laptop running linux, to build a build-system...
    ../ybd/ybd.py systems/build-system-x86_64.morph x86_64

    # on the same laptop, to build gcc (and its dependencies)
    ../ybd/ybd.py gcc x86_64

    # on the same laptop, to build the genivi stratum (and its dependencies)
    ../ybd/ybd.py strata/genivi.morph x86_64

    # on a jetson, to build a GENIVI baseline system...
    ../ybd/ybd.py systems/genivi-baseline-system-armv7lhf-jetson.morph armv7lhf

    # on anything, to build (and deploy?) parts of baserock release...
    ../ybd/ybd.py clusters/release.morph x86_64

    # in a baserock devel vm (x86_64), to build and deploy a self-upgrade...
    ../ybd/ybd.py clusters/upgrade-devel.morph x86_64

ybd generates a lot of log output to stdout, which hopefully helps to explain
what is happening. If you need a permanent log then try

    ../ybd/ybd.py clusters/upgrade-devel.morph x86_64 | tee some-memorable-name.log

More information about ybd's dependencies is in
[dependencies](docs/Dependencies.md).


## Configuration

ybd is designed to be run from the command line and/or as part of an automated
pipeline. all configuration is taken from conf files and/or environment
variables, in the following order of precedence:

    YBD_* environment variables         # if found
    ./ybd.conf                          # if found
    $path/to/ybd.py/ybd.conf            # if found
    $path/to/ybd.py/ybd/config/ybd.conf # default, as provided in the ybd repo

This means you can set custom config via env vars, or the definitions top-level
directory, or the ybd top-level directory, without having to modify the
supplied default [ybd.conf](ybd/config/ybd.conf).

NOTE we recommend you create your own ybd.conf file. that way you can git merge
new versions of ybd + definitions, and your custom settings will always take
precedence, with no possibility of a merge conflict.

To set config via environment variables, each must be prefixed with `YBD_`
and using `_` instead of `-`. ybd will strip the `YBD_` prefix and
convert `_` to `-`. For example

    export YBD_artifact_version=1     # artifact-version: 1
    export YBD_log_verbose=True       # log-verbose: True
    export YBD_instances=2            # instances: 2

ybd does not support unix-style --flags so far. If enough people complain about
that, it can be fixed.

For details about the config options themselves, see
[ybd.conf](ybd/config/ybd.conf).

## Interesting Features

### Run ybd in Parallel

ybd can fork several instances of itself to parallelise its work. There is no
intelligence in the scheduling - all of the forks just randomise their
build-order and try to build everything. For building a set of overlapping
systems in parallel on a many core machine this proves to be quite effective.
For example on a 36-core AWS c4.8xlarge machine, 4 racing instances of ybd can
build all of the x86_64 systems in definitions/clusters/ci.morph much faster
than a single instance.

To try it, just set the `instances` config variable, for example

    # as an environment variable...
    export YBD_instances=4

    # or in a ybd.conf file...
    instances: 8

You should probably think about setting `max-jobs` too, taking into account
your workloads and host machine(s). If `max-jobs` is not set, ybd will default
it to number-of-cores/instances.

### kbas Cache Server

kbas is a basic server which can be used to allow other users to access
pre-built artifacts from previous or current runs of ybd. See kbas/kbas.py for
the code. With minimal configuration it can serve artifacts to instances of ybd
on other machines, and also receive uploaded artifacts.

To launch kbas, just do

    cd kbas && ./kbas.py

By default ybd is configured to check for artifacts from a kbas server at
<http://artifacts1.baserock.org:8000/>.

Config for kbas follows the same approach as ybd, defaulting to config in
`kbas/config/kbas.conf`.

NOTES:
- the default password is 'insecure' and upload functionality is disabled
unless you change it.
- for testing kbas you can use cURL to upload files:

    curl  -X POST  -i  -F 'filename=<cachid>' -F 'file=@path/to/file' \
    -F 'password=<password>' <host>:<port>/upload

### Git LFS Support

ybd supports repositories that manage their binaries via
[Git LFS](https://github.com/git-lfs/git-lfs).  Installation of the git-lfs
binary is required in order to make use of this feature, otherwise ybd will exit
with an error.

### Concourse Pipelines

[WORK IN PROGRESS] ybd can generate concourse pipelines - see the code at
ybd/concourse.py

To start a local concourse instance:

    # in your definitions directory
    vagrant init concourse/lite
    vagrant up
    # generate pipeline
    echo "mode: parse-only" >> ./ybd.conf
    python ../ybd/ybd.py <target> <arch>
    fly -t local login -c http://192.168.100.4:8080
    fly -t local set-pipeline -p <target> -c <target>.yml

You can view the local pipelines at <http://192.168.100.4:8080>. Note ybd's
concourse pipelines are for visualisation only so far - they don't build.

## TODO

In no particular order, here's a list of things that may be interesting to try
doing to/with ybd:
- pip install
- Deterministic (bit-for-bit) build
- Fix deployment: ybd currently can deploy systems, subsystems (test this?) and
  upgrades, but it's a mess:
  - can't select which subsystems to deploy (need to iron out the definitions)
  - alter the deployment parameters (templating, or commandline?)
  - deploy other-architecture stuff
- Run without root privileges on non-Linux environments (eg Mac OS - needs a
  different isolation method instead of linux-user-chroot).
- Test assumptions by creating definitions for other OS software sets, eg
  Aboriginal Linux and non-Linux systems.
- Test/fix to work on old versions of definitions, and on morphs.git (and
  separate out the idiosyncrasies, so code is tidy for future changes)

## Project Guidelines

- Contributions are extremely welcome - feedback, ideas, suggestions, bugs,
  documentation will be appreciated just as much as code.
- Key criteria for code contributions are:
  - 'Code LESS: every line creates a work-chain. Code is a liability, not an
    asset'
  - think hard before adding dependencies
  - code should be tested and pass pep8

## License

- License is GPLv2 but other licensing can be considered on request.
- Most of the copyright is currently Codethink but don't let that put you off.
  There's no intent to keep this as a Codethink-only project, nor will there be
  any attempt to get folks to sign a contributor agreement.  contributors
  retain their own copyright.