From 9d7296eb84d604a02e5e9f989a09cb30994503c1 Mon Sep 17 00:00:00 2001 From: Tristan Van Berkom Date: Tue, 12 Jun 2018 23:20:14 -0400 Subject: doc: Overhaul of page names and titles o Giving main pages simple word titles This makes the main page: * About * Installing * Using * Reference * Contributing o Now named all rst files with their parent page name as a prefix. o Also changed some titles to make overall consistent titles. --- HACKING.rst | 4 +- README.rst | 4 +- buildstream/source.py | 2 +- doc/source/about.rst | 1 - doc/source/additional_cachekeys.rst | 96 +++++ doc/source/additional_sandboxing.rst | 200 +++++++++ doc/source/artifacts.rst | 200 --------- doc/source/authoring.rst | 79 ---- doc/source/cachekeys.rst | 96 ----- doc/source/commands.rst | 127 ------ doc/source/core_additional.rst | 4 +- doc/source/core_format.rst | 17 + doc/source/core_framework.rst | 4 +- doc/source/core_plugins.rst | 62 +++ doc/source/docker.rst | 45 -- doc/source/examples.rst | 12 - doc/source/format.rst | 418 ------------------ doc/source/format_declaring.rst | 419 ++++++++++++++++++ doc/source/format_intro.rst | 291 +++++++++++++ doc/source/format_project.rst | 811 +++++++++++++++++++++++++++++++++++ doc/source/format_project_refs.rst | 78 ++++ doc/source/format_public.rst | 102 +++++ doc/source/formatintro.rst | 291 ------------- doc/source/index.rst | 2 +- doc/source/install.rst | 175 -------- doc/source/install_artifacts.rst | 200 +++++++++ doc/source/install_docker.rst | 45 ++ doc/source/install_main.rst | 175 ++++++++ doc/source/main_about.rst | 1 + doc/source/main_core.rst | 7 +- doc/source/main_install.rst | 10 +- doc/source/main_using.rst | 10 +- doc/source/projectconf.rst | 811 ----------------------------------- doc/source/projectrefs.rst | 78 ---- doc/source/public.rst | 102 ----- doc/source/sandboxing.rst | 200 --------- doc/source/user_config.rst | 97 ----- doc/source/using_commands.rst | 127 ++++++ doc/source/using_config.rst | 97 +++++ doc/source/using_examples.rst | 12 + 40 files changed, 2757 insertions(+), 2755 deletions(-) delete mode 100644 doc/source/about.rst create mode 100644 doc/source/additional_cachekeys.rst create mode 100644 doc/source/additional_sandboxing.rst delete mode 100644 doc/source/artifacts.rst delete mode 100644 doc/source/authoring.rst delete mode 100644 doc/source/cachekeys.rst delete mode 100644 doc/source/commands.rst create mode 100644 doc/source/core_format.rst create mode 100644 doc/source/core_plugins.rst delete mode 100644 doc/source/docker.rst delete mode 100644 doc/source/examples.rst delete mode 100644 doc/source/format.rst create mode 100644 doc/source/format_declaring.rst create mode 100644 doc/source/format_intro.rst create mode 100644 doc/source/format_project.rst create mode 100644 doc/source/format_project_refs.rst create mode 100644 doc/source/format_public.rst delete mode 100644 doc/source/formatintro.rst delete mode 100644 doc/source/install.rst create mode 100644 doc/source/install_artifacts.rst create mode 100644 doc/source/install_docker.rst create mode 100644 doc/source/install_main.rst create mode 100644 doc/source/main_about.rst delete mode 100644 doc/source/projectconf.rst delete mode 100644 doc/source/projectrefs.rst delete mode 100644 doc/source/public.rst delete mode 100644 doc/source/sandboxing.rst delete mode 100644 doc/source/user_config.rst create mode 100644 doc/source/using_commands.rst create mode 100644 doc/source/using_config.rst create mode 100644 doc/source/using_examples.rst diff --git a/HACKING.rst b/HACKING.rst index a1cffc880..8ef82244f 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -1,5 +1,5 @@ -Hacking on BuildStream -====================== +Contributing +============ Some tips and guidelines for developers hacking on BuildStream diff --git a/README.rst b/README.rst index f571ebff6..6f64f47a6 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -About BuildStream ------------------ +About +----- .. image:: https://gitlab.com/BuildStream/buildstream/badges/master/pipeline.svg :target: https://gitlab.com/BuildStream/buildstream/commits/master diff --git a/buildstream/source.py b/buildstream/source.py index 470a407ad..387ed6a99 100644 --- a/buildstream/source.py +++ b/buildstream/source.py @@ -18,7 +18,7 @@ # Authors: # Tristan Van Berkom """ -Source - base source class +Source - Base source class ========================== diff --git a/doc/source/about.rst b/doc/source/about.rst deleted file mode 100644 index a6210d3d8..000000000 --- a/doc/source/about.rst +++ /dev/null @@ -1 +0,0 @@ -.. include:: ../../README.rst diff --git a/doc/source/additional_cachekeys.rst b/doc/source/additional_cachekeys.rst new file mode 100644 index 000000000..f0df796c5 --- /dev/null +++ b/doc/source/additional_cachekeys.rst @@ -0,0 +1,96 @@ + +.. _cachekeys: + + +Cache keys +========== + +Cache keys for artifacts are generated from the inputs of the build process +for the purpose of reusing artifacts in a well-defined, predictable way. + +Structure +--------- +Cache keys are SHA256 hash values generated from a pickled Python dict that +includes: + +* Environment (e.g., project configuration and variables) +* Element configuration (details depend on element kind, ``Element.get_unique_key()``) +* Sources (``Source.get_unique_key()``) +* Dependencies (depending on cache key type, see below) +* Public data + +Cache key types +--------------- +There are two types of cache keys in BuildStream, ``strong`` and ``weak``. + +The purpose of a ``strong`` cache key is to capture the state of as many aspects +as possible that can have an influence on the build output. The aim is that +builds will be fully reproducible as long as the cache key doesn't change, +with suitable module build systems that don't embed timestamps, for example. + +A ``strong`` cache key includes the strong cache key of each build dependency +(and their runtime dependencies) of the element as changes in build dependencies +(or their runtime dependencies) can result in build differences in reverse +dependencies. This means that whenever the strong cache key of a dependency +changes, the strong cache key of its reverse dependencies will change as well. + +A ``weak`` cache key has an almost identical structure, however, it includes +only the names of build dependencies, not their cache keys or their runtime +dependencies. A weak cache key will thus still change when the element itself +or the environment changes but it will not change when a dependency is updated. + +For elements without build dependencies the ``strong`` cache key is identical +to the ``weak`` cache key. + +Strict build plan +----------------- +This is the default build plan that exclusively uses ``strong`` cache keys +for the core functionality. An element's cache key can be calculated when +the cache keys of the element's build dependencies (and their runtime +dependencies) have been calculated and either tracking is not enabled or it +has already completed for this element, i.e., the ``ref`` is available. +This means that with tracking disabled the cache keys of all elements could be +calculated right at the start of a build session. + +While BuildStream only uses ``strong`` cache keys with the strict build plan +for the actual staging and build process, it will still calculate ``weak`` +cache keys for each element. This allows BuildStream to store the artifact +in the cache with both keys, reducing rebuilds when switching between strict +and non-strict build plans. If the artifact cache already contains an +artifact with the same ``weak`` cache key, it's replaced. Thus, non-strict +builds always use the latest artifact available for a given ``weak`` cache key. + +Non-strict build plan +--------------------- +The non-strict build plan disables the time-consuming automatic rebuild of +reverse dependencies at the cost of dropping the reproducibility benefits. +It uses the ``weak`` cache keys for the core staging and build process. +I.e., if an artifact is available with the calculated ``weak`` cache key, +it will be reused for staging instead of being rebuilt. ``weak`` cache keys +can be calculated early in the build session. After tracking, similar to +when ``strong`` cache keys can be calculated with a strict build plan. + +Similar to how strict build plans also calculate ``weak`` cache keys, non-strict +build plans also calculate ``strong`` cache keys. However, this is slightly +more complex. To calculate the ``strong`` cache key of an element, BuildStream +requires the ``strong`` cache keys of the build dependencies (and their runtime +dependencies). + +The build dependencies of an element may have been updated since the artifact +was built. With the non-strict build plan the artifact will still be reused. +However, this means that we cannot use a ``strong`` cache key calculated purely +based on the element definitions. We need a cache key that matches the +environment at the time the artifact was built, not the current definitions. + +The only way to get the correct ``strong`` cache key is by retrieving it from +the metadata stored in the artifact. As artifacts may need to be pulled from a +remote artifact cache, the ``strong`` cache key is not readily available early +in the build session. However, it can always be retrieved when an element is +about to be built, as the dependencies are guaranteed to be in the local +artifact cache at that point. + +``Element._get_cache_key_from_artifact()`` extracts the ``strong`` cache key +from an artifact in the local cache. ``Element._get_cache_key_for_build()`` +calculates the ``strong`` cache key that is used for a particular build job. +This is used for the embedded metadata and also as key to store the artifact in +the cache. diff --git a/doc/source/additional_sandboxing.rst b/doc/source/additional_sandboxing.rst new file mode 100644 index 000000000..531a3e5d5 --- /dev/null +++ b/doc/source/additional_sandboxing.rst @@ -0,0 +1,200 @@ + +.. _sandboxing: + + +Sandboxing +========== + +Introduction +------------ + +BuildStream assembles each element in a *sandbox*. The sandbox is a container +environment which serves two purposes: giving BuildStream control over +all build aspects in order to ensure reproducibility of build results, +and providing safety guarantees for the host system that BuildStream is +running on. + +The exact implementation of the sandbox varies depending on which platform you +are running BuildStream. See below for backend-specific details. + +There are several factors that affect the build output and must therefore be +under BuildStream's control: + +* Filesystem contents and metadata +* The user and permissions model +* Network access +* Device access + +Each of these is detailed below. + +For safety reasons, BuildStream also controls the following things: + +* Access to files outside of the sandbox directory +* Access to certain kernel-specific syscalls + +Creating a sandbox can require special priviliges. This is a safety concern too +because bugs in the `bst` program can cause damage to a host if the program is +running with extra privileges. The exact priviliges that are required depend on +your platform and backend. + +Element plugins can run arbitary commands within the sandbox using the +:mod:`sandbox API `. + +What elements can and can't do in the sandbox +--------------------------------------------- + +This section specifies how BuildStream sandboxes are intended to work. A +specific sandbox provider may not necessarily be able to achieve all of the +requirements listed below so be sure to read the "platform notes" section as +well. + +Filesystem access +~~~~~~~~~~~~~~~~~ + +The filesystem inside sandboxes should be read only during element assembly, +except for certain directories which element plugins can mark as being +read/write. Most elements plugins derive from :mod:`BuildElement +`, which marks ``%{build-root}`` and +``%{install-root}`` as read/write. + +When running integration commands or `bst shell`, the sandbox should have a +fully read-write filesystem. The changes made here do not need to persist +beyond the lifetime of that sandbox, and **must not** affect the contents of +artifacts stored in the cache. + +Certain top level directories should be treated specially in all sandboxes: + +* The ``/dev`` directory should contain device nodes, which are described in + a separate section. + +* The ``/proc`` directory should have a UNIX 'procfs' style filesystem mounted. + It should not expose any information about processes running outside of the + sandbox. + +* The ``/tmp`` directory should be writable. + +Filesystem metadata +~~~~~~~~~~~~~~~~~~~ + +The writable areas inside a BuildStream sandbox are limited in what metadata +can be written and stored. + +* All files must be owned by UID 0 and GID 0 +* No files may have the setuid or setgid bits set +* Extended file attributes (xattrs) cannot be written to or read. +* Hardlinks to other files can be created, but the information about which + files are hardlinked to each other will not be stored in the artifact + that is created from the sandbox. + +These restrictions are due to technical limitations. In future we hope to +support a wider range of filesystem metadata operations. See `issue #38 +`_ for more details. + +User and permissions model +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All commands inside the sandbox run with user ID 0 and group ID 0. It should +not be possible to become any other user ID. + +Network access +~~~~~~~~~~~~~~ + +Builds should not be able to access the network at all from the sandbox. All +remote resources needed to build an element must be specified in the element's +``sources`` list so that BuildStream is able to see when they have changed. + +A sandbox opened by `bst shell` should allow network access. + +Device access +~~~~~~~~~~~~~ + +Builds should not be able to access any hardware devices at all. + +A few standard UNIX device files are needed, the whitelist is: + +* ``/dev/full`` +* ``/dev/null`` +* ``/dev/urandom`` +* ``/dev/random`` +* ``/dev/zero`` + +It may seem odd that we have sources of randomness in the sandbox, but a lot of +tools do expect them to exist. We take the view that it's up to integrators to +ensure that elements do not deliberately include randomness in their output. + +A sandbox opened by `bst shell` can make any devices available. There needs to +be a console device so that it can be used interactively. + +Platform notes +-------------- + +BuildStream currently only carries first-class support for modern Linux-based +operating systems. + +There is also a "fallback" backend which aims to make BuildStream usable on any +POSIX-compatible operating system. The POSIX standard does not provide good +support for creating containers so this implementation makes a number of +unfortunate compromises. + +Linux +~~~~~ + +On Linux we use the following isolation and sandboxing primitives: + +* bind mounts +* FUSE +* Mount namespaces +* Network namespaces +* PID (process ID) namespaces +* User namespaces (if available) +* seccomp + +We access all of these features through a sandboxing tool named `Bubblewrap +`_. + +User namespaces are not enabled by default in all Linux distributions. +BuildStream still runs on such systems but will give a big warning on startup +and will refuse to push any artifacts built on such a system to a remote cache. +For more information, see `issue #92 +`_. + +The Linux platform can operate as a standard user provided user namespace +support is available. If user namespace support is not available you have the +option of installing bubblewrap as a setuid binary to avoid needing to run the +entire ``bst`` process as the ``root`` user. + +The artifact cache on Linux systems is implemented using `OSTree +`_, which can allow us to stage artifacts +using hardlinks instead of copying them. To avoid cache corruption it is +vital that hardlinked files cannot be overwritten. In cases where the root +filesystem inside the sandbox needs to be writable, a custom FUSE filesystem +named SafeHardlinks is used which provides a copy-on-write layer. + +Some of the operations on filesystem metadata listed above are not prohibited +by the sandbox, but will instead be silently dropped when an artifact is +created. For more details see `issue #38 +`_. + +Some details of the host machine are currently leaked by this platform backend. +For more details, see `issue #262 +`_. + +Fallback (POSIX) +~~~~~~~~~~~~~~~~ + +The fallback backend aims to be usable on a wide range of operating systems. +Any OS that implements the POSIX specification and the ``chroot()`` syscall +can be expected to work. There are no real isolation or sandboxing primitives +that work across multiple operating systems, so the protection provided by +this backend is minimal. It would be much safer to use a platform-specific +backend. + +Filesystem isolation is done using the chroot() system call. This system call +requires special privileges to use so ``bst`` usually needs to be run as the +``root`` user when using this backend. + +Network access is not blocked in the sandbox. However since there is unlikely +to be a correct `/etc/resolv.conf` file, any network access that depends on +name resolution will most likely fail anyway. + +Builds inside the sandbox execute as the ``root`` user. diff --git a/doc/source/artifacts.rst b/doc/source/artifacts.rst deleted file mode 100644 index e0808e8fe..000000000 --- a/doc/source/artifacts.rst +++ /dev/null @@ -1,200 +0,0 @@ - - -.. _artifacts: - -Installing an artifact server -============================= -BuildStream caches the results of builds in a local artifact cache, and will -avoid building an element if there is a suitable build already present in the -local artifact cache. - -In addition to the local artifact cache, you can configure one or more remote -artifact caches and BuildStream will then try to pull a suitable build from one -of the remotes, falling back to a local build if needed. - -Configuring BuildStream to use remote caches --------------------------------------------- -A project will often set up continuous build infrastructure that pushes -built artifacts to a shared cache, so developers working on the project can -make use of these pre-built artifacts instead of having to each build the whole -project locally. The project can declare this cache in its -:ref:`project configuration file `. - -Users can declare additional remote caches in the :ref:`user configuration -`. There are several use cases for this: your project may not -define its own cache, it may be useful to have a local mirror of its cache, or -you may have a reason to share artifacts privately. - -Remote artifact caches are identified by their URL. There are currently three -supported protocols: - -* ``http``: Pull-only access, without transport-layer security -* ``https``: Pull-only access, with transport-layer security -* ``ssh``: Push access, authenticated via SSH - -BuildStream allows you to configure as many caches as you like, and will query -them in a specific order: - -1. Project-specific overrides in the user config -2. Project configuration -3. User configuration - -When an artifact is built locally, BuildStream will try to push it to all the -caches which have the ``push: true`` flag set. You can also manually push -artifacts to a specific cache using the :ref:`bst pull command `. - -Artifacts are identified using the element's :ref:`cache key ` so -the builds provided by a cache should be interchangable with those provided -by any other cache. - - -Setting up a remote artifact cache ----------------------------------- -The rest of this page outlines how to set up a shared artifact cache. - -Setting up the user -~~~~~~~~~~~~~~~~~~~ -A specific user is not needed for downloading artifacts, but since we -are going to use ssh to upload the artifacts, you will want a dedicated -user to own the artifact cache. - -.. code:: bash - - useradd artifacts - - -Installing the receiver -~~~~~~~~~~~~~~~~~~~~~~~ -You will also need to install BuildStream on the artifact server in order -to receive uploaded artifacts over ssh. Follow the instructions for installing -BuildStream :ref:`here ` - -When installing BuildStream on the artifact server, it must be installed -in a system wide location, with ``pip3 install .`` in the BuildStream -checkout directory. - -Otherwise, some tinkering is required to ensure BuildStream is available -in ``PATH`` when it's companion ``bst-artifact-receive`` program is run -remotely. - -You can install only the artifact receiver companion program without -requiring BuildStream's more exigent dependencies by setting the -``BST_ARTIFACTS_ONLY`` environment variable at install time, like so: - -.. code:: - - BST_ARTIFACTS_ONLY=1 pip3 install . - - -Initializing the cache -~~~~~~~~~~~~~~~~~~~~~~ -Now that you have a dedicated user to own the artifact cache, change -to that user, and create the artifact cache ostree repository directly -in it's home directory as such: - -.. code:: bash - - ostree init --mode archive-z2 --repo artifacts - -This should result in an artifact cache residing at the path ``/home/artifacts/artifacts`` - - -Serve the cache over https -~~~~~~~~~~~~~~~~~~~~~~~~~~ -This part should be pretty simple, you can do this with various technologies, all -we really require is that you make the artifacts available over https (you can use -http but until we figure out using gpg signed ostree commits for the artifacts, it's -better to serve over https). - -Here is an example, note that you must have a certificate **pem** file to use, as -is the case for hosting anything over https. - -.. code:: python - - import http.server, ssl, os - - # Maybe use a custom port, especially if you are serving - # other web pages on the same computer - server_address = ('localhost', 443) - artifact_path = '/home/artifacts' - - # The http server will serve from it's current - # working directory - os.chdir(artifact_path) - - # Create Server - httpd = http.server.HTTPServer( - server_address, - http.server.SimpleHTTPRequestHandler) - - # Add ssl - httpd.socket = ssl.wrap_socket(httpd.socket, - server_side=True, - certfile='localhost.pem', - ssl_version=ssl.PROTOCOL_TLSv1) - - # Run it - httpd.serve_forever() - - -Configure and run sshd -~~~~~~~~~~~~~~~~~~~~~~ -You will need to run the sshd service to allow uploading artifacts. - -For this you will want something like the following in your ``/etc/ssh/sshd_config`` - -.. code:: bash - - # Allow ssh logins/commands with the artifacts user - AllowUsers artifacts - - # Some specifics for the artifacts user - Match user artifacts - - # Dont allow password authentication for artifacts user - # - PasswordAuthentication no - - # Also lets dedicate this login for only running the - # bst-artifact-receive program, note that the full - # command must be specified here; 'artifacts' is - # the HOME relative path to the artifact cache. - # The exact pull URL must also be specified. - ForceCommand bst-artifact-receive --pull-url https://example.com/artifacts --verbose artifacts - - -User configuration -~~~~~~~~~~~~~~~~~~ -The user configuration for artifacts is documented with the rest -of the :ref:`user configuration documentation `. - -Assuming you have the same setup used in this document, and that your -host is reachable on the internet as ``artifacts.com`` (for example), -then a user can use the following user configuration: - -.. code:: yaml - - # - # Artifacts - # - artifacts: - - url: https://artifacts.com/artifacts - - # Alternative form if you have push access to the cache - #url: ssh://artifacts@artifacts.com:22200/artifacts - #push: true - - -Authenticating users -~~~~~~~~~~~~~~~~~~~~ -In order to give permission to a given user to upload -artifacts, simply use the regular ``ssh`` method. - -First obtain the user's public ssh key, and add it -to the authorized keys, like so: - -.. code:: bash - - cat user_id_rsa.pub >> /home/artifacts/.ssh/authorized_keys - diff --git a/doc/source/authoring.rst b/doc/source/authoring.rst deleted file mode 100644 index 4dca72b03..000000000 --- a/doc/source/authoring.rst +++ /dev/null @@ -1,79 +0,0 @@ - - -Authoring projects -================== -This section details how to use the BuildStream YAML format to -create your own project or modify existing projects. - - -.. toctree:: - :maxdepth: 2 - :caption: Project format - - formatintro - projectconf - format - public - projectrefs - - -Plugins -------- -Plugins provide their own individual plugin specific YAML configurations, -The element ``.bst`` files can specify plugin specific configuration in -the :ref:`config section `, while sources declared on a -given element specify their plugin specific configuration directly -:ref:`in their source declarations `. - - -General elements -~~~~~~~~~~~~~~~~ -.. toctree:: - :maxdepth: 1 - - elements/stack - elements/import - elements/compose - elements/script - elements/junction - elements/filter - - -Build elements -~~~~~~~~~~~~~~ -.. toctree:: - :maxdepth: 1 - - elements/manual - elements/make - elements/autotools - elements/cmake - elements/qmake - elements/distutils - elements/makemaker - elements/modulebuild - elements/meson - elements/pip - - -Sources -~~~~~~~ -.. toctree:: - :maxdepth: 1 - - sources/local - sources/tar - sources/zip - sources/git - sources/bzr - sources/ostree - sources/patch - sources/deb - - -External plugins ----------------- -External plugins need to be installed separately, here is -a list of BuildStream plugin projects known to us at this time: - -* `bst-external `_ diff --git a/doc/source/cachekeys.rst b/doc/source/cachekeys.rst deleted file mode 100644 index f0df796c5..000000000 --- a/doc/source/cachekeys.rst +++ /dev/null @@ -1,96 +0,0 @@ - -.. _cachekeys: - - -Cache keys -========== - -Cache keys for artifacts are generated from the inputs of the build process -for the purpose of reusing artifacts in a well-defined, predictable way. - -Structure ---------- -Cache keys are SHA256 hash values generated from a pickled Python dict that -includes: - -* Environment (e.g., project configuration and variables) -* Element configuration (details depend on element kind, ``Element.get_unique_key()``) -* Sources (``Source.get_unique_key()``) -* Dependencies (depending on cache key type, see below) -* Public data - -Cache key types ---------------- -There are two types of cache keys in BuildStream, ``strong`` and ``weak``. - -The purpose of a ``strong`` cache key is to capture the state of as many aspects -as possible that can have an influence on the build output. The aim is that -builds will be fully reproducible as long as the cache key doesn't change, -with suitable module build systems that don't embed timestamps, for example. - -A ``strong`` cache key includes the strong cache key of each build dependency -(and their runtime dependencies) of the element as changes in build dependencies -(or their runtime dependencies) can result in build differences in reverse -dependencies. This means that whenever the strong cache key of a dependency -changes, the strong cache key of its reverse dependencies will change as well. - -A ``weak`` cache key has an almost identical structure, however, it includes -only the names of build dependencies, not their cache keys or their runtime -dependencies. A weak cache key will thus still change when the element itself -or the environment changes but it will not change when a dependency is updated. - -For elements without build dependencies the ``strong`` cache key is identical -to the ``weak`` cache key. - -Strict build plan ------------------ -This is the default build plan that exclusively uses ``strong`` cache keys -for the core functionality. An element's cache key can be calculated when -the cache keys of the element's build dependencies (and their runtime -dependencies) have been calculated and either tracking is not enabled or it -has already completed for this element, i.e., the ``ref`` is available. -This means that with tracking disabled the cache keys of all elements could be -calculated right at the start of a build session. - -While BuildStream only uses ``strong`` cache keys with the strict build plan -for the actual staging and build process, it will still calculate ``weak`` -cache keys for each element. This allows BuildStream to store the artifact -in the cache with both keys, reducing rebuilds when switching between strict -and non-strict build plans. If the artifact cache already contains an -artifact with the same ``weak`` cache key, it's replaced. Thus, non-strict -builds always use the latest artifact available for a given ``weak`` cache key. - -Non-strict build plan ---------------------- -The non-strict build plan disables the time-consuming automatic rebuild of -reverse dependencies at the cost of dropping the reproducibility benefits. -It uses the ``weak`` cache keys for the core staging and build process. -I.e., if an artifact is available with the calculated ``weak`` cache key, -it will be reused for staging instead of being rebuilt. ``weak`` cache keys -can be calculated early in the build session. After tracking, similar to -when ``strong`` cache keys can be calculated with a strict build plan. - -Similar to how strict build plans also calculate ``weak`` cache keys, non-strict -build plans also calculate ``strong`` cache keys. However, this is slightly -more complex. To calculate the ``strong`` cache key of an element, BuildStream -requires the ``strong`` cache keys of the build dependencies (and their runtime -dependencies). - -The build dependencies of an element may have been updated since the artifact -was built. With the non-strict build plan the artifact will still be reused. -However, this means that we cannot use a ``strong`` cache key calculated purely -based on the element definitions. We need a cache key that matches the -environment at the time the artifact was built, not the current definitions. - -The only way to get the correct ``strong`` cache key is by retrieving it from -the metadata stored in the artifact. As artifacts may need to be pulled from a -remote artifact cache, the ``strong`` cache key is not readily available early -in the build session. However, it can always be retrieved when an element is -about to be built, as the dependencies are guaranteed to be in the local -artifact cache at that point. - -``Element._get_cache_key_from_artifact()`` extracts the ``strong`` cache key -from an artifact in the local cache. ``Element._get_cache_key_for_build()`` -calculates the ``strong`` cache key that is used for a particular build job. -This is used for the embedded metadata and also as key to store the artifact in -the cache. diff --git a/doc/source/commands.rst b/doc/source/commands.rst deleted file mode 100644 index 18affc6e5..000000000 --- a/doc/source/commands.rst +++ /dev/null @@ -1,127 +0,0 @@ - -.. _commands: - -Commands -======== -This page contains documentation for each BuildStream command, -along with their possible options and arguments. Each command can be -invoked on the command line, where, in most cases, this will be from the -project's main directory. - - ----- - -.. The bst options e.g. bst --version, or bst --verbose etc. -.. _invoking_bst: - -.. click:: buildstream._frontend:cli - :prog: bst - -.. Further description of the command goes here - ----- - -.. the `bst init` command -.. _invoking_init: - -.. click:: buildstream._frontend.cli:init - :prog: bst init - ----- - -.. the `bst build` command -.. _invoking_build: - -.. click:: buildstream._frontend.cli:build - :prog: bst build - ----- - -.. _invoking_fetch: - -.. click:: buildstream._frontend.cli:fetch - :prog: bst fetch - ----- - -.. _invoking_track: - -.. click:: buildstream._frontend.cli:track - :prog: bst track - ----- - -.. _invoking_pull: - -.. click:: buildstream._frontend.cli:pull - :prog: bst pull - ----- - -.. _invoking_push: - -.. click:: buildstream._frontend.cli:push - :prog: bst push - ----- - -.. _invoking_show: - -.. click:: buildstream._frontend.cli:show - :prog: bst show - ----- - -.. _invoking_shell: - -.. click:: buildstream._frontend.cli:shell - :prog: bst shell - ----- - -.. _invoking_checkout: - -.. click:: buildstream._frontend.cli:checkout - :prog: bst checkout - ----- - -.. _invoking_source_bundle: - -.. click:: buildstream._frontend.cli:source_bundle - :prog: bst source bundle - ----- - -.. _invoking_workspace: - -.. click:: buildstream._frontend.cli:workspace - :prog: bst workspace - ----- - -.. _invoking_workspace_open: - -.. click:: buildstream._frontend.cli:workspace_open - :prog: bst workspace open - ----- - -.. _invoking_workspace_close: - -.. click:: buildstream._frontend.cli:workspace_close - :prog: bst workspace close - ----- - -.. _invoking_workspace_reset: - -.. click:: buildstream._frontend.cli:workspace_reset - :prog: bst workspace reset - ----- - -.. _invoking_workspace_list: - -.. click:: buildstream._frontend.cli:workspace_list - :prog: bst workspace list diff --git a/doc/source/core_additional.rst b/doc/source/core_additional.rst index 10b74cbd2..76e523ee8 100644 --- a/doc/source/core_additional.rst +++ b/doc/source/core_additional.rst @@ -6,5 +6,5 @@ Additional writings .. toctree:: :maxdepth: 2 - cachekeys - sandboxing + additional_cachekeys + additional_sandboxing diff --git a/doc/source/core_format.rst b/doc/source/core_format.rst new file mode 100644 index 000000000..294867f27 --- /dev/null +++ b/doc/source/core_format.rst @@ -0,0 +1,17 @@ + + +Project format +============== +This section details how to use the BuildStream YAML format to +create your own project or modify existing projects. + + +.. toctree:: + :maxdepth: 2 + :caption: Project format + + format_intro + format_project + format_declaring + format_public + format_project_refs diff --git a/doc/source/core_framework.rst b/doc/source/core_framework.rst index 5f0db07ab..c3b84a9b1 100644 --- a/doc/source/core_framework.rst +++ b/doc/source/core_framework.rst @@ -2,8 +2,8 @@ .. _core_framework: -Core framework --------------- +Plugin API reference +==================== The core public APIs are of interest to anyone who wishes to implement custom :mod:`Element ` or :mod:`Source ` plugins, and can also be diff --git a/doc/source/core_plugins.rst b/doc/source/core_plugins.rst new file mode 100644 index 000000000..e038c2ef9 --- /dev/null +++ b/doc/source/core_plugins.rst @@ -0,0 +1,62 @@ + + +Plugin specific documentation +============================= +Plugins provide their own individual plugin specific YAML configurations, +The element ``.bst`` files can specify plugin specific configuration in +the :ref:`config section `, while sources declared on a +given element specify their plugin specific configuration directly +:ref:`in their source declarations `. + + +General elements +---------------- +.. toctree:: + :maxdepth: 1 + + elements/stack + elements/import + elements/compose + elements/script + elements/junction + elements/filter + + +Build elements +-------------- +.. toctree:: + :maxdepth: 1 + + elements/manual + elements/make + elements/autotools + elements/cmake + elements/qmake + elements/distutils + elements/makemaker + elements/modulebuild + elements/meson + elements/pip + + +Sources +------- +.. toctree:: + :maxdepth: 1 + + sources/local + sources/tar + sources/zip + sources/git + sources/bzr + sources/ostree + sources/patch + sources/deb + + +External plugins +---------------- +External plugins need to be installed separately, here is +a list of BuildStream plugin projects known to us at this time: + +* `bst-external `_ diff --git a/doc/source/docker.rst b/doc/source/docker.rst deleted file mode 100644 index 9b0f8aa73..000000000 --- a/doc/source/docker.rst +++ /dev/null @@ -1,45 +0,0 @@ - - -.. _docker: - -BuildStream inside Docker -------------------------- -If your system cannot provide the base system requirements for BuildStream, then it is possible to run buildstream within a Docker image. - -The BuildStream project provides -`Docker images `_ -containing BuildStream and its dependencies. -This gives you an easy way to get started using BuildStream on any Unix-like -platform where Docker is available, including Mac OS X. - -We recommend using the -`bst-here wrapper script `_ -which automates the necessary container setup. You can download it and make -it executable like this: - -.. code:: bash - - mkdir -p ~/.local/bin - curl --get https://gitlab.com/BuildStream/buildstream/raw/master/contrib/bst-here > ~/.local/bin/bst-here - chmod +x ~/.local/bin/bst-here - -Check if ``~/.local/bin`` appears in your PATH environment variable -- if it -doesn't, you should -`edit your ~/.profile so that it does `_. - -Once the script is available in your PATH, you can run ``bst-here`` to open a -shell session inside a new container based off the latest version of the -buildstream-fedora Docker image. The current working directory will be mounted -inside the container at ``/src``. - -You can also run individual BuildStream commands as ``bst-here COMMAND``. For -example: ``bst-here show systems/my-system.bst``. Note that BuildStream won't -be able to integrate with Bash tab-completion if you invoke it in this way. - -Two Docker volumes are set up by the ``bst-here`` script: - - * buildstream-cache -- mounted at ``~/.cache/buildstream`` - * buildstream-config -- mounted at ``~/.config/`` - -These are necessary so that your BuildStream cache and configuration files -persist between invocations of ``bst-here``. diff --git a/doc/source/examples.rst b/doc/source/examples.rst deleted file mode 100644 index aa100d007..000000000 --- a/doc/source/examples.rst +++ /dev/null @@ -1,12 +0,0 @@ - -Examples -======== -This page contains documentation for real examples of BuildStream projects, -described step by step. All run under CI, so you can trust they are -maintained and work as expected. - - -.. toctree:: - :maxdepth: 1 - - examples/flatpak-autotools diff --git a/doc/source/format.rst b/doc/source/format.rst deleted file mode 100644 index 2b9e0ea25..000000000 --- a/doc/source/format.rst +++ /dev/null @@ -1,418 +0,0 @@ - -Element constructs -================== - - -.. _format_basics: - -Element basics --------------- -Here is a rather complete example using the autotools element kind and git source kind: - -.. code:: yaml - - # Specify the kind of element this is - kind: autotools - - # Specify some dependencies - depends: - - element1.bst - - element2.bst - - # Specify the source which should be built - sources: - - kind: git - url: upstream:modulename.git - track: master - ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 - - # Override some variables - variables: - sysconfdir: %{prefix}/etc - - # Tweak the sandbox shell environment - environment: - LD_LIBRARY_PATH: /some/custom/path - - # Specify the configuration of the element - config: - - # Override autotools element default configure-commands - configure-commands: - - "%{configure} --enable-fancy-feature" - - # Specify public domain data, visible to other elements. - public: - bst: - integration-commands: - - /usr/bin/update-fancy-feature-cache - - # Specify a user id and group id to use in the build sandbox. - sandbox: - build-uid: 0 - build-gid: 0 - - -For most use cases you would not need to specify this much detail, we've provided -details here in order to have a more complete initial example. - -Let's break down the above and give a brief explanation of what these attributes mean. - - -Kind -~~~~ - -.. code:: yaml - - # Specify the kind of element this is - kind: autotools - -The ``kind`` attribute specifies which plugin will be operating on the element's input to -produce its output. Plugins define element types and each of them can be referred to by -name with the ``kind`` attribute. - -To refer to a third party plugin, prefix the plugin with its package, for example: - -.. code:: yaml - - kind: buildstream-plugins:dpkg_build - - -.. _format_depends: - -Depends -~~~~~~~ - -.. code:: yaml - - # Specify some dependencies - depends: - - element1.bst - - element2.bst - -Relationships between elements are specified with the ``depends`` attribute. Elements -may depend on other elements by specifying the :ref:`element path ` -relative filename to the elements they depend on here. - -See :ref:`format_dependencies` for more information on the dependency model. - - -.. _format_sources: - -Sources -~~~~~~~ - -.. code:: yaml - - # Specify the source which should be built - sources: - - kind: git - url: upstream:modulename.git - track: master - ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 - -Here we specify some input for the element, any number of sources may be specified. -By default the sources will be staged in the root of the element's build directory -in the build sandbox, but sources may specify a ``directory`` attribute to control -where the sources will be staged. The ``directory`` attribute may specify a build -sandbox relative subdirectory. - -For example, one might encounter a component which requires a separate data package -in order to build itself, in this case the sources might be listed as: - -.. code:: yaml - - sources: - - # Specify the source which should be built - - kind: git - url: upstream:modulename.git - track: master - ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 - - # Specify the data package we need for build frobnication, - # we need it to be unpacked in a src/frobdir - - kind: tarball - directory: src/frobdir - url: data:frobs.tgz - ref: 9d4b1147f8cf244b0002ba74bfb0b8dfb3... - -Like Elements, Source types are plugins which are indicated by the ``kind`` attribute. -Asides from the common ``kind`` and ``directory`` attributes which may be applied to all -Sources, refer to the Source specific documentation for meaningful attributes for the -particular Source. - - -Variables -~~~~~~~~~ - -.. code:: yaml - - # Override some variables - variables: - sysconfdir: "%{prefix}/etc" - -Variables can be declared or overridden from an element. Variables can also be -declared and overridden in the :ref:`projectconf` - -See :ref:`format_variables` below for a more in depth discussion on variables in BuildStream. - - -.. _format_environment: - -Environment -~~~~~~~~~~~ - -.. code:: yaml - - # Tweak the sandbox shell environment - environment: - LD_LIBRARY_PATH: /some/custom/path - -Environment variables can be set to literal values here, these environment -variables will be effective in the :mod:`Sandbox ` where -build instructions are run for this element. - - -Environment variables can also be declared and overridden in the :ref:`projectconf` - - -.. _format_config: - -Config -~~~~~~ - -.. code:: yaml - - # Specify the configuration of the element - config: - - # Override autotools element default configure-commands - configure-commands: - - "%{configure} --enable-fancy-feature" - -Here we configure the element itself. The autotools element provides sane defaults for -building sources which use autotools. Element default configurations can be overridden -in the ``project.conf`` file and additionally overridden in the declaration of an element. - -For meaningful documentation on what can be specified in the ``config`` section for a given -element ``kind``, refer to the element specific documentation. - - -.. _format_public: - -Public -~~~~~~ - -.. code:: yaml - - # Specify public domain data, visible to other elements. - public: - bst: - integration-commands: - - /usr/bin/update-fancy-feature-cache - -Metadata declared in the ``public`` section of an element is visible to -any other element which depends on the declaring element in a given pipeline. -BuildStream itself consumes public data from the ``bst`` domain. The ``integration-commands`` -demonstrated above for example, describe commands which should be run in an -environment where the given element is installed but before anything should be run. - -An element is allowed to read domain data from any element it depends on, and users -may specify additional domains to be understood and processed by their own element -plugins. - - -.. _format_sandbox: - -Sandbox -~~~~~~~ -Configuration for the build sandbox (other than :ref:`environment variables `) -can be placed in the ``sandbox`` configuration. At present, only the -UID and GID used by the user in the group can be specified. - -.. code:: yaml - - # Specify a user id and group id to use in the build sandbox. - sandbox: - build-uid: 1003 - build-gid: 1001 - -BuildStream normally uses uid 0 and gid 0 (root) to perform all -builds. However, the behaviour of certain tools depends on user id, -behaving differently when run as non-root. To support those builds, -you can supply a different uid or gid for the sandbox. Only -bwrap-style sandboxes support custom user IDs at the moment, and hence -this will only work on Linux host platforms. - -.. note:: - - The ``sandbox`` configuration is available since :ref:`format version 6 ` - - -.. _format_dependencies: - -Dependencies ------------- -The dependency model in BuildStream is simplified by treating software distribution -and software building as separate problem spaces. This is to say that one element -can only ever depend on another element but never on a subset of the product which -another element produces. - -In this section we'll quickly go over the few features BuildStream offers in its -dependency model. - - -Expressing dependencies -~~~~~~~~~~~~~~~~~~~~~~~ -Dependencies in BuildStream are parameterizable objects, however as demonstrated -in the :ref:`above example `, they can also be expressed as simple -strings as a convenience shorthand in most cases, whenever the default dependency -attributes are suitable. - -.. note:: - - Note the order in which element dependencies are declared in the ``depends`` - list is not meaningful. - -Dependency dictionary: - -.. code:: yaml - - # Fully specified dependency - depends: - - filename: foo.bst - type: build - junction: baseproject.bst - -Attributes: - -* ``filename`` - - The :ref:`element path ` relative filename of the element to - depend on in the project. - -* ``type`` - - This attribute is used to express the :ref:`dependency type `. - -* ``junction`` - - This attribute can be used to depend on elements in other projects. - - If a junction is specified, then it must be an :ref:`element path ` - relative filename of the junction element in the project. - - In the case that a *junction* is specified, the ``filename`` attribute indicates an element - in the *junctioned project*. - - See :mod:`junction `. - - .. note:: - - The ``junction`` attribute is available since :ref:`format version 1 ` - - -.. _format_dependencies_types: - -Dependency types -~~~~~~~~~~~~~~~~ -The dependency ``type`` attribute defines what the dependency is required for -and is essential to how BuildStream plots a build plan. - -There are two types which one can specify for a dependency: - -* ``build`` - - A ``build`` dependency type states that the given element's product must - be staged in order to build the depending element. Depending on an element - which has ``build`` dependencies will not implicitly depend on that element's - ``build`` dependencies. - -* ``runtime`` - - A ``runtime`` dependency type states that the given element's product - must be present for the depending element to function. An element's - ``runtime`` dependencies need not be staged in order to build the element. - -If ``type`` is not specified, then it is assumed that the dependency is -required both at build time and runtime. - -.. note:: - - It is assumed that a dependency which is required for building an - element must run while building the depending element. This means that - ``build`` depending on a given element implies that that element's - ``runtime`` dependencies will also be staged for the purpose of building. - - -.. _format_variables: - -Using variables ---------------- -Variables in BuildStream are a way to make your build instructions and -element configurations more dynamic. - - -Referring to variables -~~~~~~~~~~~~~~~~~~~~~~ -Variables are expressed as ``%{...}``, where ``...`` must contain only -alphanumeric characters and the separators ``_`` and ``-``. Further, the -first letter of ``...`` must be an alphabetic character. - -.. code:: yaml - - This is release version %{version} - - -Declaring and overriding variables -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To declare or override a variable, one need only specify a value -in the relevant *variables* section: - -.. code:: yaml - - variables: - hello: Hello World - -You can refer to another variable while declaring a variable: - -.. code:: yaml - - variables: - release-text: This is release version %{version} - -The order in which you declare variables is arbitrary, so long as there is no cyclic -dependency and that all referenced variables are declared, the following is fine: - -.. code:: yaml - - variables: - release-text: This is release version %{version} - version: 5.5 - -.. note:: - - It should be noted that variable resolution only happens after all - :ref:`Element Composition ` has already taken place. - - This is to say that overriding ``%{version}`` at a higher priority will effect - the final result of ``%{release-text}``. - - -**Example:** - -.. code:: yaml - - kind: autotools - - # Declare variable, expect %{version} was already declared - variables: - release-text: This is release version %{version} - - config: - - # Customize the installation - install-commands: - - | - %{make-install} RELEASE_TEXT="%{release-text}" diff --git a/doc/source/format_declaring.rst b/doc/source/format_declaring.rst new file mode 100644 index 000000000..39c099dcf --- /dev/null +++ b/doc/source/format_declaring.rst @@ -0,0 +1,419 @@ + + +Declaring elements +================== + + +.. _format_basics: + +Element basics +-------------- +Here is a rather complete example using the autotools element kind and git source kind: + +.. code:: yaml + + # Specify the kind of element this is + kind: autotools + + # Specify some dependencies + depends: + - element1.bst + - element2.bst + + # Specify the source which should be built + sources: + - kind: git + url: upstream:modulename.git + track: master + ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 + + # Override some variables + variables: + sysconfdir: %{prefix}/etc + + # Tweak the sandbox shell environment + environment: + LD_LIBRARY_PATH: /some/custom/path + + # Specify the configuration of the element + config: + + # Override autotools element default configure-commands + configure-commands: + - "%{configure} --enable-fancy-feature" + + # Specify public domain data, visible to other elements. + public: + bst: + integration-commands: + - /usr/bin/update-fancy-feature-cache + + # Specify a user id and group id to use in the build sandbox. + sandbox: + build-uid: 0 + build-gid: 0 + + +For most use cases you would not need to specify this much detail, we've provided +details here in order to have a more complete initial example. + +Let's break down the above and give a brief explanation of what these attributes mean. + + +Kind +~~~~ + +.. code:: yaml + + # Specify the kind of element this is + kind: autotools + +The ``kind`` attribute specifies which plugin will be operating on the element's input to +produce its output. Plugins define element types and each of them can be referred to by +name with the ``kind`` attribute. + +To refer to a third party plugin, prefix the plugin with its package, for example: + +.. code:: yaml + + kind: buildstream-plugins:dpkg_build + + +.. _format_depends: + +Depends +~~~~~~~ + +.. code:: yaml + + # Specify some dependencies + depends: + - element1.bst + - element2.bst + +Relationships between elements are specified with the ``depends`` attribute. Elements +may depend on other elements by specifying the :ref:`element path ` +relative filename to the elements they depend on here. + +See :ref:`format_dependencies` for more information on the dependency model. + + +.. _format_sources: + +Sources +~~~~~~~ + +.. code:: yaml + + # Specify the source which should be built + sources: + - kind: git + url: upstream:modulename.git + track: master + ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 + +Here we specify some input for the element, any number of sources may be specified. +By default the sources will be staged in the root of the element's build directory +in the build sandbox, but sources may specify a ``directory`` attribute to control +where the sources will be staged. The ``directory`` attribute may specify a build +sandbox relative subdirectory. + +For example, one might encounter a component which requires a separate data package +in order to build itself, in this case the sources might be listed as: + +.. code:: yaml + + sources: + + # Specify the source which should be built + - kind: git + url: upstream:modulename.git + track: master + ref: d0b38561afb8122a3fc6bafc5a733ec502fcaed6 + + # Specify the data package we need for build frobnication, + # we need it to be unpacked in a src/frobdir + - kind: tarball + directory: src/frobdir + url: data:frobs.tgz + ref: 9d4b1147f8cf244b0002ba74bfb0b8dfb3... + +Like Elements, Source types are plugins which are indicated by the ``kind`` attribute. +Asides from the common ``kind`` and ``directory`` attributes which may be applied to all +Sources, refer to the Source specific documentation for meaningful attributes for the +particular Source. + + +Variables +~~~~~~~~~ + +.. code:: yaml + + # Override some variables + variables: + sysconfdir: "%{prefix}/etc" + +Variables can be declared or overridden from an element. Variables can also be +declared and overridden in the :ref:`projectconf` + +See :ref:`format_variables` below for a more in depth discussion on variables in BuildStream. + + +.. _format_environment: + +Environment +~~~~~~~~~~~ + +.. code:: yaml + + # Tweak the sandbox shell environment + environment: + LD_LIBRARY_PATH: /some/custom/path + +Environment variables can be set to literal values here, these environment +variables will be effective in the :mod:`Sandbox ` where +build instructions are run for this element. + + +Environment variables can also be declared and overridden in the :ref:`projectconf` + + +.. _format_config: + +Config +~~~~~~ + +.. code:: yaml + + # Specify the configuration of the element + config: + + # Override autotools element default configure-commands + configure-commands: + - "%{configure} --enable-fancy-feature" + +Here we configure the element itself. The autotools element provides sane defaults for +building sources which use autotools. Element default configurations can be overridden +in the ``project.conf`` file and additionally overridden in the declaration of an element. + +For meaningful documentation on what can be specified in the ``config`` section for a given +element ``kind``, refer to the element specific documentation. + + +.. _format_public: + +Public +~~~~~~ + +.. code:: yaml + + # Specify public domain data, visible to other elements. + public: + bst: + integration-commands: + - /usr/bin/update-fancy-feature-cache + +Metadata declared in the ``public`` section of an element is visible to +any other element which depends on the declaring element in a given pipeline. +BuildStream itself consumes public data from the ``bst`` domain. The ``integration-commands`` +demonstrated above for example, describe commands which should be run in an +environment where the given element is installed but before anything should be run. + +An element is allowed to read domain data from any element it depends on, and users +may specify additional domains to be understood and processed by their own element +plugins. + + +.. _format_sandbox: + +Sandbox +~~~~~~~ +Configuration for the build sandbox (other than :ref:`environment variables `) +can be placed in the ``sandbox`` configuration. At present, only the +UID and GID used by the user in the group can be specified. + +.. code:: yaml + + # Specify a user id and group id to use in the build sandbox. + sandbox: + build-uid: 1003 + build-gid: 1001 + +BuildStream normally uses uid 0 and gid 0 (root) to perform all +builds. However, the behaviour of certain tools depends on user id, +behaving differently when run as non-root. To support those builds, +you can supply a different uid or gid for the sandbox. Only +bwrap-style sandboxes support custom user IDs at the moment, and hence +this will only work on Linux host platforms. + +.. note:: + + The ``sandbox`` configuration is available since :ref:`format version 6 ` + + +.. _format_dependencies: + +Dependencies +------------ +The dependency model in BuildStream is simplified by treating software distribution +and software building as separate problem spaces. This is to say that one element +can only ever depend on another element but never on a subset of the product which +another element produces. + +In this section we'll quickly go over the few features BuildStream offers in its +dependency model. + + +Expressing dependencies +~~~~~~~~~~~~~~~~~~~~~~~ +Dependencies in BuildStream are parameterizable objects, however as demonstrated +in the :ref:`above example `, they can also be expressed as simple +strings as a convenience shorthand in most cases, whenever the default dependency +attributes are suitable. + +.. note:: + + Note the order in which element dependencies are declared in the ``depends`` + list is not meaningful. + +Dependency dictionary: + +.. code:: yaml + + # Fully specified dependency + depends: + - filename: foo.bst + type: build + junction: baseproject.bst + +Attributes: + +* ``filename`` + + The :ref:`element path ` relative filename of the element to + depend on in the project. + +* ``type`` + + This attribute is used to express the :ref:`dependency type `. + +* ``junction`` + + This attribute can be used to depend on elements in other projects. + + If a junction is specified, then it must be an :ref:`element path ` + relative filename of the junction element in the project. + + In the case that a *junction* is specified, the ``filename`` attribute indicates an element + in the *junctioned project*. + + See :mod:`junction `. + + .. note:: + + The ``junction`` attribute is available since :ref:`format version 1 ` + + +.. _format_dependencies_types: + +Dependency types +~~~~~~~~~~~~~~~~ +The dependency ``type`` attribute defines what the dependency is required for +and is essential to how BuildStream plots a build plan. + +There are two types which one can specify for a dependency: + +* ``build`` + + A ``build`` dependency type states that the given element's product must + be staged in order to build the depending element. Depending on an element + which has ``build`` dependencies will not implicitly depend on that element's + ``build`` dependencies. + +* ``runtime`` + + A ``runtime`` dependency type states that the given element's product + must be present for the depending element to function. An element's + ``runtime`` dependencies need not be staged in order to build the element. + +If ``type`` is not specified, then it is assumed that the dependency is +required both at build time and runtime. + +.. note:: + + It is assumed that a dependency which is required for building an + element must run while building the depending element. This means that + ``build`` depending on a given element implies that that element's + ``runtime`` dependencies will also be staged for the purpose of building. + + +.. _format_variables: + +Using variables +--------------- +Variables in BuildStream are a way to make your build instructions and +element configurations more dynamic. + + +Referring to variables +~~~~~~~~~~~~~~~~~~~~~~ +Variables are expressed as ``%{...}``, where ``...`` must contain only +alphanumeric characters and the separators ``_`` and ``-``. Further, the +first letter of ``...`` must be an alphabetic character. + +.. code:: yaml + + This is release version %{version} + + +Declaring and overriding variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To declare or override a variable, one need only specify a value +in the relevant *variables* section: + +.. code:: yaml + + variables: + hello: Hello World + +You can refer to another variable while declaring a variable: + +.. code:: yaml + + variables: + release-text: This is release version %{version} + +The order in which you declare variables is arbitrary, so long as there is no cyclic +dependency and that all referenced variables are declared, the following is fine: + +.. code:: yaml + + variables: + release-text: This is release version %{version} + version: 5.5 + +.. note:: + + It should be noted that variable resolution only happens after all + :ref:`Element Composition ` has already taken place. + + This is to say that overriding ``%{version}`` at a higher priority will effect + the final result of ``%{release-text}``. + + +**Example:** + +.. code:: yaml + + kind: autotools + + # Declare variable, expect %{version} was already declared + variables: + release-text: This is release version %{version} + + config: + + # Customize the installation + install-commands: + - | + %{make-install} RELEASE_TEXT="%{release-text}" diff --git a/doc/source/format_intro.rst b/doc/source/format_intro.rst new file mode 100644 index 000000000..b1780f9dc --- /dev/null +++ b/doc/source/format_intro.rst @@ -0,0 +1,291 @@ + + +Introduction +============ +At the core of BuildStream is a data model of :mod:`Elements ` which +are parsed from ``.bst`` files in a project directory and configured from a few different +sources. + +When BuildStream loads your project, various levels of composition occur, allowing +configuration on various levels with different priority. + +This page provides an introduction to the project directory structure, explains the +basic *directives* supported inherently throughout the format, and outlines how composition +occurs and what configurations are considered in which order. + +The meaning of the various constructs expressed in the BuildStream format are covered +in other sections of the documentation. + +.. _format_structure: + + +Directory structure +------------------- +A BuildStream project is a directory consisting of: + +* A project configuration file +* BuildStream element files +* Optional user defined plugins +* An optional project.refs file + +A typical project structure may look like this:: + + myproject/project.conf + myproject/project.refs + myproject/elements/element1.bst + myproject/elements/element2.bst + myproject/elements/... + myproject/plugins/customelement.py + myproject/plugins/customelement.yaml + myproject/plugins/... + + +Except for the project configuration file, the user is allowed to structure +their project directory in any way. For documentation on the format of the project +configuration file, refer to the :ref:`projectconf` documentation. + +Simpler projects may choose to place all element definition files at the +root of the project directory while more complex projects may decide to +put stacks in one directory and other floating elements into other directories, +perhaps placing deployment elements in another directory, this is all fine. + +The important part to remember is that when you declare dependency relationships, +a project relative path to the element one depends on must be provided. + + +.. _format_composition: + +Composition +----------- +Below are the various sources of configuration which go into an element or source in the +order in which they are applied. Configurations which are applied later have a higher +priority and override configurations which precede them. + + +1. Builtin defaults +~~~~~~~~~~~~~~~~~~~ +The :ref:`builtin defaults ` provide a set of builtin +default default values for ``project.conf``. + +The project wide defaults defined in the builtin project configuration, such as the +*variables* or *environment* sections, form the base configuration of all elements. + + +2. Project configuration +~~~~~~~~~~~~~~~~~~~~~~~~ +The :ref:`project wide defaults ` specified in your +``project.conf`` are now applied on top of builtin defaults. + +Defaults such as the :ref:`variables ` or +:ref:`environment ` which are specified in +your ``project.conf`` override the builtin defaults for elements. + +Note that :ref:`plugin type specific configuration ` +in ``project.conf`` is not applied until later. + + +3. Plugin defaults +~~~~~~~~~~~~~~~~~~ +Elements and Sources are all implemented as plugins. + +Each Element plugin installs a ``.yaml`` file along side their plugin to +define the default *variables*, *environment* and *config*. The *config* +is element specific and as such this is the first place where defaults +can be set on the *config* section. + +The *variables* and *environment* specified in the declaring plugin's +defaults here override the project configuration defaults for the given +element ``kind``. + +Source plugins do not have a ``.yaml`` file, and do not have *variables* or +*environment*. + + +4. Project configuration overrides +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The ``project.conf`` now gives you :ref:`another opportunity ` to +override configuration on a per plugin basis. + +Configurations specified in the :ref:`elements ` or +:ref:`sources ` sections of the ``project.conf`` +will override the given plugin's defaults. + +In this phase, it is possible to override any configurations of a given plugin, +including configuration in element specific *config* sections. + +See also :ref:`project_overrides` + + +5. Plugin declarations +~~~~~~~~~~~~~~~~~~~~~~~ +Finally, after having resolved any :ref:`conditionals ` +in the parsing phase of loading element declarations; the configurations specified in a +``.bst`` file have the last word on any configuration in the data model. + + +.. _format_directives: + +Directives +---------- + +.. _format_directives_conditional: + +(?) Conditionals +~~~~~~~~~~~~~~~~ +The ``(?)`` directive allows expression of conditional statements which +test :ref:`project option ` values. + +The ``(?)`` directive may appear as a key in any dictionary expressed +in YAML, and its value is a list of conditional expressions. Each conditional +expression must be a single key dictionary, where the key is the conditional +expression itself, and the value is a dictionary to be composited into the +parent dictionary containing the ``(?)`` directive if the expression evaluates +to a truthy value. + +**Example:** + +.. code:: yaml + + variables: + prefix: "/usr" + enable-debug: False + (?): + - relocate == True: + prefix: "/opt" + - debug == True: + enable-debug: True + + +Expressions are evaluated in the specified order, and each time an expression +evaluates to a truthy value, its value will be composited to the parent dictionary +in advance of processing other elements, allowing for logically overriding previous +decisions in the condition list. + +Nesting of conditional statements is also supported. + +**Example:** + +.. code:: yaml + + variables: + enable-logging: False + enable-debug: False + (?): + - logging == True: + enable-logging: True + (?): + - debugging == True: + enable-debug: True + + +Conditionals are expressed in a pythonic syntax, the specifics for +testing the individually supported option types are described in +their :ref:`respective documentation `. + +Compound conditionals are also allowed. + +**Example:** + +.. code:: yaml + + variables: + enable-debug: False + (?): + - (logging == True and debugging == True): + enable-debug: True + + +.. _format_directives_assertion: + +(!) Assertions +~~~~~~~~~~~~~~ +Assertions allow the project author to abort processing and present +a custom error message to the user building their project. + +This is only useful when used with conditionals, allowing the project +author to assert some invalid configurations. + + +**Example:** + +.. code:: yaml + + variables: + (?): + - (logging == False and debugging == True): + + (!): | + + Impossible to print any debugging information when + logging is disabled. + + +.. _format_directives_list_prepend: + +(<) List Prepend +~~~~~~~~~~~~~~~~ +Indicates that the list should be prepended to the target list, +instead of the default behavior which is to replace the target list. + +**Example:** + +.. code:: yaml + + config: + configure-commands: + # Before configuring, lets make sure we're using + # the latest config.sub & config.guess + (<): + - cp %{datadir}/automake-*/config.{sub,guess} . + + +.. _format_directives_list_append: + +(>) List Append +~~~~~~~~~~~~~~~ +Indicates that the list should be appended to the target list, instead +of the default behavior which is to replace the target list. + +**Example:** + +.. code:: yaml + + public: + bst: + split-rules: + devel: + # This element also adds some extra stubs which + # need to be included in the devel domain + (>): + - "%{libdir}/*.stub" + + +.. _format_directives_list_overwrite: + +(=) List Overwrite +~~~~~~~~~~~~~~~~~~ +Indicates that the list should be overwritten completely. + +This exists mostly for completeness, and we recommend using literal +lists most of the time instead of list overwrite directives when the +intent is to overwrite a list. + +This has the same behavior as a literal list, except that an +error will be triggered in the case that there is no underlying +list to overwrite; whereas a literal list will simply create a new +list. + +The added error protection can be useful when intentionally +overwriting a list in an element's *public data*, which is mostly +free form and not validated. + + +**Example:** + +.. code:: yaml + + config: + install-commands: + # This element's `make install` is broken, replace it. + (=): + - cp src/program %{bindir} diff --git a/doc/source/format_project.rst b/doc/source/format_project.rst new file mode 100644 index 000000000..6174ee728 --- /dev/null +++ b/doc/source/format_project.rst @@ -0,0 +1,811 @@ + +.. _projectconf: + + +Project configuration +===================== +The project configuration file should be named ``project.conf`` and +be located at the project root. It holds information such as Source +aliases relevant for the sources used in the given project as well as +overrides for the configuration of element types used in the project. + +Values specified in the project configuration override any of the +default BuildStream project configuration, which is included +:ref:`here ` for reference. + + +.. _project_essentials: + +Essentials +---------- + + +.. _project_format_name: + +Project name +~~~~~~~~~~~~ +The project name is a unique symbol for your project and will +be used to distinguish your project from others in user preferences, +namspaceing of your project's artifacts in shared artifact caches, +and in any case where BuildStream needs to distinguish between multiple +projects. + +The first thing to setup in your ``project.conf`` should be the name +of your project. + +.. code:: yaml + + name: my-project-name + +.. note:: + + The project name may contain alphanumeric characters, dashes and + underscores, and may not start with a leading digit. + + +.. _project_format_version: + +Format version +~~~~~~~~~~~~~~ +The BuildStream format is guaranteed to be backwards compatible +with any earlier releases. The project's minimum required format +version of BuildStream can be specified in ``project.conf`` with +the ``format-version`` field, e.g.: + +.. code:: yaml + + # The minimum base BuildStream format + format-version: 0 + +BuildStream will increment it's core YAML format version at least once +in any given minor point release where the format has been extended +to support a new feature. + +.. note:: + + External :mod:`Element ` and :mod:`Source ` + plugins also implement their own YAML configuration fragments and as + such are revisioned separately from the core format. See :ref:`project_plugins` + for details on specifying a minimum version of a specific plugin. + + Core :mod:`Elements ` and :mod:`Sources ` + which are maintained and distributed as a part of BuildStream are revisioned + under the same global ``format-version`` described here. + + +.. _project_element_path: + +Element path +~~~~~~~~~~~~ +To allow the user to structure their project nicely, BuildStream +allows the user to specify a project subdirectory where element +``.bst`` files are stored. + +.. code:: yaml + + element-path: elements + +Note that elements are referred to by their relative paths, whenever +elements are referred to in a ``.bst`` file or on the command line. + + +.. _project_format_ref_storage: + +Ref storage +~~~~~~~~~~~ +By default, BuildStream expects to read and write source references +directly in the :ref:`source declaration `, but this +can be inconvenient and prohibitive in some workflows. + +Alternatively, BuildStream allows source references to be stored +centrally in a :ref:`project.refs file ` in the toplevel +:ref:`project directory `. + +This can be controlled with the ``ref-storage`` option, which is +allowed to be configured with the following values: + +* ``inline`` + + Source references are stored directly in the + :ref:`source declaration ` + +* ``project.refs`` + + Source references are stored in the ``project.refs`` file, and + junction source references are stored in the ``junction.refs`` file. + +To enable storing of source references in ``project.refs``, add the +following to your ``project.conf``: + +.. code:: yaml + + ref-storage: project.refs + +.. note:: + + The ``ref-storage`` configuration is available since :ref:`format version 8 ` + + +Fail on overlaps +~~~~~~~~~~~~~~~~ +When multiple elements are staged, there's a possibility that different +elements will try and stage different versions of the same file. + +When ``fail-on-overlap`` is true, if an overlap is detected +that hasn't been allowed by the element's +:ref:`overlap whitelist`, +then an error will be raised and the build will fail. + +otherwise, a warning will be raised indicating which files had overlaps, +and the order that the elements were overlapped. + +.. code:: yaml + + fail-on-overlap: true + + +.. _project_source_aliases: + +Source aliases +~~~~~~~~~~~~~~ +In order to abstract the download location of source code and +any assets which need to be downloaded, and also as a matter of +convenience, BuildStream allows one to create named aliases for +URLs which are to be used in the individual ``.bst`` files. + +.. code:: yaml + + aliases: + foo: git://git.foo.org/ + bar: http://bar.com/downloads/ + + +Sandbox options +~~~~~~~~~~~~~~~ +Sandbox options for the whole project can be supplied in +``project.conf`` in the same way as in an element. See :ref:`element configuration ` +for more detail. + +.. code:: yaml + + # Specify a user id and group id to use in the build sandbox. + sandbox: + build-uid: 1003 + build-gid: 1001 + +.. note:: + + The ``sandbox`` configuration is available since :ref:`format version 6 ` + + +.. _project_essentials_artifacts: + +Artifact server +~~~~~~~~~~~~~~~ +If you have setup an :ref:`artifact server ` for your +project then it is convenient to configure this in your ``project.conf`` +so that users need not have any additional configuration to communicate +with an artifact share. + +.. code:: yaml + + artifacts: + + # A url from which to download prebuilt artifacts + url: https://foo.com/artifacts + +You can also specify a list of caches here; earlier entries in the list +will have higher priority than later ones. + + +.. _project_plugins: + +External plugins +---------------- +If your project makes use of any custom :mod:`Element ` or +:mod:`Source ` plugins, then the project must inform BuildStream +of the plugins it means to make use of and the origin from which they can be loaded. + +Note that plugins with the same name from different origins are not permitted. + + +Local plugins +~~~~~~~~~~~~~ +Local plugins are expected to be found in a subdirectory of the actual +BuildStream project. :mod:`Element ` and +:mod:`Source ` plugins should be stored in separate +directories to avoid namespace collisions. + +The versions of local plugins are largely immaterial since they are +revisioned along with the project by the user, usually in a VCS like git. +However, for the sake of consistency with other plugin loading origins +we require that you specify a version, this can always be ``0`` for a local +plugin. + + +.. code:: yaml + + plugins: + + - origin: local + path: plugins/sources + + # We want to use the `mysource` source plugin located in our + # project's `plugins/sources` subdirectory. + sources: + mysource: 0 + + +Pip plugins +~~~~~~~~~~~ +Plugins loaded from the ``pip`` origin are expected to be installed +separately on the host operating system using python's package management +system. + +.. code:: yaml + + plugins: + + - origin: pip + + # Specify the name of the python package containing + # the plugins we want to load. The name one would use + # on the `pip install` command line. + # + package-name: potato + + # We again must specify a minimal format version for the + # external plugin, it is allowed to be `0`. + # + elements: + potato: 0 + + +.. _project_options: + +Options +------- +Options are how BuildStream projects can define parameters which +can be configured by users invoking BuildStream to build your project. + +Options are declared in the ``project.conf`` in the main ``options`` +dictionary. + +.. code:: yaml + + options: + debug: + type: bool + description: Whether to enable debugging + default: False + +Users can configure those options when invoking BuildStream with the +``--option`` argument:: + + $ bst --option debug True ... + +.. note:: + + The name of the option may contain alphanumeric characters + underscores, and may not start with a leading digit. + + +Common properties +~~~~~~~~~~~~~~~~~ +All option types accept the following common attributes + +* ``type`` + + Indicates the type of option to declare + +* ``description`` + + A description of the meaning of the option + +* ``variable`` + + Optionally indicate a :ref:`variable ` name to + export the option to. A string form of the selected option will + be used to set the exported value. + + If used, this value will override any existing value for the + variable declared in ``project.conf``, and will be overridden in + the regular :ref:`composition order `. + + .. note:: + + The name of the variable to export may contain alphanumeric + characters, dashes, underscores, and may not start with a leading + digit. + + +.. _project_options_bool: + +Boolean +~~~~~~~ +The ``bool`` option type allows specifying boolean values which +can be cased in conditional expressions. + + +**Declaring** + +.. code:: yaml + + options: + debug: + type: bool + description: Whether to enable debugging + default: False + + +**Evaluating** + +Boolean options can be tested in expressions with equality tests: + +.. code:: yaml + + variables: + enable-debug: False + (?): + - debug == True: + enable-debug: True + +Or simply treated as truthy values: + +.. code:: yaml + + variables: + enable-debug: False + (?): + - debug: + enable-debug: True + + +**Exporting** + +When exporting boolean options as variables, a ``True`` option value +will be exported as ``1`` and a ``False`` option as ``0`` + + +.. _project_options_enum: + +Enumeration +~~~~~~~~~~~ +The ``enum`` option type allows specifying a string value +with a restricted set of possible values. + + +**Declaring** + +.. code:: yaml + + options: + loglevel: + type: enum + description: The logging level + values: + - debug + - info + - warning + default: info + + +**Evaluating** + +Enumeration options must be tested as strings in conditional +expressions: + +.. code:: yaml + + variables: + enable-debug: False + (?): + - loglevel == "debug": + enable-debug: True + + +**Exporting** + +When exporting enumeration options as variables, the value is +exported as a variable directly, as it is a simple string. + + +.. _project_options_flags: + +Flags +~~~~~ +The ``flags`` option type allows specifying a list of string +values with a restricted set of possible values. + +In contrast with the ``enum`` option type, the *default* value +need not be specified and will default to an empty set. + + +**Declaring** + +.. code:: yaml + + options: + logmask: + type: flags + description: The logging mask + values: + - debug + - info + - warning + default: + - info + + +**Evaluating** + +Options of type ``flags`` can be tested in conditional expressions using +a pythonic *in* syntax to test if an element is present in a set: + +.. code:: yaml + + variables: + enable-debug: False + (?): + - ("debug" in logmask): + enable-debug: True + + +**Exporting** + +When exporting flags options as variables, the value is +exported as a comma separated list of selected value strings. + + +.. _project_options_arch: + +Architecture +~~~~~~~~~~~~ +The ``arch`` option type is special enumeration option which +defaults to the result of `uname -m`, and does not support +assigning any default in the project configuration. + +.. code:: yaml + + options: + machine_arch: + type: arch + description: The machine architecture + values: + - arm + - aarch64 + - i386 + - x86_64 + + +Architecture options can be tested with the same expressions +as other Enumeration options. + + +.. _project_options_element_mask: + +Element mask +~~~~~~~~~~~~ +The ``element-mask`` option type is a special Flags option +which automatically allows only element names as values. + +.. code:: yaml + + options: + debug_elements: + type: element-mask + description: The elements to build in debug mode + +This can be convenient for automatically declaring an option +which might apply to any element, and can be tested with the +same syntax as other Flag options. + + +.. code:: yaml + + variables: + enable-debug: False + (?): + - ("element.bst" in debug_elements): + enable-debug: True + + + +.. _project_defaults: + +Element default configuration +----------------------------- +The ``project.conf`` plays a role in defining elements by +providing default values and also by overriding values declared +by plugins on a plugin wide basis. + +See the :ref:`composition ` documentation for +more detail on how elements are composed. + + +.. _project_defaults_variables: + +Variables +~~~~~~~~~ +The defaults for :ref:`Variables ` used in your +project is defined here. + +.. code:: yaml + + variables: + prefix: "/usr" + + +.. _project_defaults_environment: + +Environment +~~~~~~~~~~~ +The defaults environment for the build sandbox is defined here. + +.. code:: yaml + + environment: + PATH: /usr/bin:/bin:/usr/sbin:/sbin + +Additionally, the special ``environment-nocache`` list which specifies +which environment variables do not effect build output, and are thus +not considered in the calculation of artifact keys can be defined here. + +.. code:: yaml + + environment-nocache: + - MAXJOBS + +Note that the ``environment-nocache`` list only exists so that we can +control parameters such as ``make -j ${MAXJOBS}``, allowing us to control +the number of jobs for a given build without effecting the resulting +cache key. + + +Split rules +~~~~~~~~~~~ +The project wide :ref:`split rules ` defaults can +be specified here. + +.. code:: yaml + + split-rules: + devel: + - | + %{includedir} + - | + %{includedir}/** + - | + %{libdir}/lib*.a + - | + %{libdir}/lib*.la + + +.. _project_overrides: + +Overriding plugin defaults +-------------------------- +Base attributes declared by element and source plugins can be overridden +on a project wide basis. This section explains how to make project wide +statements which augment the configuration of an element or source plugin. + + +.. _project_element_overrides: + +Element overrides +~~~~~~~~~~~~~~~~~ +The elements dictionary can be used to override variables, environments +or plugin specific configuration data as shown below. + + +.. code:: yaml + + elements: + + # Override default values for all autotools elements + autotools: + + variables: + bindir: "%{prefix}/bin" + + config: + configure-commands: ... + + environment: + PKG_CONFIG_PATH=%{libdir}/pkgconfig + + +.. _project_source_overrides: + +Source overrides +~~~~~~~~~~~~~~~~ +The sources dictionary can be used to override source plugin +specific configuration data as shown below. + + +.. code:: yaml + + sources: + + # Override default values for all git sources + git: + + config: + checkout-submodules: False + +.. note:: + + The ``sources`` override is available since :ref:`format version 1 ` + + +.. _project_shell: + +Customizing the shell +--------------------- +Since BuildStream cannot know intimate details about your host or about +the nature of the runtime and software that you are building, the shell +environment for debugging and testing applications may need some help. + +The ``shell`` section allows some customization of the shell environment. + +.. note:: + + The ``shell`` section is available since :ref:`format version 1 ` + + +Interactive shell command +~~~~~~~~~~~~~~~~~~~~~~~~~ +By default, BuildStream will use ``sh -i`` when running an interactive +shell, unless a specific command is given to the ``bst shell`` command. + +BuildStream will automatically set a convenient prompt via the ``PS1`` +environment variable for interactive shells; which might be overwritten +depending on the shell you use in your runtime. + +If you are using ``bash``, we recommend the following configuration to +ensure that the customized prompt is not overwritten: + +.. code:: yaml + + shell: + + # Specify the command to run by default for interactive shells + command: [ 'bash', '--noprofile', '--norc', '-i' ] + + +Environment assignments +~~~~~~~~~~~~~~~~~~~~~~~ +In order to cooperate with your host environment, a debugging shell +sometimes needs to be configured with some extra knowledge inheriting +from your host environment. + +This can be achieved by setting up the shell ``environment`` configuration, +which is expressed as a dictionary very similar to the +:ref:`default environment `, except that it +supports host side environment variable expansion in values. + +.. note:: + + The ``environment`` configuration is available since :ref:`format version 4 ` + +For example, to share your host ``DISPLAY`` and ``DBUS_SESSION_BUS_ADDRESS`` +environments with debugging shells for your project, specify the following: + +.. code:: yaml + + shell: + + # Share some environment variables from the host environment + environment: + DISPLAY: '$DISPLAY' + DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS' + +Or, a more complex example is how one might share the host pulseaudio +server with a ``bst shell`` environment: + +.. code:: yaml + + shell: + + # Set some environment variables explicitly + environment: + PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native' + + +Host files +~~~~~~~~~~ +It can be useful to share some files on the host with a shell so that +it can integrate better with the host environment. + +The ``host-files`` configuration allows one to specify files and +directories on the host to be bind mounted into the sandbox. + +.. note:: + + The ``host-files`` configuration is available since :ref:`format version 4 ` + +.. warning:: + + One should never mount directories where one expects to + find data and files which belong to the user, such as ``/home`` + on POSIX platforms. + + This is because the unsuspecting user may corrupt their own + files accidentally as a result. Instead users can use the + ``--mount`` option of ``bst shell`` to mount data into the shell. + + +The ``host-files`` configuration is an ordered list of *mount specifications*. + +Members of the list can be *fully specified* as a dictionary, or a simple +string can be used if only the defaults are required. + +The fully specified dictionary has the following members: + +* ``path`` + + The path inside the sandbox. This is the only mandatory + member of the mount specification. + +* ``host_path`` + + The host path to mount at ``path`` in the sandbox. This + will default to ``path`` if left unspecified. + +* ``optional`` + + Whether the mount should be considered optional. This + is ``False`` by default. + + +Here is an example of a *fully specified mount specification*: + +.. code:: yaml + + shell: + + # Mount an arbitrary resolv.conf from the host to + # /etc/resolv.conf in the sandbox, and avoid any + # warnings if the host resolv.conf doesnt exist. + host-files: + - host_path: '/usr/local/work/etc/resolv.conf' + path: '/etc/resolv.conf' + optional: True + +Here is an example of using *shorthand mount specifications*: + +.. code:: yaml + + shell: + + # Specify a list of files to mount in the sandbox + # directory from the host. + # + # If these do not exist on the host, a warning will + # be issued but the shell will still be launched. + host-files: + - '/etc/passwd' + - '/etc/group' + - '/etc/resolv.conf' + +Host side environment variable expansion is also supported: + +.. code:: yaml + + shell: + + # Mount a host side pulseaudio server socket into + # the shell environment at the same location. + host-files: + - '${XDG_RUNTIME_DIR}/pulse/native' + + +.. _project_builtin_defaults: + +Builtin defaults +---------------- +BuildStream defines some default values for convenience, the default +values overridden by your project's ``project.conf`` are presented here: + + .. literalinclude:: ../../buildstream/data/projectconfig.yaml + :language: yaml diff --git a/doc/source/format_project_refs.rst b/doc/source/format_project_refs.rst new file mode 100644 index 000000000..799b5e761 --- /dev/null +++ b/doc/source/format_project_refs.rst @@ -0,0 +1,78 @@ + + +.. _projectrefs: + +The project.refs file +===================== +If one has elected to store source references in a single ``project.refs`` +file, then it will be stored at the toplevel of your project directory +adjacent to ``project.conf``. This can be configured in your project +using the :ref:`ref-storage configuration ` + +Sources for :mod:`junction ` elements are stored +separately in an adjacent ``junction.refs`` file of the same format. + + +.. _projectrefs_basics: + +Basic behavior +-------------- +When a ``project.refs`` file is in use, any source references found +in the :ref:`inline source declarations ` are considered +invalid and will be ignored, and a warning will be emitted for them. + +When ``bst track`` is run for your project, the ``project.refs`` file +will be updated instead of the inline source declarations. In the absence +of a ``project.refs`` file, ``bst track`` will create one automatically +with the tracking results. + +An interesting property of ``project.refs`` is that it allows for +*cross junction tracking*. This is to say that it is possible to override +the *ref* of a given source in a project that your project depends on via +a :mod:`junction `, without actually modifying the +junctioned project. + + +.. _projectrefs_format: + +Format +------ +The ``project.refs`` uses the same YAML format used throughout BuildStream, +and supports the same :ref:`directives ` which apply to +``project.conf`` and element declaration files (i.e. *element.bst* files). + +The ``project.refs`` file format itself is very simple, it contains a single ``projects`` +key at the toplevel, which is a dictionary of :ref:`project names `. +Each *project name* is a dictionary of *element names*, and each *element name* holds +a list of dictionaries corresponding to the element's :ref:`sources `. + + +**Example** + +.. code:: yaml + + # Main toplevel "projects" key + projects: + + # The local project's name is "core" + core: + + # A dictionary of element names + base/automake.bst: + + # A list of sources corresponding to the element + # in the same order in which they were declared. + # + # The values of this list are dictionaries of the + # symbolic "ref" portion understood by the given + # source plugin implementation. + # + - ref: af6ba39142220687c500f79b4aa2f181d9b24e4... + + # The "core" project depends on the "bootstrap" project, + # here we are allowed to override the refs for the projects + # we depend on through junctions. + bootstrap: + + zlib.bst: + - ref: 4ff941449631ace0d4d203e3483be9dbc9da4540... diff --git a/doc/source/format_public.rst b/doc/source/format_public.rst new file mode 100644 index 000000000..e640de7bb --- /dev/null +++ b/doc/source/format_public.rst @@ -0,0 +1,102 @@ + +Builtin public data +=================== +Elements can provide public data which can be read by other elements +later in the pipeline, the format for exposing public data on a given +element is :ref:`described here `. + +Any element may use public data for whatever purpose it wants, but +BuildStream has some built-in expectations of public data, which resides +completely in the ``bst`` domain. + +In this section we will describe the public data in the ``bst`` domain. + + +.. _public_integration: + +Integration commands +-------------------- + +.. code:: yaml + + # Specify some integration commands + public: + bst: + integration-commands: + - /usr/bin/update-fancy-feature-cache + +The built-in ``integration-commands`` list indicates that depending elements +should run this set of commands before expecting the staged runtime environment +to be functional. + +Typical cases for this include running ``ldconfig`` at the base of a pipeline, +or running commands to update various system caches. + +Integration commands of a given element are automatically run by the +:func:`Element.integrate() ` method +and are used by various plugins. + +Notably the :mod:`BuildElement ` derived classes +will always integrate the build dependencies after staging and before running +any build commands. + + +.. _public_split_rules: + +Split rules +----------- + +.. code:: yaml + + # Specify some split rules + public: + bst: + split-rules: + runtime: + - | + %{bindir}/* + - | + %{sbindir}/* + - | + %{libexecdir}/* + - | + %{libdir}/lib*.so* + +Split rules indicate how the output of an element can be categorized +into *domains*. + +The ``split-rules`` domains are used by the +:func:`Element.stage_artifact() ` +method when deciding what domains of an artifact should be staged. + +The strings listed in each domain are first substituted with the +:ref:`variables ` in context of the given element, and +then applied as a glob style match, as understood by +:func:`utils.glob() ` + +This is used for creating compositions with the :mod:`compose ` +element and can be used by other deployment related elements for the purpose of +splitting element artifacts into separate packages. + + +.. _public_overlap_whitelist: + +Overlap whitelist +----------------- + +The overlap whitelist indicates which files this element is allowed to overlap +over other elements when staged together with other elements. + +Each item in the overlap whitelist has substitutions applied from +:ref:`variables `, and is then applied as a glob-style match +(i.e. :func:`utils.glob() `). + +.. code:: yaml + + public: + bst: + overlap-whitelist: + - | + %{sysconfdir}/* + - | + /etc/fontcache diff --git a/doc/source/formatintro.rst b/doc/source/formatintro.rst deleted file mode 100644 index b1780f9dc..000000000 --- a/doc/source/formatintro.rst +++ /dev/null @@ -1,291 +0,0 @@ - - -Introduction -============ -At the core of BuildStream is a data model of :mod:`Elements ` which -are parsed from ``.bst`` files in a project directory and configured from a few different -sources. - -When BuildStream loads your project, various levels of composition occur, allowing -configuration on various levels with different priority. - -This page provides an introduction to the project directory structure, explains the -basic *directives* supported inherently throughout the format, and outlines how composition -occurs and what configurations are considered in which order. - -The meaning of the various constructs expressed in the BuildStream format are covered -in other sections of the documentation. - -.. _format_structure: - - -Directory structure -------------------- -A BuildStream project is a directory consisting of: - -* A project configuration file -* BuildStream element files -* Optional user defined plugins -* An optional project.refs file - -A typical project structure may look like this:: - - myproject/project.conf - myproject/project.refs - myproject/elements/element1.bst - myproject/elements/element2.bst - myproject/elements/... - myproject/plugins/customelement.py - myproject/plugins/customelement.yaml - myproject/plugins/... - - -Except for the project configuration file, the user is allowed to structure -their project directory in any way. For documentation on the format of the project -configuration file, refer to the :ref:`projectconf` documentation. - -Simpler projects may choose to place all element definition files at the -root of the project directory while more complex projects may decide to -put stacks in one directory and other floating elements into other directories, -perhaps placing deployment elements in another directory, this is all fine. - -The important part to remember is that when you declare dependency relationships, -a project relative path to the element one depends on must be provided. - - -.. _format_composition: - -Composition ------------ -Below are the various sources of configuration which go into an element or source in the -order in which they are applied. Configurations which are applied later have a higher -priority and override configurations which precede them. - - -1. Builtin defaults -~~~~~~~~~~~~~~~~~~~ -The :ref:`builtin defaults ` provide a set of builtin -default default values for ``project.conf``. - -The project wide defaults defined in the builtin project configuration, such as the -*variables* or *environment* sections, form the base configuration of all elements. - - -2. Project configuration -~~~~~~~~~~~~~~~~~~~~~~~~ -The :ref:`project wide defaults ` specified in your -``project.conf`` are now applied on top of builtin defaults. - -Defaults such as the :ref:`variables ` or -:ref:`environment ` which are specified in -your ``project.conf`` override the builtin defaults for elements. - -Note that :ref:`plugin type specific configuration ` -in ``project.conf`` is not applied until later. - - -3. Plugin defaults -~~~~~~~~~~~~~~~~~~ -Elements and Sources are all implemented as plugins. - -Each Element plugin installs a ``.yaml`` file along side their plugin to -define the default *variables*, *environment* and *config*. The *config* -is element specific and as such this is the first place where defaults -can be set on the *config* section. - -The *variables* and *environment* specified in the declaring plugin's -defaults here override the project configuration defaults for the given -element ``kind``. - -Source plugins do not have a ``.yaml`` file, and do not have *variables* or -*environment*. - - -4. Project configuration overrides -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``project.conf`` now gives you :ref:`another opportunity ` to -override configuration on a per plugin basis. - -Configurations specified in the :ref:`elements ` or -:ref:`sources ` sections of the ``project.conf`` -will override the given plugin's defaults. - -In this phase, it is possible to override any configurations of a given plugin, -including configuration in element specific *config* sections. - -See also :ref:`project_overrides` - - -5. Plugin declarations -~~~~~~~~~~~~~~~~~~~~~~~ -Finally, after having resolved any :ref:`conditionals ` -in the parsing phase of loading element declarations; the configurations specified in a -``.bst`` file have the last word on any configuration in the data model. - - -.. _format_directives: - -Directives ----------- - -.. _format_directives_conditional: - -(?) Conditionals -~~~~~~~~~~~~~~~~ -The ``(?)`` directive allows expression of conditional statements which -test :ref:`project option ` values. - -The ``(?)`` directive may appear as a key in any dictionary expressed -in YAML, and its value is a list of conditional expressions. Each conditional -expression must be a single key dictionary, where the key is the conditional -expression itself, and the value is a dictionary to be composited into the -parent dictionary containing the ``(?)`` directive if the expression evaluates -to a truthy value. - -**Example:** - -.. code:: yaml - - variables: - prefix: "/usr" - enable-debug: False - (?): - - relocate == True: - prefix: "/opt" - - debug == True: - enable-debug: True - - -Expressions are evaluated in the specified order, and each time an expression -evaluates to a truthy value, its value will be composited to the parent dictionary -in advance of processing other elements, allowing for logically overriding previous -decisions in the condition list. - -Nesting of conditional statements is also supported. - -**Example:** - -.. code:: yaml - - variables: - enable-logging: False - enable-debug: False - (?): - - logging == True: - enable-logging: True - (?): - - debugging == True: - enable-debug: True - - -Conditionals are expressed in a pythonic syntax, the specifics for -testing the individually supported option types are described in -their :ref:`respective documentation `. - -Compound conditionals are also allowed. - -**Example:** - -.. code:: yaml - - variables: - enable-debug: False - (?): - - (logging == True and debugging == True): - enable-debug: True - - -.. _format_directives_assertion: - -(!) Assertions -~~~~~~~~~~~~~~ -Assertions allow the project author to abort processing and present -a custom error message to the user building their project. - -This is only useful when used with conditionals, allowing the project -author to assert some invalid configurations. - - -**Example:** - -.. code:: yaml - - variables: - (?): - - (logging == False and debugging == True): - - (!): | - - Impossible to print any debugging information when - logging is disabled. - - -.. _format_directives_list_prepend: - -(<) List Prepend -~~~~~~~~~~~~~~~~ -Indicates that the list should be prepended to the target list, -instead of the default behavior which is to replace the target list. - -**Example:** - -.. code:: yaml - - config: - configure-commands: - # Before configuring, lets make sure we're using - # the latest config.sub & config.guess - (<): - - cp %{datadir}/automake-*/config.{sub,guess} . - - -.. _format_directives_list_append: - -(>) List Append -~~~~~~~~~~~~~~~ -Indicates that the list should be appended to the target list, instead -of the default behavior which is to replace the target list. - -**Example:** - -.. code:: yaml - - public: - bst: - split-rules: - devel: - # This element also adds some extra stubs which - # need to be included in the devel domain - (>): - - "%{libdir}/*.stub" - - -.. _format_directives_list_overwrite: - -(=) List Overwrite -~~~~~~~~~~~~~~~~~~ -Indicates that the list should be overwritten completely. - -This exists mostly for completeness, and we recommend using literal -lists most of the time instead of list overwrite directives when the -intent is to overwrite a list. - -This has the same behavior as a literal list, except that an -error will be triggered in the case that there is no underlying -list to overwrite; whereas a literal list will simply create a new -list. - -The added error protection can be useful when intentionally -overwriting a list in an element's *public data*, which is mostly -free form and not validated. - - -**Example:** - -.. code:: yaml - - config: - install-commands: - # This element's `make install` is broken, replace it. - (=): - - cp src/program %{bindir} diff --git a/doc/source/index.rst b/doc/source/index.rst index 16a2df3e8..5c4a62007 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -17,7 +17,7 @@ Later sections provide detailed information on BuilStream internals. .. toctree:: :maxdepth: 1 - about + main_about main_install main_using main_core diff --git a/doc/source/install.rst b/doc/source/install.rst deleted file mode 100644 index 500c94ac2..000000000 --- a/doc/source/install.rst +++ /dev/null @@ -1,175 +0,0 @@ - - -.. _install: - -Installing BuildStream -====================== -BuildStream requires the following base system requirements: - -* python3 >= 3.4 -* libostree >= v2017.8 with introspection data -* bubblewrap >= 0.1.2 -* fuse2 -* PyGObject introspection bindings -* psutil python library (so you don't have to install GCC and python-devel to build it yourself) - -If you intend to push built artifacts to a remote artifact server, -which requires special permissions, you will also need: - -* ssh - - -Installing from source (recommended) ------------------------------------- -Until BuildStream is available in your distro, you will need to install -it yourself from the `git repository `_ -using python's ``pip`` package manager. - -For the purpose of installing BuildStream while there are no distro packages, -you will additionally need: - -* pip for python3 (only required for setup) -* Python 3 development libraries and headers -* git (to checkout buildstream) - - -Installing dependencies -~~~~~~~~~~~~~~~~~~~~~~~ - - -Arch Linux -++++++++++ -Install the dependencies with:: - - sudo pacman -S fuse2 ostree bubblewrap git \ - python python-pip python-gobject python-psutil - - -Debian -++++++ - - -Stretch -^^^^^^^ -With stretch, you first need to ensure that you have the backports repository -setup as described `here `_ - -By adding the following line to your sources.list:: - - deb http://ftp.debian.org/debian stretch-backports main - -And then running:: - - sudo apt-get update - -At this point you should be able to get the system requirements with:: - - sudo apt-get install \ - fuse ostree gir1.2-ostree-1.0 bubblewrap git \ - python3 python3-pip python3-gi python3-psutil - sudo apt-get install -t stretch-backports \ - gir1.2-ostree-1.0 ostree - - -Buster or Sid -^^^^^^^^^^^^^ -For debian unstable or testing, only the following line should be enough -to get the base system requirements installed:: - - sudo apt-get install \ - fuse ostree gir1.2-ostree-1.0 bubblewrap git \ - python3 python3-pip python3-gi python3-psutil - - -Fedora -++++++ -For recent fedora systems, the following line should get you the system -requirements you need:: - - dnf install -y fuse ostree bubblewrap git \ - python3 python3-pip python3-gobject python3-psutil - - -Installing -~~~~~~~~~~ -Once you have the base system dependencies, you can clone the BuildStream -git repository and install it as a regular user:: - - git clone https://gitlab.com/BuildStream/buildstream.git - cd buildstream - pip3 install --user -e . - -This will install buildstream's pure python dependencies into -your user's homedir in ``~/.local`` and will run BuildStream directly -from the git checkout directory. - -Keep following the instructions below to ensure that the ``bst`` -command is in your ``PATH`` and to enable bash completions for it. - -.. note:: - - We recommend the ``-e`` option because you can upgrade your - installation by simply updating the checked out git repository. - - If you want a full installation that is not linked to your - git checkout, just omit the ``-e`` option from the above commands. - - -Adjust PATH -~~~~~~~~~~~ -Since BuildStream is now installed under your local user's install directories, -you need to ensure that ``PATH`` is adjusted. - -A regular way to do this is to add the following line to the end of your ``~/.bashrc``:: - - export PATH="${PATH}:${HOME}/.local/bin" - -.. note:: - - You will have to restart your terminal in order for these changes to take effect. - - -Bash completions -~~~~~~~~~~~~~~~~ -Bash completions are supported by sourcing the ``buildstream/data/bst`` -script found in the BuildStream repository. On many systems this script -can be installed into a completions directory but when installing BuildStream -without a package manager this is not an option. - -To enable completions for an installation of BuildStream you -installed yourself from git, just append the script verbatim -to your ``~/.bash_completion``: - -.. literalinclude:: ../../buildstream/data/bst - :language: yaml - - -Upgrading BuildStream -~~~~~~~~~~~~~~~~~~~~~ -Assuming you have followed the default instructions above, all -you need to do to upgrade BuildStream is to update your local git -checkout:: - - cd /path/to/buildstream - git pull --rebase - -If you did not specify the ``-e`` option at install time, you will -need to cleanly reinstall BuildStream:: - - pip3 uninstall buildstream - cd /path/to/buildstream - git pull --rebase - pip3 install --user . - - -Installing from distro packages -------------------------------- - - -Arch Linux -~~~~~~~~~~ -Install `buildstream `_ -from `AUR `_. -Alternatively, use -`buildstream-git `_ -for the lastest version of the development branch. diff --git a/doc/source/install_artifacts.rst b/doc/source/install_artifacts.rst new file mode 100644 index 000000000..e0808e8fe --- /dev/null +++ b/doc/source/install_artifacts.rst @@ -0,0 +1,200 @@ + + +.. _artifacts: + +Installing an artifact server +============================= +BuildStream caches the results of builds in a local artifact cache, and will +avoid building an element if there is a suitable build already present in the +local artifact cache. + +In addition to the local artifact cache, you can configure one or more remote +artifact caches and BuildStream will then try to pull a suitable build from one +of the remotes, falling back to a local build if needed. + +Configuring BuildStream to use remote caches +-------------------------------------------- +A project will often set up continuous build infrastructure that pushes +built artifacts to a shared cache, so developers working on the project can +make use of these pre-built artifacts instead of having to each build the whole +project locally. The project can declare this cache in its +:ref:`project configuration file `. + +Users can declare additional remote caches in the :ref:`user configuration +`. There are several use cases for this: your project may not +define its own cache, it may be useful to have a local mirror of its cache, or +you may have a reason to share artifacts privately. + +Remote artifact caches are identified by their URL. There are currently three +supported protocols: + +* ``http``: Pull-only access, without transport-layer security +* ``https``: Pull-only access, with transport-layer security +* ``ssh``: Push access, authenticated via SSH + +BuildStream allows you to configure as many caches as you like, and will query +them in a specific order: + +1. Project-specific overrides in the user config +2. Project configuration +3. User configuration + +When an artifact is built locally, BuildStream will try to push it to all the +caches which have the ``push: true`` flag set. You can also manually push +artifacts to a specific cache using the :ref:`bst pull command `. + +Artifacts are identified using the element's :ref:`cache key ` so +the builds provided by a cache should be interchangable with those provided +by any other cache. + + +Setting up a remote artifact cache +---------------------------------- +The rest of this page outlines how to set up a shared artifact cache. + +Setting up the user +~~~~~~~~~~~~~~~~~~~ +A specific user is not needed for downloading artifacts, but since we +are going to use ssh to upload the artifacts, you will want a dedicated +user to own the artifact cache. + +.. code:: bash + + useradd artifacts + + +Installing the receiver +~~~~~~~~~~~~~~~~~~~~~~~ +You will also need to install BuildStream on the artifact server in order +to receive uploaded artifacts over ssh. Follow the instructions for installing +BuildStream :ref:`here ` + +When installing BuildStream on the artifact server, it must be installed +in a system wide location, with ``pip3 install .`` in the BuildStream +checkout directory. + +Otherwise, some tinkering is required to ensure BuildStream is available +in ``PATH`` when it's companion ``bst-artifact-receive`` program is run +remotely. + +You can install only the artifact receiver companion program without +requiring BuildStream's more exigent dependencies by setting the +``BST_ARTIFACTS_ONLY`` environment variable at install time, like so: + +.. code:: + + BST_ARTIFACTS_ONLY=1 pip3 install . + + +Initializing the cache +~~~~~~~~~~~~~~~~~~~~~~ +Now that you have a dedicated user to own the artifact cache, change +to that user, and create the artifact cache ostree repository directly +in it's home directory as such: + +.. code:: bash + + ostree init --mode archive-z2 --repo artifacts + +This should result in an artifact cache residing at the path ``/home/artifacts/artifacts`` + + +Serve the cache over https +~~~~~~~~~~~~~~~~~~~~~~~~~~ +This part should be pretty simple, you can do this with various technologies, all +we really require is that you make the artifacts available over https (you can use +http but until we figure out using gpg signed ostree commits for the artifacts, it's +better to serve over https). + +Here is an example, note that you must have a certificate **pem** file to use, as +is the case for hosting anything over https. + +.. code:: python + + import http.server, ssl, os + + # Maybe use a custom port, especially if you are serving + # other web pages on the same computer + server_address = ('localhost', 443) + artifact_path = '/home/artifacts' + + # The http server will serve from it's current + # working directory + os.chdir(artifact_path) + + # Create Server + httpd = http.server.HTTPServer( + server_address, + http.server.SimpleHTTPRequestHandler) + + # Add ssl + httpd.socket = ssl.wrap_socket(httpd.socket, + server_side=True, + certfile='localhost.pem', + ssl_version=ssl.PROTOCOL_TLSv1) + + # Run it + httpd.serve_forever() + + +Configure and run sshd +~~~~~~~~~~~~~~~~~~~~~~ +You will need to run the sshd service to allow uploading artifacts. + +For this you will want something like the following in your ``/etc/ssh/sshd_config`` + +.. code:: bash + + # Allow ssh logins/commands with the artifacts user + AllowUsers artifacts + + # Some specifics for the artifacts user + Match user artifacts + + # Dont allow password authentication for artifacts user + # + PasswordAuthentication no + + # Also lets dedicate this login for only running the + # bst-artifact-receive program, note that the full + # command must be specified here; 'artifacts' is + # the HOME relative path to the artifact cache. + # The exact pull URL must also be specified. + ForceCommand bst-artifact-receive --pull-url https://example.com/artifacts --verbose artifacts + + +User configuration +~~~~~~~~~~~~~~~~~~ +The user configuration for artifacts is documented with the rest +of the :ref:`user configuration documentation `. + +Assuming you have the same setup used in this document, and that your +host is reachable on the internet as ``artifacts.com`` (for example), +then a user can use the following user configuration: + +.. code:: yaml + + # + # Artifacts + # + artifacts: + + url: https://artifacts.com/artifacts + + # Alternative form if you have push access to the cache + #url: ssh://artifacts@artifacts.com:22200/artifacts + #push: true + + +Authenticating users +~~~~~~~~~~~~~~~~~~~~ +In order to give permission to a given user to upload +artifacts, simply use the regular ``ssh`` method. + +First obtain the user's public ssh key, and add it +to the authorized keys, like so: + +.. code:: bash + + cat user_id_rsa.pub >> /home/artifacts/.ssh/authorized_keys + diff --git a/doc/source/install_docker.rst b/doc/source/install_docker.rst new file mode 100644 index 000000000..9b0f8aa73 --- /dev/null +++ b/doc/source/install_docker.rst @@ -0,0 +1,45 @@ + + +.. _docker: + +BuildStream inside Docker +------------------------- +If your system cannot provide the base system requirements for BuildStream, then it is possible to run buildstream within a Docker image. + +The BuildStream project provides +`Docker images `_ +containing BuildStream and its dependencies. +This gives you an easy way to get started using BuildStream on any Unix-like +platform where Docker is available, including Mac OS X. + +We recommend using the +`bst-here wrapper script `_ +which automates the necessary container setup. You can download it and make +it executable like this: + +.. code:: bash + + mkdir -p ~/.local/bin + curl --get https://gitlab.com/BuildStream/buildstream/raw/master/contrib/bst-here > ~/.local/bin/bst-here + chmod +x ~/.local/bin/bst-here + +Check if ``~/.local/bin`` appears in your PATH environment variable -- if it +doesn't, you should +`edit your ~/.profile so that it does `_. + +Once the script is available in your PATH, you can run ``bst-here`` to open a +shell session inside a new container based off the latest version of the +buildstream-fedora Docker image. The current working directory will be mounted +inside the container at ``/src``. + +You can also run individual BuildStream commands as ``bst-here COMMAND``. For +example: ``bst-here show systems/my-system.bst``. Note that BuildStream won't +be able to integrate with Bash tab-completion if you invoke it in this way. + +Two Docker volumes are set up by the ``bst-here`` script: + + * buildstream-cache -- mounted at ``~/.cache/buildstream`` + * buildstream-config -- mounted at ``~/.config/`` + +These are necessary so that your BuildStream cache and configuration files +persist between invocations of ``bst-here``. diff --git a/doc/source/install_main.rst b/doc/source/install_main.rst new file mode 100644 index 000000000..500c94ac2 --- /dev/null +++ b/doc/source/install_main.rst @@ -0,0 +1,175 @@ + + +.. _install: + +Installing BuildStream +====================== +BuildStream requires the following base system requirements: + +* python3 >= 3.4 +* libostree >= v2017.8 with introspection data +* bubblewrap >= 0.1.2 +* fuse2 +* PyGObject introspection bindings +* psutil python library (so you don't have to install GCC and python-devel to build it yourself) + +If you intend to push built artifacts to a remote artifact server, +which requires special permissions, you will also need: + +* ssh + + +Installing from source (recommended) +------------------------------------ +Until BuildStream is available in your distro, you will need to install +it yourself from the `git repository `_ +using python's ``pip`` package manager. + +For the purpose of installing BuildStream while there are no distro packages, +you will additionally need: + +* pip for python3 (only required for setup) +* Python 3 development libraries and headers +* git (to checkout buildstream) + + +Installing dependencies +~~~~~~~~~~~~~~~~~~~~~~~ + + +Arch Linux +++++++++++ +Install the dependencies with:: + + sudo pacman -S fuse2 ostree bubblewrap git \ + python python-pip python-gobject python-psutil + + +Debian +++++++ + + +Stretch +^^^^^^^ +With stretch, you first need to ensure that you have the backports repository +setup as described `here `_ + +By adding the following line to your sources.list:: + + deb http://ftp.debian.org/debian stretch-backports main + +And then running:: + + sudo apt-get update + +At this point you should be able to get the system requirements with:: + + sudo apt-get install \ + fuse ostree gir1.2-ostree-1.0 bubblewrap git \ + python3 python3-pip python3-gi python3-psutil + sudo apt-get install -t stretch-backports \ + gir1.2-ostree-1.0 ostree + + +Buster or Sid +^^^^^^^^^^^^^ +For debian unstable or testing, only the following line should be enough +to get the base system requirements installed:: + + sudo apt-get install \ + fuse ostree gir1.2-ostree-1.0 bubblewrap git \ + python3 python3-pip python3-gi python3-psutil + + +Fedora +++++++ +For recent fedora systems, the following line should get you the system +requirements you need:: + + dnf install -y fuse ostree bubblewrap git \ + python3 python3-pip python3-gobject python3-psutil + + +Installing +~~~~~~~~~~ +Once you have the base system dependencies, you can clone the BuildStream +git repository and install it as a regular user:: + + git clone https://gitlab.com/BuildStream/buildstream.git + cd buildstream + pip3 install --user -e . + +This will install buildstream's pure python dependencies into +your user's homedir in ``~/.local`` and will run BuildStream directly +from the git checkout directory. + +Keep following the instructions below to ensure that the ``bst`` +command is in your ``PATH`` and to enable bash completions for it. + +.. note:: + + We recommend the ``-e`` option because you can upgrade your + installation by simply updating the checked out git repository. + + If you want a full installation that is not linked to your + git checkout, just omit the ``-e`` option from the above commands. + + +Adjust PATH +~~~~~~~~~~~ +Since BuildStream is now installed under your local user's install directories, +you need to ensure that ``PATH`` is adjusted. + +A regular way to do this is to add the following line to the end of your ``~/.bashrc``:: + + export PATH="${PATH}:${HOME}/.local/bin" + +.. note:: + + You will have to restart your terminal in order for these changes to take effect. + + +Bash completions +~~~~~~~~~~~~~~~~ +Bash completions are supported by sourcing the ``buildstream/data/bst`` +script found in the BuildStream repository. On many systems this script +can be installed into a completions directory but when installing BuildStream +without a package manager this is not an option. + +To enable completions for an installation of BuildStream you +installed yourself from git, just append the script verbatim +to your ``~/.bash_completion``: + +.. literalinclude:: ../../buildstream/data/bst + :language: yaml + + +Upgrading BuildStream +~~~~~~~~~~~~~~~~~~~~~ +Assuming you have followed the default instructions above, all +you need to do to upgrade BuildStream is to update your local git +checkout:: + + cd /path/to/buildstream + git pull --rebase + +If you did not specify the ``-e`` option at install time, you will +need to cleanly reinstall BuildStream:: + + pip3 uninstall buildstream + cd /path/to/buildstream + git pull --rebase + pip3 install --user . + + +Installing from distro packages +------------------------------- + + +Arch Linux +~~~~~~~~~~ +Install `buildstream `_ +from `AUR `_. +Alternatively, use +`buildstream-git `_ +for the lastest version of the development branch. diff --git a/doc/source/main_about.rst b/doc/source/main_about.rst new file mode 100644 index 000000000..a6210d3d8 --- /dev/null +++ b/doc/source/main_about.rst @@ -0,0 +1 @@ +.. include:: ../../README.rst diff --git a/doc/source/main_core.rst b/doc/source/main_core.rst index 9e43bef6e..c0757b7fa 100644 --- a/doc/source/main_core.rst +++ b/doc/source/main_core.rst @@ -1,7 +1,7 @@ -Reference documentation -======================= +Reference +========= This section details the core API reference along with other more elaborate details about BuildStream internals. @@ -9,7 +9,8 @@ other more elaborate details about BuildStream internals. .. toctree:: :maxdepth: 2 - authoring + core_format + core_plugins core_framework core_additional diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 31e4680fb..c65e7ca96 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -1,12 +1,12 @@ -Installing -========== +Install +======= This section covers how to install BuildStream onto your machine, how to run BuildStream inside a docker image and also how to configure an artifact server. .. toctree:: :maxdepth: 2 - install - docker - artifacts + install_main + install_docker + install_artifacts diff --git a/doc/source/main_using.rst b/doc/source/main_using.rst index 62846baf8..dc0f5e69b 100644 --- a/doc/source/main_using.rst +++ b/doc/source/main_using.rst @@ -1,12 +1,12 @@ -Using BuildStream -================= +Using +===== This section details how to use the BuildStream command line interface and work with existing BuildStream projects. .. toctree:: :maxdepth: 2 - examples - user_config - commands + using_examples + using_config + using_commands diff --git a/doc/source/projectconf.rst b/doc/source/projectconf.rst deleted file mode 100644 index 6174ee728..000000000 --- a/doc/source/projectconf.rst +++ /dev/null @@ -1,811 +0,0 @@ - -.. _projectconf: - - -Project configuration -===================== -The project configuration file should be named ``project.conf`` and -be located at the project root. It holds information such as Source -aliases relevant for the sources used in the given project as well as -overrides for the configuration of element types used in the project. - -Values specified in the project configuration override any of the -default BuildStream project configuration, which is included -:ref:`here ` for reference. - - -.. _project_essentials: - -Essentials ----------- - - -.. _project_format_name: - -Project name -~~~~~~~~~~~~ -The project name is a unique symbol for your project and will -be used to distinguish your project from others in user preferences, -namspaceing of your project's artifacts in shared artifact caches, -and in any case where BuildStream needs to distinguish between multiple -projects. - -The first thing to setup in your ``project.conf`` should be the name -of your project. - -.. code:: yaml - - name: my-project-name - -.. note:: - - The project name may contain alphanumeric characters, dashes and - underscores, and may not start with a leading digit. - - -.. _project_format_version: - -Format version -~~~~~~~~~~~~~~ -The BuildStream format is guaranteed to be backwards compatible -with any earlier releases. The project's minimum required format -version of BuildStream can be specified in ``project.conf`` with -the ``format-version`` field, e.g.: - -.. code:: yaml - - # The minimum base BuildStream format - format-version: 0 - -BuildStream will increment it's core YAML format version at least once -in any given minor point release where the format has been extended -to support a new feature. - -.. note:: - - External :mod:`Element ` and :mod:`Source ` - plugins also implement their own YAML configuration fragments and as - such are revisioned separately from the core format. See :ref:`project_plugins` - for details on specifying a minimum version of a specific plugin. - - Core :mod:`Elements ` and :mod:`Sources ` - which are maintained and distributed as a part of BuildStream are revisioned - under the same global ``format-version`` described here. - - -.. _project_element_path: - -Element path -~~~~~~~~~~~~ -To allow the user to structure their project nicely, BuildStream -allows the user to specify a project subdirectory where element -``.bst`` files are stored. - -.. code:: yaml - - element-path: elements - -Note that elements are referred to by their relative paths, whenever -elements are referred to in a ``.bst`` file or on the command line. - - -.. _project_format_ref_storage: - -Ref storage -~~~~~~~~~~~ -By default, BuildStream expects to read and write source references -directly in the :ref:`source declaration `, but this -can be inconvenient and prohibitive in some workflows. - -Alternatively, BuildStream allows source references to be stored -centrally in a :ref:`project.refs file ` in the toplevel -:ref:`project directory `. - -This can be controlled with the ``ref-storage`` option, which is -allowed to be configured with the following values: - -* ``inline`` - - Source references are stored directly in the - :ref:`source declaration ` - -* ``project.refs`` - - Source references are stored in the ``project.refs`` file, and - junction source references are stored in the ``junction.refs`` file. - -To enable storing of source references in ``project.refs``, add the -following to your ``project.conf``: - -.. code:: yaml - - ref-storage: project.refs - -.. note:: - - The ``ref-storage`` configuration is available since :ref:`format version 8 ` - - -Fail on overlaps -~~~~~~~~~~~~~~~~ -When multiple elements are staged, there's a possibility that different -elements will try and stage different versions of the same file. - -When ``fail-on-overlap`` is true, if an overlap is detected -that hasn't been allowed by the element's -:ref:`overlap whitelist`, -then an error will be raised and the build will fail. - -otherwise, a warning will be raised indicating which files had overlaps, -and the order that the elements were overlapped. - -.. code:: yaml - - fail-on-overlap: true - - -.. _project_source_aliases: - -Source aliases -~~~~~~~~~~~~~~ -In order to abstract the download location of source code and -any assets which need to be downloaded, and also as a matter of -convenience, BuildStream allows one to create named aliases for -URLs which are to be used in the individual ``.bst`` files. - -.. code:: yaml - - aliases: - foo: git://git.foo.org/ - bar: http://bar.com/downloads/ - - -Sandbox options -~~~~~~~~~~~~~~~ -Sandbox options for the whole project can be supplied in -``project.conf`` in the same way as in an element. See :ref:`element configuration ` -for more detail. - -.. code:: yaml - - # Specify a user id and group id to use in the build sandbox. - sandbox: - build-uid: 1003 - build-gid: 1001 - -.. note:: - - The ``sandbox`` configuration is available since :ref:`format version 6 ` - - -.. _project_essentials_artifacts: - -Artifact server -~~~~~~~~~~~~~~~ -If you have setup an :ref:`artifact server ` for your -project then it is convenient to configure this in your ``project.conf`` -so that users need not have any additional configuration to communicate -with an artifact share. - -.. code:: yaml - - artifacts: - - # A url from which to download prebuilt artifacts - url: https://foo.com/artifacts - -You can also specify a list of caches here; earlier entries in the list -will have higher priority than later ones. - - -.. _project_plugins: - -External plugins ----------------- -If your project makes use of any custom :mod:`Element ` or -:mod:`Source ` plugins, then the project must inform BuildStream -of the plugins it means to make use of and the origin from which they can be loaded. - -Note that plugins with the same name from different origins are not permitted. - - -Local plugins -~~~~~~~~~~~~~ -Local plugins are expected to be found in a subdirectory of the actual -BuildStream project. :mod:`Element ` and -:mod:`Source ` plugins should be stored in separate -directories to avoid namespace collisions. - -The versions of local plugins are largely immaterial since they are -revisioned along with the project by the user, usually in a VCS like git. -However, for the sake of consistency with other plugin loading origins -we require that you specify a version, this can always be ``0`` for a local -plugin. - - -.. code:: yaml - - plugins: - - - origin: local - path: plugins/sources - - # We want to use the `mysource` source plugin located in our - # project's `plugins/sources` subdirectory. - sources: - mysource: 0 - - -Pip plugins -~~~~~~~~~~~ -Plugins loaded from the ``pip`` origin are expected to be installed -separately on the host operating system using python's package management -system. - -.. code:: yaml - - plugins: - - - origin: pip - - # Specify the name of the python package containing - # the plugins we want to load. The name one would use - # on the `pip install` command line. - # - package-name: potato - - # We again must specify a minimal format version for the - # external plugin, it is allowed to be `0`. - # - elements: - potato: 0 - - -.. _project_options: - -Options -------- -Options are how BuildStream projects can define parameters which -can be configured by users invoking BuildStream to build your project. - -Options are declared in the ``project.conf`` in the main ``options`` -dictionary. - -.. code:: yaml - - options: - debug: - type: bool - description: Whether to enable debugging - default: False - -Users can configure those options when invoking BuildStream with the -``--option`` argument:: - - $ bst --option debug True ... - -.. note:: - - The name of the option may contain alphanumeric characters - underscores, and may not start with a leading digit. - - -Common properties -~~~~~~~~~~~~~~~~~ -All option types accept the following common attributes - -* ``type`` - - Indicates the type of option to declare - -* ``description`` - - A description of the meaning of the option - -* ``variable`` - - Optionally indicate a :ref:`variable ` name to - export the option to. A string form of the selected option will - be used to set the exported value. - - If used, this value will override any existing value for the - variable declared in ``project.conf``, and will be overridden in - the regular :ref:`composition order `. - - .. note:: - - The name of the variable to export may contain alphanumeric - characters, dashes, underscores, and may not start with a leading - digit. - - -.. _project_options_bool: - -Boolean -~~~~~~~ -The ``bool`` option type allows specifying boolean values which -can be cased in conditional expressions. - - -**Declaring** - -.. code:: yaml - - options: - debug: - type: bool - description: Whether to enable debugging - default: False - - -**Evaluating** - -Boolean options can be tested in expressions with equality tests: - -.. code:: yaml - - variables: - enable-debug: False - (?): - - debug == True: - enable-debug: True - -Or simply treated as truthy values: - -.. code:: yaml - - variables: - enable-debug: False - (?): - - debug: - enable-debug: True - - -**Exporting** - -When exporting boolean options as variables, a ``True`` option value -will be exported as ``1`` and a ``False`` option as ``0`` - - -.. _project_options_enum: - -Enumeration -~~~~~~~~~~~ -The ``enum`` option type allows specifying a string value -with a restricted set of possible values. - - -**Declaring** - -.. code:: yaml - - options: - loglevel: - type: enum - description: The logging level - values: - - debug - - info - - warning - default: info - - -**Evaluating** - -Enumeration options must be tested as strings in conditional -expressions: - -.. code:: yaml - - variables: - enable-debug: False - (?): - - loglevel == "debug": - enable-debug: True - - -**Exporting** - -When exporting enumeration options as variables, the value is -exported as a variable directly, as it is a simple string. - - -.. _project_options_flags: - -Flags -~~~~~ -The ``flags`` option type allows specifying a list of string -values with a restricted set of possible values. - -In contrast with the ``enum`` option type, the *default* value -need not be specified and will default to an empty set. - - -**Declaring** - -.. code:: yaml - - options: - logmask: - type: flags - description: The logging mask - values: - - debug - - info - - warning - default: - - info - - -**Evaluating** - -Options of type ``flags`` can be tested in conditional expressions using -a pythonic *in* syntax to test if an element is present in a set: - -.. code:: yaml - - variables: - enable-debug: False - (?): - - ("debug" in logmask): - enable-debug: True - - -**Exporting** - -When exporting flags options as variables, the value is -exported as a comma separated list of selected value strings. - - -.. _project_options_arch: - -Architecture -~~~~~~~~~~~~ -The ``arch`` option type is special enumeration option which -defaults to the result of `uname -m`, and does not support -assigning any default in the project configuration. - -.. code:: yaml - - options: - machine_arch: - type: arch - description: The machine architecture - values: - - arm - - aarch64 - - i386 - - x86_64 - - -Architecture options can be tested with the same expressions -as other Enumeration options. - - -.. _project_options_element_mask: - -Element mask -~~~~~~~~~~~~ -The ``element-mask`` option type is a special Flags option -which automatically allows only element names as values. - -.. code:: yaml - - options: - debug_elements: - type: element-mask - description: The elements to build in debug mode - -This can be convenient for automatically declaring an option -which might apply to any element, and can be tested with the -same syntax as other Flag options. - - -.. code:: yaml - - variables: - enable-debug: False - (?): - - ("element.bst" in debug_elements): - enable-debug: True - - - -.. _project_defaults: - -Element default configuration ------------------------------ -The ``project.conf`` plays a role in defining elements by -providing default values and also by overriding values declared -by plugins on a plugin wide basis. - -See the :ref:`composition ` documentation for -more detail on how elements are composed. - - -.. _project_defaults_variables: - -Variables -~~~~~~~~~ -The defaults for :ref:`Variables ` used in your -project is defined here. - -.. code:: yaml - - variables: - prefix: "/usr" - - -.. _project_defaults_environment: - -Environment -~~~~~~~~~~~ -The defaults environment for the build sandbox is defined here. - -.. code:: yaml - - environment: - PATH: /usr/bin:/bin:/usr/sbin:/sbin - -Additionally, the special ``environment-nocache`` list which specifies -which environment variables do not effect build output, and are thus -not considered in the calculation of artifact keys can be defined here. - -.. code:: yaml - - environment-nocache: - - MAXJOBS - -Note that the ``environment-nocache`` list only exists so that we can -control parameters such as ``make -j ${MAXJOBS}``, allowing us to control -the number of jobs for a given build without effecting the resulting -cache key. - - -Split rules -~~~~~~~~~~~ -The project wide :ref:`split rules ` defaults can -be specified here. - -.. code:: yaml - - split-rules: - devel: - - | - %{includedir} - - | - %{includedir}/** - - | - %{libdir}/lib*.a - - | - %{libdir}/lib*.la - - -.. _project_overrides: - -Overriding plugin defaults --------------------------- -Base attributes declared by element and source plugins can be overridden -on a project wide basis. This section explains how to make project wide -statements which augment the configuration of an element or source plugin. - - -.. _project_element_overrides: - -Element overrides -~~~~~~~~~~~~~~~~~ -The elements dictionary can be used to override variables, environments -or plugin specific configuration data as shown below. - - -.. code:: yaml - - elements: - - # Override default values for all autotools elements - autotools: - - variables: - bindir: "%{prefix}/bin" - - config: - configure-commands: ... - - environment: - PKG_CONFIG_PATH=%{libdir}/pkgconfig - - -.. _project_source_overrides: - -Source overrides -~~~~~~~~~~~~~~~~ -The sources dictionary can be used to override source plugin -specific configuration data as shown below. - - -.. code:: yaml - - sources: - - # Override default values for all git sources - git: - - config: - checkout-submodules: False - -.. note:: - - The ``sources`` override is available since :ref:`format version 1 ` - - -.. _project_shell: - -Customizing the shell ---------------------- -Since BuildStream cannot know intimate details about your host or about -the nature of the runtime and software that you are building, the shell -environment for debugging and testing applications may need some help. - -The ``shell`` section allows some customization of the shell environment. - -.. note:: - - The ``shell`` section is available since :ref:`format version 1 ` - - -Interactive shell command -~~~~~~~~~~~~~~~~~~~~~~~~~ -By default, BuildStream will use ``sh -i`` when running an interactive -shell, unless a specific command is given to the ``bst shell`` command. - -BuildStream will automatically set a convenient prompt via the ``PS1`` -environment variable for interactive shells; which might be overwritten -depending on the shell you use in your runtime. - -If you are using ``bash``, we recommend the following configuration to -ensure that the customized prompt is not overwritten: - -.. code:: yaml - - shell: - - # Specify the command to run by default for interactive shells - command: [ 'bash', '--noprofile', '--norc', '-i' ] - - -Environment assignments -~~~~~~~~~~~~~~~~~~~~~~~ -In order to cooperate with your host environment, a debugging shell -sometimes needs to be configured with some extra knowledge inheriting -from your host environment. - -This can be achieved by setting up the shell ``environment`` configuration, -which is expressed as a dictionary very similar to the -:ref:`default environment `, except that it -supports host side environment variable expansion in values. - -.. note:: - - The ``environment`` configuration is available since :ref:`format version 4 ` - -For example, to share your host ``DISPLAY`` and ``DBUS_SESSION_BUS_ADDRESS`` -environments with debugging shells for your project, specify the following: - -.. code:: yaml - - shell: - - # Share some environment variables from the host environment - environment: - DISPLAY: '$DISPLAY' - DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS' - -Or, a more complex example is how one might share the host pulseaudio -server with a ``bst shell`` environment: - -.. code:: yaml - - shell: - - # Set some environment variables explicitly - environment: - PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native' - - -Host files -~~~~~~~~~~ -It can be useful to share some files on the host with a shell so that -it can integrate better with the host environment. - -The ``host-files`` configuration allows one to specify files and -directories on the host to be bind mounted into the sandbox. - -.. note:: - - The ``host-files`` configuration is available since :ref:`format version 4 ` - -.. warning:: - - One should never mount directories where one expects to - find data and files which belong to the user, such as ``/home`` - on POSIX platforms. - - This is because the unsuspecting user may corrupt their own - files accidentally as a result. Instead users can use the - ``--mount`` option of ``bst shell`` to mount data into the shell. - - -The ``host-files`` configuration is an ordered list of *mount specifications*. - -Members of the list can be *fully specified* as a dictionary, or a simple -string can be used if only the defaults are required. - -The fully specified dictionary has the following members: - -* ``path`` - - The path inside the sandbox. This is the only mandatory - member of the mount specification. - -* ``host_path`` - - The host path to mount at ``path`` in the sandbox. This - will default to ``path`` if left unspecified. - -* ``optional`` - - Whether the mount should be considered optional. This - is ``False`` by default. - - -Here is an example of a *fully specified mount specification*: - -.. code:: yaml - - shell: - - # Mount an arbitrary resolv.conf from the host to - # /etc/resolv.conf in the sandbox, and avoid any - # warnings if the host resolv.conf doesnt exist. - host-files: - - host_path: '/usr/local/work/etc/resolv.conf' - path: '/etc/resolv.conf' - optional: True - -Here is an example of using *shorthand mount specifications*: - -.. code:: yaml - - shell: - - # Specify a list of files to mount in the sandbox - # directory from the host. - # - # If these do not exist on the host, a warning will - # be issued but the shell will still be launched. - host-files: - - '/etc/passwd' - - '/etc/group' - - '/etc/resolv.conf' - -Host side environment variable expansion is also supported: - -.. code:: yaml - - shell: - - # Mount a host side pulseaudio server socket into - # the shell environment at the same location. - host-files: - - '${XDG_RUNTIME_DIR}/pulse/native' - - -.. _project_builtin_defaults: - -Builtin defaults ----------------- -BuildStream defines some default values for convenience, the default -values overridden by your project's ``project.conf`` are presented here: - - .. literalinclude:: ../../buildstream/data/projectconfig.yaml - :language: yaml diff --git a/doc/source/projectrefs.rst b/doc/source/projectrefs.rst deleted file mode 100644 index 799b5e761..000000000 --- a/doc/source/projectrefs.rst +++ /dev/null @@ -1,78 +0,0 @@ - - -.. _projectrefs: - -The project.refs file -===================== -If one has elected to store source references in a single ``project.refs`` -file, then it will be stored at the toplevel of your project directory -adjacent to ``project.conf``. This can be configured in your project -using the :ref:`ref-storage configuration ` - -Sources for :mod:`junction ` elements are stored -separately in an adjacent ``junction.refs`` file of the same format. - - -.. _projectrefs_basics: - -Basic behavior --------------- -When a ``project.refs`` file is in use, any source references found -in the :ref:`inline source declarations ` are considered -invalid and will be ignored, and a warning will be emitted for them. - -When ``bst track`` is run for your project, the ``project.refs`` file -will be updated instead of the inline source declarations. In the absence -of a ``project.refs`` file, ``bst track`` will create one automatically -with the tracking results. - -An interesting property of ``project.refs`` is that it allows for -*cross junction tracking*. This is to say that it is possible to override -the *ref* of a given source in a project that your project depends on via -a :mod:`junction `, without actually modifying the -junctioned project. - - -.. _projectrefs_format: - -Format ------- -The ``project.refs`` uses the same YAML format used throughout BuildStream, -and supports the same :ref:`directives ` which apply to -``project.conf`` and element declaration files (i.e. *element.bst* files). - -The ``project.refs`` file format itself is very simple, it contains a single ``projects`` -key at the toplevel, which is a dictionary of :ref:`project names `. -Each *project name* is a dictionary of *element names*, and each *element name* holds -a list of dictionaries corresponding to the element's :ref:`sources `. - - -**Example** - -.. code:: yaml - - # Main toplevel "projects" key - projects: - - # The local project's name is "core" - core: - - # A dictionary of element names - base/automake.bst: - - # A list of sources corresponding to the element - # in the same order in which they were declared. - # - # The values of this list are dictionaries of the - # symbolic "ref" portion understood by the given - # source plugin implementation. - # - - ref: af6ba39142220687c500f79b4aa2f181d9b24e4... - - # The "core" project depends on the "bootstrap" project, - # here we are allowed to override the refs for the projects - # we depend on through junctions. - bootstrap: - - zlib.bst: - - ref: 4ff941449631ace0d4d203e3483be9dbc9da4540... diff --git a/doc/source/public.rst b/doc/source/public.rst deleted file mode 100644 index e640de7bb..000000000 --- a/doc/source/public.rst +++ /dev/null @@ -1,102 +0,0 @@ - -Builtin public data -=================== -Elements can provide public data which can be read by other elements -later in the pipeline, the format for exposing public data on a given -element is :ref:`described here `. - -Any element may use public data for whatever purpose it wants, but -BuildStream has some built-in expectations of public data, which resides -completely in the ``bst`` domain. - -In this section we will describe the public data in the ``bst`` domain. - - -.. _public_integration: - -Integration commands --------------------- - -.. code:: yaml - - # Specify some integration commands - public: - bst: - integration-commands: - - /usr/bin/update-fancy-feature-cache - -The built-in ``integration-commands`` list indicates that depending elements -should run this set of commands before expecting the staged runtime environment -to be functional. - -Typical cases for this include running ``ldconfig`` at the base of a pipeline, -or running commands to update various system caches. - -Integration commands of a given element are automatically run by the -:func:`Element.integrate() ` method -and are used by various plugins. - -Notably the :mod:`BuildElement ` derived classes -will always integrate the build dependencies after staging and before running -any build commands. - - -.. _public_split_rules: - -Split rules ------------ - -.. code:: yaml - - # Specify some split rules - public: - bst: - split-rules: - runtime: - - | - %{bindir}/* - - | - %{sbindir}/* - - | - %{libexecdir}/* - - | - %{libdir}/lib*.so* - -Split rules indicate how the output of an element can be categorized -into *domains*. - -The ``split-rules`` domains are used by the -:func:`Element.stage_artifact() ` -method when deciding what domains of an artifact should be staged. - -The strings listed in each domain are first substituted with the -:ref:`variables ` in context of the given element, and -then applied as a glob style match, as understood by -:func:`utils.glob() ` - -This is used for creating compositions with the :mod:`compose ` -element and can be used by other deployment related elements for the purpose of -splitting element artifacts into separate packages. - - -.. _public_overlap_whitelist: - -Overlap whitelist ------------------ - -The overlap whitelist indicates which files this element is allowed to overlap -over other elements when staged together with other elements. - -Each item in the overlap whitelist has substitutions applied from -:ref:`variables `, and is then applied as a glob-style match -(i.e. :func:`utils.glob() `). - -.. code:: yaml - - public: - bst: - overlap-whitelist: - - | - %{sysconfdir}/* - - | - /etc/fontcache diff --git a/doc/source/sandboxing.rst b/doc/source/sandboxing.rst deleted file mode 100644 index 531a3e5d5..000000000 --- a/doc/source/sandboxing.rst +++ /dev/null @@ -1,200 +0,0 @@ - -.. _sandboxing: - - -Sandboxing -========== - -Introduction ------------- - -BuildStream assembles each element in a *sandbox*. The sandbox is a container -environment which serves two purposes: giving BuildStream control over -all build aspects in order to ensure reproducibility of build results, -and providing safety guarantees for the host system that BuildStream is -running on. - -The exact implementation of the sandbox varies depending on which platform you -are running BuildStream. See below for backend-specific details. - -There are several factors that affect the build output and must therefore be -under BuildStream's control: - -* Filesystem contents and metadata -* The user and permissions model -* Network access -* Device access - -Each of these is detailed below. - -For safety reasons, BuildStream also controls the following things: - -* Access to files outside of the sandbox directory -* Access to certain kernel-specific syscalls - -Creating a sandbox can require special priviliges. This is a safety concern too -because bugs in the `bst` program can cause damage to a host if the program is -running with extra privileges. The exact priviliges that are required depend on -your platform and backend. - -Element plugins can run arbitary commands within the sandbox using the -:mod:`sandbox API `. - -What elements can and can't do in the sandbox ---------------------------------------------- - -This section specifies how BuildStream sandboxes are intended to work. A -specific sandbox provider may not necessarily be able to achieve all of the -requirements listed below so be sure to read the "platform notes" section as -well. - -Filesystem access -~~~~~~~~~~~~~~~~~ - -The filesystem inside sandboxes should be read only during element assembly, -except for certain directories which element plugins can mark as being -read/write. Most elements plugins derive from :mod:`BuildElement -`, which marks ``%{build-root}`` and -``%{install-root}`` as read/write. - -When running integration commands or `bst shell`, the sandbox should have a -fully read-write filesystem. The changes made here do not need to persist -beyond the lifetime of that sandbox, and **must not** affect the contents of -artifacts stored in the cache. - -Certain top level directories should be treated specially in all sandboxes: - -* The ``/dev`` directory should contain device nodes, which are described in - a separate section. - -* The ``/proc`` directory should have a UNIX 'procfs' style filesystem mounted. - It should not expose any information about processes running outside of the - sandbox. - -* The ``/tmp`` directory should be writable. - -Filesystem metadata -~~~~~~~~~~~~~~~~~~~ - -The writable areas inside a BuildStream sandbox are limited in what metadata -can be written and stored. - -* All files must be owned by UID 0 and GID 0 -* No files may have the setuid or setgid bits set -* Extended file attributes (xattrs) cannot be written to or read. -* Hardlinks to other files can be created, but the information about which - files are hardlinked to each other will not be stored in the artifact - that is created from the sandbox. - -These restrictions are due to technical limitations. In future we hope to -support a wider range of filesystem metadata operations. See `issue #38 -`_ for more details. - -User and permissions model -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -All commands inside the sandbox run with user ID 0 and group ID 0. It should -not be possible to become any other user ID. - -Network access -~~~~~~~~~~~~~~ - -Builds should not be able to access the network at all from the sandbox. All -remote resources needed to build an element must be specified in the element's -``sources`` list so that BuildStream is able to see when they have changed. - -A sandbox opened by `bst shell` should allow network access. - -Device access -~~~~~~~~~~~~~ - -Builds should not be able to access any hardware devices at all. - -A few standard UNIX device files are needed, the whitelist is: - -* ``/dev/full`` -* ``/dev/null`` -* ``/dev/urandom`` -* ``/dev/random`` -* ``/dev/zero`` - -It may seem odd that we have sources of randomness in the sandbox, but a lot of -tools do expect them to exist. We take the view that it's up to integrators to -ensure that elements do not deliberately include randomness in their output. - -A sandbox opened by `bst shell` can make any devices available. There needs to -be a console device so that it can be used interactively. - -Platform notes --------------- - -BuildStream currently only carries first-class support for modern Linux-based -operating systems. - -There is also a "fallback" backend which aims to make BuildStream usable on any -POSIX-compatible operating system. The POSIX standard does not provide good -support for creating containers so this implementation makes a number of -unfortunate compromises. - -Linux -~~~~~ - -On Linux we use the following isolation and sandboxing primitives: - -* bind mounts -* FUSE -* Mount namespaces -* Network namespaces -* PID (process ID) namespaces -* User namespaces (if available) -* seccomp - -We access all of these features through a sandboxing tool named `Bubblewrap -`_. - -User namespaces are not enabled by default in all Linux distributions. -BuildStream still runs on such systems but will give a big warning on startup -and will refuse to push any artifacts built on such a system to a remote cache. -For more information, see `issue #92 -`_. - -The Linux platform can operate as a standard user provided user namespace -support is available. If user namespace support is not available you have the -option of installing bubblewrap as a setuid binary to avoid needing to run the -entire ``bst`` process as the ``root`` user. - -The artifact cache on Linux systems is implemented using `OSTree -`_, which can allow us to stage artifacts -using hardlinks instead of copying them. To avoid cache corruption it is -vital that hardlinked files cannot be overwritten. In cases where the root -filesystem inside the sandbox needs to be writable, a custom FUSE filesystem -named SafeHardlinks is used which provides a copy-on-write layer. - -Some of the operations on filesystem metadata listed above are not prohibited -by the sandbox, but will instead be silently dropped when an artifact is -created. For more details see `issue #38 -`_. - -Some details of the host machine are currently leaked by this platform backend. -For more details, see `issue #262 -`_. - -Fallback (POSIX) -~~~~~~~~~~~~~~~~ - -The fallback backend aims to be usable on a wide range of operating systems. -Any OS that implements the POSIX specification and the ``chroot()`` syscall -can be expected to work. There are no real isolation or sandboxing primitives -that work across multiple operating systems, so the protection provided by -this backend is minimal. It would be much safer to use a platform-specific -backend. - -Filesystem isolation is done using the chroot() system call. This system call -requires special privileges to use so ``bst`` usually needs to be run as the -``root`` user when using this backend. - -Network access is not blocked in the sandbox. However since there is unlikely -to be a correct `/etc/resolv.conf` file, any network access that depends on -name resolution will most likely fail anyway. - -Builds inside the sandbox execute as the ``root`` user. diff --git a/doc/source/user_config.rst b/doc/source/user_config.rst deleted file mode 100644 index d02aaf3d4..000000000 --- a/doc/source/user_config.rst +++ /dev/null @@ -1,97 +0,0 @@ - -.. _user_config: - - -User configuration -================== -User configuration and preferences can be specified in a user provided -configuration file, and usually also on the command line. - -Values specified in a user provided configuration file override the -defaults, while command line options take precedence over any other -specified configurations. - - -Configuration file ------------------- -Users can provide a configuration file to override parameters in -the default configuration. - -Unless a configuration file is explicitly specified on the command line when -invoking ``bst``, an attempt is made to load user specific configuration from -``$XDG_CONFIG_HOME/buildstream.conf``. On most Linux based systems, the location -will be ``~/.config/buildstream.conf`` - - -Project specific value ----------------------- -The ``projects`` key can be used to specify project specific configurations, -the supported configurations on a project wide basis are listed here. - -.. _config_artifacts: - -Artifact server -~~~~~~~~~~~~~~~ -The project you build will often specify a :ref:`remote artifact cache -` already, but you may want to specify extra caches. There are two -ways to do this. You can add one or more global caches: - -**Example** - -.. code:: yaml - - artifacts: - url: https://artifacts.com/artifacts - -Caches listed there will be considered lower priority than those specified -by the project configuration. - -You can also add project-specific caches: - -**Example** - -.. code:: yaml - - projects: - project-name: - artifacts: - - url: https://artifacts.com/artifacts1 - - url: ssh://user@artifacts.com/artifacts2 - push: true - -Caches listed here will be considered higher priority than those specified -by the project. - -If you give a list of URLs, earlier entries in the list will have higher -priority than later ones. - -Strict build plan -~~~~~~~~~~~~~~~~~ -The strict build plan option decides whether you want elements -to rebuild when their dependencies have changed. This is enabled -by default, but recommended to turn off in developer scenarios where -you might want to build a large system and test it quickly after -modifying some low level component. - - -**Example** - -.. code:: yaml - - projects: - project-name: - strict: False - - -.. note:: - - It is always possible to override this at invocation time using - the ``--strict`` and ``--no-strict`` command line options. - - -Default configuration ---------------------- -The default BuildStream configuration is specified here for reference: - - .. literalinclude:: ../../buildstream/data/userconfig.yaml - :language: yaml diff --git a/doc/source/using_commands.rst b/doc/source/using_commands.rst new file mode 100644 index 000000000..18affc6e5 --- /dev/null +++ b/doc/source/using_commands.rst @@ -0,0 +1,127 @@ + +.. _commands: + +Commands +======== +This page contains documentation for each BuildStream command, +along with their possible options and arguments. Each command can be +invoked on the command line, where, in most cases, this will be from the +project's main directory. + + +---- + +.. The bst options e.g. bst --version, or bst --verbose etc. +.. _invoking_bst: + +.. click:: buildstream._frontend:cli + :prog: bst + +.. Further description of the command goes here + +---- + +.. the `bst init` command +.. _invoking_init: + +.. click:: buildstream._frontend.cli:init + :prog: bst init + +---- + +.. the `bst build` command +.. _invoking_build: + +.. click:: buildstream._frontend.cli:build + :prog: bst build + +---- + +.. _invoking_fetch: + +.. click:: buildstream._frontend.cli:fetch + :prog: bst fetch + +---- + +.. _invoking_track: + +.. click:: buildstream._frontend.cli:track + :prog: bst track + +---- + +.. _invoking_pull: + +.. click:: buildstream._frontend.cli:pull + :prog: bst pull + +---- + +.. _invoking_push: + +.. click:: buildstream._frontend.cli:push + :prog: bst push + +---- + +.. _invoking_show: + +.. click:: buildstream._frontend.cli:show + :prog: bst show + +---- + +.. _invoking_shell: + +.. click:: buildstream._frontend.cli:shell + :prog: bst shell + +---- + +.. _invoking_checkout: + +.. click:: buildstream._frontend.cli:checkout + :prog: bst checkout + +---- + +.. _invoking_source_bundle: + +.. click:: buildstream._frontend.cli:source_bundle + :prog: bst source bundle + +---- + +.. _invoking_workspace: + +.. click:: buildstream._frontend.cli:workspace + :prog: bst workspace + +---- + +.. _invoking_workspace_open: + +.. click:: buildstream._frontend.cli:workspace_open + :prog: bst workspace open + +---- + +.. _invoking_workspace_close: + +.. click:: buildstream._frontend.cli:workspace_close + :prog: bst workspace close + +---- + +.. _invoking_workspace_reset: + +.. click:: buildstream._frontend.cli:workspace_reset + :prog: bst workspace reset + +---- + +.. _invoking_workspace_list: + +.. click:: buildstream._frontend.cli:workspace_list + :prog: bst workspace list diff --git a/doc/source/using_config.rst b/doc/source/using_config.rst new file mode 100644 index 000000000..d02aaf3d4 --- /dev/null +++ b/doc/source/using_config.rst @@ -0,0 +1,97 @@ + +.. _user_config: + + +User configuration +================== +User configuration and preferences can be specified in a user provided +configuration file, and usually also on the command line. + +Values specified in a user provided configuration file override the +defaults, while command line options take precedence over any other +specified configurations. + + +Configuration file +------------------ +Users can provide a configuration file to override parameters in +the default configuration. + +Unless a configuration file is explicitly specified on the command line when +invoking ``bst``, an attempt is made to load user specific configuration from +``$XDG_CONFIG_HOME/buildstream.conf``. On most Linux based systems, the location +will be ``~/.config/buildstream.conf`` + + +Project specific value +---------------------- +The ``projects`` key can be used to specify project specific configurations, +the supported configurations on a project wide basis are listed here. + +.. _config_artifacts: + +Artifact server +~~~~~~~~~~~~~~~ +The project you build will often specify a :ref:`remote artifact cache +` already, but you may want to specify extra caches. There are two +ways to do this. You can add one or more global caches: + +**Example** + +.. code:: yaml + + artifacts: + url: https://artifacts.com/artifacts + +Caches listed there will be considered lower priority than those specified +by the project configuration. + +You can also add project-specific caches: + +**Example** + +.. code:: yaml + + projects: + project-name: + artifacts: + - url: https://artifacts.com/artifacts1 + - url: ssh://user@artifacts.com/artifacts2 + push: true + +Caches listed here will be considered higher priority than those specified +by the project. + +If you give a list of URLs, earlier entries in the list will have higher +priority than later ones. + +Strict build plan +~~~~~~~~~~~~~~~~~ +The strict build plan option decides whether you want elements +to rebuild when their dependencies have changed. This is enabled +by default, but recommended to turn off in developer scenarios where +you might want to build a large system and test it quickly after +modifying some low level component. + + +**Example** + +.. code:: yaml + + projects: + project-name: + strict: False + + +.. note:: + + It is always possible to override this at invocation time using + the ``--strict`` and ``--no-strict`` command line options. + + +Default configuration +--------------------- +The default BuildStream configuration is specified here for reference: + + .. literalinclude:: ../../buildstream/data/userconfig.yaml + :language: yaml diff --git a/doc/source/using_examples.rst b/doc/source/using_examples.rst new file mode 100644 index 000000000..aa100d007 --- /dev/null +++ b/doc/source/using_examples.rst @@ -0,0 +1,12 @@ + +Examples +======== +This page contains documentation for real examples of BuildStream projects, +described step by step. All run under CI, so you can trust they are +maintained and work as expected. + + +.. toctree:: + :maxdepth: 1 + + examples/flatpak-autotools -- cgit v1.2.1