Add a walkthrough on creating a project from scratchjonathan/doc-project-from-scratch

2 files changed, 184 insertions, 0 deletions

diff --git a/doc/source/index.rst b/doc/source/index.rst
index 70c7354fe..1d8c4414a 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -37,6 +37,7 @@ off the ground installing it on a new host.
 * :ref:`format`
 * :ref:`public`
 * :ref:`artifacts`
+* :ref:`project_from_scratch`
 
 
 Elements
diff --git a/doc/source/project-from-scratch.rst b/doc/source/project-from-scratch.rst
new file mode 100644
index 000000000..ae135b8cb
--- /dev/null
+++ b/doc/source/project-from-scratch.rst
@@ -0,0 +1,183 @@
+.. _project_from_scratch:
+
+How to set up a buildstream repo
+================================
+
+Overview
+--------
+
+Buildstream operates out of a directory that contains a file named "project.conf", and contains within it a collection of ".bst" files which defines how to build an artifact.
+For convenience, we put "project.conf" in the root of a git repository, and all of the ".bst" files are arranged in a subdirectory.
+
+Creating the git repository
+---------------------------
+
+If you are using a git hosting service, follow their instructions for how to create a git repository.
+
+If not:
+
+1. Create your directory, e.g.
+   ::
+       mkdir buildstream-repo
+
+2. Initialize the git repo, e.g.
+   ::
+       cd buildstream-repo
+       git init
+
+3. Set the remote to push to/pull from. For example, if there's a git server serving a repository to "git://git.example.org/buildstream-repo", the command to set the remote would be
+   ::
+       git remote add origin git://git.example.org/buildstream-repo
+
+Setting up project.conf
+-----------------------
+
+The bare minimum a project.conf requires is a "name" and an "element-path".
+
+"name" is the name of the project. This is used to make sure that artifacts from separate projects don't get mixed together in the cache.
+
+"element-path" is the path to where the elements are stored, relative to the directory that contains project.conf. In the name of tidiness, this is usually a subdirectory called "elements", but for simplicity, we'll set it to the current directory.
+
+minimal project.conf
+~~~~~~~~~~~~~~~~~~~~
+So a simple project.conf would look like ::
+
+  cat >project.conf <<EOF
+  name: test
+  element-path: elements
+  EOF
+
+This specifies that elements live in a subdirectory called "elements" ::
+
+  mkdir elements
+
+Creating elements
+-----------------
+
+There are a wide variety of elements at your disposal, full documentation of all the elements is at https://buildstream.gitlab.io/buildstream/#elements.
+
+Ultimately, the lowest-level element of buildstream is an "import" element. This is because buildstream scrupulously prevents the host's environment from affecting builds by running everything inside a sandbox. The downside of this is that they need a suite of tools to be provided before they can do anything.
+
+The "import" element is used to take a source and provide the files in that source as an artifact.
+
+minimal tar import element
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For example, a minimal build environment hosted somewhere as a tarball ::
+
+  cat > elements/base.bst <<EOF
+  kind: import
+  description: Import a tarball as a build environment
+  sources:
+  - kind: tar
+    url: http://www.example.com/x86-build-env.tar.gz
+    ref: 0123456789abcdef
+  EOF
+
+arch-specific tar import element
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If this repository is going to be used for multiple architectures, it would look like ::
+
+  cat > elements/base.bst <<EOF
+  kind: import
+  description: Import a tarball as a build environment
+  arches:
+    x86_64:
+      sources:
+      - kind: tar
+        url: http://www.example.com/x86_64-build-env.tar.gz
+        ref: 0123456789abcdef
+    i386:
+      sources:
+      - kind: tar
+        url: http://www.example.com/x86_32-build-env.tar.gz
+        ref: fdecba987654321
+  EOF
+
+ostree import element
+~~~~~~~~~~~~~~~~~~~~~
+
+Buildstream typically uses an ostree repo for the minimal build environment, which looks like ::
+
+  cat > elements/ostree-base.bst <<EOF
+  kind: import
+  description: Import the base freedesktop platform
+  environment:
+    PATH: /tools/bin:/tools/sbin:/usr/bin:/bin:/usr/sbin:/sbin
+  public:
+    bst:
+      integration-commands:
+      - ldconfig
+  host-arches:
+    x86_64:
+      sources:
+      - kind: ostree
+        url: https://ostree.baserock.org/cache/
+        track: baserock/bootstrap-stage3-sysroot/12c20460fb3c3c50d0ed9133aa19839a89626c0d66736c439c3deb0b66263684
+        ref: 4788d14185c415c9cef20a1d36286d792dac7a3271504e21c6903987221bfccd
+      config:
+        source: files
+    ppc64b:
+      sources:
+      - kind: ostree
+        url: https://ostree.baserock.org/cache/
+        track: baserock/bootstrap-stage3-sysroot/04e669a8a1b0252ac6307dc268afc4e5f472baeec8ba664bdccae9e612c86d69
+        ref: 4bbdb9fff190f52d5534efe4e2f35ef701cf741254399639bcf9c52c94e5f030
+      config:
+        source: files
+  EOF
+
+A test element
+~~~~~~~~~~~~~~
+
+Now that we have a build environment, we can start creating elements. For this experiment, I'll write an element using shell scripts ::
+
+  cat > elements/test-element.bst <<EOF
+  kind: manual
+  depends:
+  - filename: ostree-base.bst
+    type: build
+  config:
+    install-commands:
+    - "mkdir -p %{install-root}"
+    - "echo hello > %{install-root}/hello"
+  EOF
+
+Because test-element depends on ostree-base.bst, it will include that in the sandbox, providing enough tools to run shell commands like "mkdir" and "echo".
+
+Running ``bst build test-element.bst`` will create an artifact in the cache that contains the file "hello". Note that all element paths are relative to the "elements" subdir.
+
+You can inspect this file by checking out the element, e.g. ``bst checkout test-element.bst test`` will create a directory named "test", which contains a file called "hello"
+
+A build element
+~~~~~~~~~~~~~~~
+
+In the real world, you'll be building real elements. The majority of the time, they build with a well-defined build system (e.g. autotools, cmake, qmake), will be made of a single source, and will have a number of dependencies. ::
+
+  cat > elements/example-element.bst <<EOF
+  # An element's kind determines its behaviour. "autotools" elements will try
+  # to build the sources as if they are source code for autotools projects.
+  kind: autotools
+  description: An example of a build element.
+  # This element depends on "base" to provide the necessary tools to compile
+  # GNU hello. "type: build" means that "base" will only be imported into the
+  # staging area when building this element.
+  depends:
+  - filename: ostree-base.bst
+    type: build
+  # We fetch a tar from gnu.org to build, ref is a sha256sum of the expected file,
+  # so that you won't start building something unexpected.
+  sources:
+  - kind: tar
+    url: http://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz
+    ref: 31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b
+  EOF
+
+Creating your own build environment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In many cases, you don't want to use the build environment provided in a "base" element. For example, if you want your system to use a specific libc that isn't provided by "base". 
+
+In cases like these, you will want to build a new build environment from scratch inside buildstream.
+For an example of this, see https://gitlab.com/baserock/definitions/tree/master/elements.