From 5c20869895181cd5fe87f8155daab22d5d118db4 Mon Sep 17 00:00:00 2001 From: Jim MacArthur Date: Fri, 19 Oct 2018 10:14:51 +0100 Subject: Docs: Add remote execution architecture documentation. --- doc/source/arch_remote_execution.rst | 32 ++ doc/source/image-sources/arch-remote-execution.odg | Bin 0 -> 17662 bytes doc/source/images/arch-remote-execution.svg | 499 +++++++++++++++++++++ doc/source/main_architecture.rst | 2 + 4 files changed, 533 insertions(+) create mode 100644 doc/source/arch_remote_execution.rst create mode 100644 doc/source/image-sources/arch-remote-execution.odg create mode 100644 doc/source/images/arch-remote-execution.svg diff --git a/doc/source/arch_remote_execution.rst b/doc/source/arch_remote_execution.rst new file mode 100644 index 000000000..4e66d7719 --- /dev/null +++ b/doc/source/arch_remote_execution.rst @@ -0,0 +1,32 @@ +Remote execution +================ +Remote execution allows builds to take place on different machines from the machine *bst* is run on, allowing faster builds, shared resources and different build capabilities, for example different machine architectures. + +Sandbox extension +~~~~~~~~~~~~~~~~~ +The previous section :ref:`sandboxing` describes the two forms of local sandbox: the *bubblewrap* sandbox and the less common *chroot* sandbox (which is used on non-Linux POSIX environments). Remote execution uses a third type, the *remote sandbox*, which functions similarly to the local sandbox types, but is responsible for causing the build to occur on a remote system. Remote sandboxes should produce the same result as local sandboxes. + +Artifact caches and other storage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +BuildStream can transmit the results of local builds to remote artifact caches and retrieve them later. The same mechanism is used for remote execution. The filesystem staged before building is stored in a local content-addressable store, which may be the same as the local artifact cache. The command to execute is also stored as an object in local CAS. Both the initial source filesystem and command are transmitted to remote storage specific to the remote execution service, and after the build is complete, the filesystem after build is retrieved from remote storage to the local CAS. The remote execution service uses the same communication protocol as artifact caches, and may use the same internal storage, but may not implement the extensions used by BuildStream to store full artifacts. + +.. image:: images/arch-remote-execution.svg + :align: center + +After sending the prerequisite file system and command to the remote execution cache, BuildStream uses the *Remote Execution API (REAPI)* [1]_ to signal to the build server that it should perform a build. +How the build service does this is not BuildStream's concern, but typically there will be a worker that will be assigned the work, retrieve the inputs of the build from CAS, carry out the build, and store the results in CAS. The remote execution service is entitled to return a cached result, if the filesystem and command have already been executed. + + +BuildStream will continue to poll the remote execution server until the build is completed or lost. If it's completed (successfully or otherwise) the resulting objects (typically the finished file system and logs of stdout and stderr) will be pulled to the local cache. BuildStream will retry jobs that are lost by the remote build server or which complete with certain error types. + +*sandboxremote.py* contains all the communication with the remote execution API. + +After a successful build, BuildStream will push the completed artifact to the remote artifact servers as if it had built it locally. + +Use of sandboxes outside builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Because staging requires *integration-commands* to be run, a sandbox is necessary for the *bst checkout* command to run. + +A sandbox is also required for the *bst shell* command. Because the REAPI does not provide any mechanism for interactive use, *bst shell* will always use a local sandbox. + +.. [1] See https://github.com/bazelbuild/remote-apis. diff --git a/doc/source/image-sources/arch-remote-execution.odg b/doc/source/image-sources/arch-remote-execution.odg new file mode 100644 index 000000000..4e92f992b Binary files /dev/null and b/doc/source/image-sources/arch-remote-execution.odg differ diff --git a/doc/source/images/arch-remote-execution.svg b/doc/source/images/arch-remote-execution.svg new file mode 100644 index 000000000..e55bebb2d --- /dev/null +++ b/doc/source/images/arch-remote-execution.svg @@ -0,0 +1,499 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Remote Execution service + + + + + + + + + + + + + Local artifact cache + + + + + + push + + + + + + + + + + + + + + REAPI + + + + + + + + Remote artifact cache 1 + + + + + + + + + + + + + + + + + + + + pull + + + + + + + + Remote artifact cache 2 + + + + + + push + + + + + + + + + + + + + + + + + + + + pull + + + + + + + + + + + + + + Send (initial environments, commands) + + + + + + + Retrieve (finished environments, logs) + + + + + + + + Cache manager(cascache.py) + + + + + + + + Sandbox.run(commands, flags, environment) + + + + + + + + Remote execution cache + + + + + + + + REAPI-compatible build service + + + + + + BuildStream + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Remote execution service implementation + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/source/main_architecture.rst b/doc/source/main_architecture.rst index 55b607dae..b0a117ed9 100644 --- a/doc/source/main_architecture.rst +++ b/doc/source/main_architecture.rst @@ -15,3 +15,5 @@ This section provides details on the overall BuildStream architecture. arch_scheduler arch_cachekeys arch_sandboxing + arch_remote_execution + -- cgit v1.2.1