diff options
Diffstat (limited to 'doc/source/arch_data_model.rst')
-rw-r--r-- | doc/source/arch_data_model.rst | 157 |
1 files changed, 157 insertions, 0 deletions
diff --git a/doc/source/arch_data_model.rst b/doc/source/arch_data_model.rst new file mode 100644 index 000000000..467859a7d --- /dev/null +++ b/doc/source/arch_data_model.rst @@ -0,0 +1,157 @@ + + +Data model +========== +This section details the data model on which the BuildStream core operates. This +includes an overview of the project data model which is BuildStream's main input, +the user preferences, and local state. + + +Project +------- +The ``Project`` object is the main component of a given BuildStream *project*, and +is responsible for loading and validating the :ref:`project.conf <projectconf>`, and +providing this loaded *project data* in a convenient way to the BuildStream core. + +Conceptually, the *project* is a container for the :mod:`Elements <buildstream.element>`, +which are declared within a user's project, and as such acts as a factory for instantiating +elements at load time. + + +Element +------- +:mod:`Elements <buildstream.element>` are the main processing unit in a pipeline. These +are the loaded representation of the ``.bst`` files loaded from the :ref:`project's element path +<project_element_path>`. + +The *Element* is an abstract base class which cannot do anything on its own, its +concrete class is defined by *plugins* which are either included in the BuildStream +:ref:`core set of plugins <plugins>` or loaded from external sources :ref:`defined by the project +<project_plugins>` + +The responsibilities of an element include: + +* Loading the element's configuration from the core provided dictionary +* Providing a unique key for any element specific configuration which might + effect the output produced by the element +* Configuring the sandbox +* Staging the data into the sandbox, which might include Sources and + the outputs of previous elements +* Assembling the output *artifact* + + +Element data structure +~~~~~~~~~~~~~~~~~~~~~~ +The properties of an element are a composition of what the BuildStream core understands, +the configurations exposed by the Element plugin, and free form data which allows +annotations and configurations which can be read back by reverse dependencies during +processing, as illustrated here: + +.. image:: images/arch-datamodel-element.svg + :align: center + + +Element composition +~~~~~~~~~~~~~~~~~~~ +The element is composed of configurations which are sourced from various entry +points using the low level YAML utilities. + +This composition takes place after :ref:`includes <format_directives_include>` and +:ref:`conditional <format_directives_conditional>` directives are processed, while +:ref:`list composition <format_directives_list_prepend>` directives are processed +as a result of this composition. + +Here is a diagram showing which sources take precedence in the composition process +which results in the final element configuration being resolved: + +.. image:: images/arch-datamodel-element-composition.svg + :align: center + +Note that not all *BuildStream Core Data* is understood by the *Element*, but a great +deal of configurations understood by the *Element* is also understood by the core and +has default configurations built into BuildStream and configurable with the project +configuration. These include values such as *variables*, *environment*, *sandbox*, etc. + +As shown above, composition is performed in two stages, as we only need to composite +the data from the toplevel element declaration against the composition of previous +stages every time we instantiate an element. + + +Source +------ +:mod:`Sources <buildstream.element>` are the abstract objects which are responsible +for obtaining remote source code or data to import into the build environment, and +ensuring that it is done in a bit-for-bit reproducible way without any contamination +of the host or build environment. + +This is to say that: + +* User configuration on the host, or filesystem outside of BuildStream designated + directories, must never be modified as a side effect of running BuildStream. + +* When the Source uses host tools, host side configurations must never result in + deviations of what is staged to a build directory. The Source must behave exactly + the same way regardless of host side configurations. + +The responsibilities of a source include: + +* Loading the source's configuration from the core provided dictionary +* Providing a unique key for any source specific configuration which might + effect the staged source +* Implement discovery of new versions of the source upstream (referred to as *"tracking"*) +* Staging the unpacked source to a given directory +* Preparing workspaces + + +Source data structure +~~~~~~~~~~~~~~~~~~~~~ +Similar to the *Element*, the properties of a source are a composition of what +the BuildStream core understands and the configurations exposed by the Source +plugin: + +.. image:: images/arch-datamodel-source.svg + :align: center + +.. note:: + + In .bst files, the BuildStream core configurations and Source specific configurations + share the same dictionary. + + Strictly speaking this is limiting, but provides a measure of convenience as .bst + files are a bit less wordy to express. + + +Source composition +~~~~~~~~~~~~~~~~~~ +Source composition is much simpler than Element composition, because defaults +cannot be specified at the project level, excepting for Source type specific +value overrides. + +.. image:: images/arch-datamodel-source-composition.svg + :align: center + + +Context +------- +The Context object is a very centric part of the BuildStream data model, and is +not a part of the Project data described above but rather is where we load and +store all of the user preferences. + +User preferences are sourced from various locations, but usually have a default, +an option in the user configuration file, and an option to override it on the +command line. + +.. image:: images/arch-datamodel-context.svg + :align: center + +Asides from being a focal point for loading and storing all user configuration, +the Context object also plays a central role in the logging framework. + + +Workspaces +---------- +The Workspaces object is yet another kind of state. Unlike the Context and +the Project data model, the Workspaces object loads, saves and stores in +memory the local state regarding a user's active and open workspaces. + +These are stored in the local state ``.bst/`` subdirectory of users projects. |