1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
|
.. _handling_files_composition:
Composition
===========
In this chapter we will explore how to create *compositions* of multiple
input filesystem trees, using the :mod:`compose <elements.compose>` element.
.. note::
This example is distributed with BuildStream
in the `doc/examples/composition
<https://gitlab.com/BuildStream/buildstream/tree/master/doc/examples/composition>`_
subdirectory.
Overview
--------
Composing a directory tree based on a set of build dependencies is often
one of the important steps you might perform in order to create a single artifact
which can be checked out and deployed.
In order to use the :mod:`compose <elements.compose>` element, it is important
to first understand the concept of :ref:`split rules <public_split_rules>`, which
we will cover in this chapter.
Introducing split rules
~~~~~~~~~~~~~~~~~~~~~~~
The :ref:`split rules <public_split_rules>` of an element declaration denote
which sets of files in the given element's resulting artifact belong to which
*domain name*.
The *domains* can then be used in various ways, using plugins which understand
*split rule domains*.
BuildStream's :ref:`default project configuration <project_builtin_defaults>`
contains a sensible set of default *split rule domains* for the purpose of
artifact splitting, they can be overridden in :ref:`your project.conf <project_split_rules>`,
and finally on a per element basis in the :ref:`public data <public_builtin>`
of your element declarations.
.. note::
Projects are free to add additional *split rule domains* on top of the
default domains provided by the default project configuration.
There is nothing wrong with defining split rule domains which *overlap*,
possibly capturing some of the same files also captured by another
*domain*, however you should be aware of this when later using your
split rules with a plugin which processes them, like the
:mod:`compose <elements.compose>` element described in this chapter.
Example of split rule declaration
'''''''''''''''''''''''''''''''''
In an element, you might need to define or extend the ``split-rules``
in order to capture files in custom locations in a logical *domain*.
Here is an example of how you might use the
:ref:`list append directive <format_directives_list_append>`
to append an additional rule to your ``split-rules`` list in order to
capture additional data files which your application or library might
want to include in the *runtime domain*:
.. code:: yaml
# Add our .dat files to the runtime domain
public:
bst:
split-rules:
runtime:
(>):
- |
%{datadir}/foo/*.dat
Split rules are absolute paths which denote files within an artifact's root
directory. The globbing patterns supported in split rules are defined in the
:func:`reference documentation here <buildstream.utils.glob>`.
.. important::
Note that because of variable expansion, split rules can often be
*resolved differently* for elements which have overridden path
related variables, like ``%{prefix}``.
This usually means that you do not need to explicitly extend or override
split rules on a specific element unless your element installs files to
special case locations.
Project structure
-----------------
In this example we expand on the chapter about
:ref:`integration commands <tutorial_integration_commands>`, so we will
only discuss the files which are added or changed from that example.
``elements/base/alpine.bst``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: ../../examples/composition/elements/base/alpine.bst
:language: yaml
Here we have modified the base runtime, so as to specify that for this
element, we want to also include the runtime linker into the *runtime domain*.
``elements/runtime-only.bst``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: ../../examples/composition/elements/runtime-only.bst
:language: yaml
As we can see, this :mod:`compose <elements.compose>` element has
been configured to only include files from the *runtime domain*.
Using the project
-----------------
Now that we've presented how :ref:`split rules <public_split_rules>`
work and shown how to use them in the context of this example, lets
use the :mod:`compose <elements.compose>` element we've created and
observe the results.
Building the project
~~~~~~~~~~~~~~~~~~~~
.. raw:: html
:file: ../sessions/composition-build.html
As you can see in the output, this composition has only a few hundred
files, but the complete ``alpine.bst`` runtime has several thousand
files.
List the content
~~~~~~~~~~~~~~~~
At the risk of this being a long list, let's :ref:`list the
contents of this artifact <invoking_artifact_list_contents>`
.. raw:: html
:file: ../sessions/composition-list-contents.html
Some things to observe here:
* The list does include the ``/usr/bin/hello`` program and
also the ``/usr/lib/libhello.so`` shared library.
These paths are both captured by the default split rules
for the *runtime domain*.
* The list does not include the ``/usr/include/libhello.h``
header file which was used to compile ``/usr/bin/hello``.
The header file is not captured by the *runtime domain*
by default. It is however captured by the *devel domain*.
* The runtime linker ``/lib/ld-musl-x86_64.so.1``, as this was
explicitly added to the *runtime domain* for the ``base/alpine.bst``
element which provides this file.
.. tip::
The reader at this time might want to list the content of
other elements built from this project, such as the
``hello.bst`` element by itself, or the ``base/alpine.bst``
element.
Run the program
~~~~~~~~~~~~~~~
Finally, lets just run the program we built.
.. raw:: html
:file: ../sessions/composition-shell.html
Here we can see that we at least have the required files to run
our hello world program, however we would not have if we were
missing the runtime linker which we added in ``base/alpine.bst``.
Summary
-------
In this chapter we've gotten familiar with :ref:`split rules <public_split_rules>`
annotations, and we've learned enough about the :mod:`compose <elements.compose>`
element such that we can start creating our own compositions using
*split domains*.
We've also used the :ref:`list append directive <format_directives_list_append>`
and we are now observing the contents of artifacts using
:ref:`bst artifact list-contents <invoking_artifact_list_contents>`.
|
