summaryrefslogtreecommitdiff
path: root/yarns/deployment.yarn
blob: 6ec8c0af53ecc633d2c0ab122257138d958d1965 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
Morph Deployment Tests
======================

    SCENARIO deploying a non-cluster morphology
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    AND the user attempts to deploy the system systems/test-system.morph in branch master
    THEN morph failed
    AND the deploy error message includes the string "morph deployment commands are only supported for cluster morphologies"
    FINALLY the git server is shut down

    SCENARIO deploying a cluster morphology as a tarfile
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    AND the user builds the system systems/test-system.morph in branch master
    AND the user attempts to deploy the cluster clusters/test-cluster.morph in branch master
    THEN morph succeeded
    FINALLY the git server is shut down

    SCENARIO deploying a cluster using a relative path
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    AND the user builds the system systems/test-system.morph in branch master
    AND from directory workspace/master/test/morphs/clusters the user attempts to deploy the cluster test-cluster.morph in branch master
    THEN morph succeeded
    FINALLY the git server is shut down

    SCENARIO deploying a cluster using a relative path (second variant)
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    AND the user builds the system systems/test-system.morph in branch master
    AND from directory workspace/master/test/morphs/clusters the user attempts to deploy the cluster ../clusters/test-cluster.morph in branch master
    THEN morph succeeded
    FINALLY the git server is shut down

    SCENARIO deploying a cluster using an absolute path
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    AND the user builds the system systems/test-system.morph in branch master
    AND from directory workspace/master/test/morphs/clusters the user attempts to deploy the cluster using the absolute path to test-cluster.morph in branch master
    THEN morph succeeded
    FINALLY the git server is shut down

Some deployment types support upgrades, but some do not and Morph needs to make
this clear.

    SCENARIO attempting to upgrade a tarfile deployment
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    AND the user builds the system systems/test-system.morph in branch master
    AND the user attempts to upgrade the cluster clusters/test-cluster.morph in branch master
    THEN morph failed
    FINALLY the git server is shut down

The rawdisk write extension supports both initial deployment and subsequent
upgrades. Note that the rawdisk upgrade code needs bringing up to date to use
the new Baserock OS version manager tool. Also, the test deploys an identical
base OS as an upgrade. While pointless, this is permitted and does exercise
the same code paths as a real upgrade.

    SCENARIO deploying a cluster morphology as rawdisk and then upgrading it
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    GIVEN a cluster called test-cluster.morph in system branch master
    AND a system in cluster test-cluster.morph in branch master called test-system
    AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph
    AND system test-system in cluster test-cluster.morph in branch master has deployment type: rawdisk
    AND system test-system in cluster test-cluster.morph in branch master has deployment location: test.tar
    WHEN the user builds the system systems/test-system.morph in branch master
    AND the user attempts to deploy the cluster test-cluster.morph in branch master with options test-system.DISK_SIZE=20M test-system.VERSION_LABEL=test1
    THEN morph succeeded
    WHEN the user attempts to upgrade the cluster test-cluster.morph in branch master with options test-system.VERSION_LABEL=test2
    THEN morph succeeded
    FINALLY the git server is shut down

Nested deployments
==================

For the use-cases of:

1.  Installer CD/USB
2.  NFS/VM host
3.  System with multiple containerised applications
4.  System with a toolchain targetting the sysroot of another target
5.  Any nested combination of the above

It is convenient to be able to deploy one system inside another.

    SCENARIO deploying a cluster morphology with nested systems
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    GIVEN a cluster called test-cluster.morph in system branch master
    AND a system in cluster test-cluster.morph in branch master called test-system
    AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph
    AND system test-system in cluster test-cluster.morph in branch master has deployment type: tar

After the usual setup, we also add a subsystem to the cluster.

    GIVEN a subsystem in cluster test-cluster.morph in branch master called test-system.sysroot
    AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master builds systems/test-system.morph
    AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master has deployment type: sysroot

We specify the location as a file path, this is relative to the parent
system's extracted rootfs, before it is configured.

    AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master has deployment location: var/lib/sysroots/test-system
    WHEN the user builds the system systems/test-system.morph in branch master
    AND the user attempts to deploy the cluster test-cluster.morph in branch master with options test-system.location="$DATADIR/test.tar"
    THEN morph succeeded

Morph succeeding alone is not sufficient to check whether it actually
worked, since if it ignored the subsystems field, or got the location
wrong for the subsystem. To actually test it, we have to check that our
deployed system contains the other. Since the baserock directory is in
every system, we can check for that.

    AND tarball test.tar contains var/lib/sysroots/test-system/baserock
    FINALLY the git server is shut down

Initramfs deployments
=====================

There's a few ways of creating an initramfs. We could:
1.  Build a sysroot and:
    1.  Have a chunk turn that into a cpio archive, written into /boot.
    2.  Embed it in the Linux kernel image, having the initramfs as part
        of the BSP.
2.  Deploy an existing system as a cpio archive
    1.  As a stand-alone system, without a rootfs
    2.  Nested inside another system

1.1 and 1.2 require system engineering work, so won't be mentioned here.

    SCENARIO deploying a system with an initramfs
    ASSUMING there is space for 5 512M disk images
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    GIVEN a cluster called C.morph in system branch master
    AND a system in cluster C.morph in branch master called S

2.2 needs a nested system that is deployed with the initramfs write
extension.

    GIVEN a subsystem in cluster C.morph in branch master called S.I
    AND subsystem S.I in cluster C.morph in branch master builds systems/test-system.morph
    AND subsystem S.I in cluster C.morph in branch master has deployment type: initramfs

The nested system needs to be placed somewhere in the parent. The
traditional place for an initramfs is `/boot`.

    AND subsystem S.I in cluster C.morph in branch master has deployment location: boot/initramfs.gz

1.1 and 2.2 need the write extension to configure the boot-loader to
use the produced initramfs. Only write extensions that involve creating a disk image care, so we'll use `rawdisk.write`.

    GIVEN system S in cluster C.morph in branch master builds systems/test-system.morph
    AND system S in cluster C.morph in branch master has deployment type: rawdisk
    AND system S in cluster C.morph in branch master has deployment location: test.img
    AND system S in cluster C.morph in branch master has deployment variable: DISK_SIZE=512M

Initramfs support is triggered by the `INITRAMFS_PATH` variable. It could have been made automatic, triggering the behaviour if `/boot/initramfs.gz` exists, but:

1.  There are a bunch of possible names, some of which imply different formats.
2.  If we decide on one specific name, how do we pick.
3.  If we allow multiple possible names, how do we handle multiple being possible.
4.  We may need to pick a non-standard name: e.g. We have a deployment
    where the system loads a kernel and initramfs from a disk, then boots
    the target in KVM, so the bootloader we want to use for the guest is
    `initramfs.gz`, while the host's initramfs is `hyp-initramfs.gz`.
5.  We may have the initramfs come from a chunk the system built, but
    for speed, we want this particular deployment not to use an initramfs,
    even though we have a generic image that may support one.

For all these reasons, despite there being redundancy in some cases,
we're going to set `INITRAMFS_PATH` to the same as the nested deployment's
location.

    GIVEN system S in cluster C.morph in branch master has deployment variable: INITRAMFS_PATH=boot/initramfs.gz

Fully testing that the system is bootable requires a lot more time,
infrastructure and dependencies, so we're just going to build it and
inspect the result of the deployment.

    WHEN the user builds the system systems/test-system.morph in branch master
    AND the user attempts to deploy the cluster C.morph in branch master
    THEN morph succeeded
    AND file workspace/master/test/morphs/test.img exists

If the initramfs write extension works, the rootfs image should contain
`boot/initramfs.gz`.

    WHEN disk image workspace/master/test/morphs/test.img is mounted at mnt
    THEN file mnt/systems/default/run/boot/initramfs.gz exists

If the `rawdisk` write extension worked, then the bootloader config file
will mention the initramfs, and the UUID of the disk.

    AND file mnt/extlinux.conf matches initramfs
    AND file mnt/extlinux.conf matches root=UUID=
    FINALLY mnt is unmounted
    AND the git server is shut down

Partial deployments
===================

Deploy part of a cluster
------------------------

Starting from the well-defined position of having a cluster morphology
with only one definition.

    SCENARIO partially deploying a cluster morphology
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    AND the user builds the system systems/test-system.morph in branch master
    GIVEN a cluster called test-cluster.morph in system branch master
    AND a system in cluster test-cluster.morph in branch master called test-system
    AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph
    AND system test-system in cluster test-cluster.morph in branch master has deployment type: tar
    AND system test-system in cluster test-cluster.morph in branch master has deployment location: test-system.tar

It is useful to group related deployments together, so we support adding
another deployment to the same cluster morphology.

    GIVEN a system in cluster test-cluster.morph in branch master called second-system
    AND system second-system in cluster test-cluster.morph in branch master builds systems/test-system.morph
    AND system second-system in cluster test-cluster.morph in branch master has deployment type: tar
    AND system second-system in cluster test-cluster.morph in branch master has deployment location: second-system.tar

When we don't tell `morph deploy` which system we want to deploy, all
of the systems in the cluster are deployed. Here a successful deployment
will have morph exit sucessfully and in the case of tarball deployments,
the tarballs for both the systems will be created.

    WHEN the user attempts to deploy the cluster test-cluster.morph in branch master
    THEN morph succeeded
    AND file workspace/master/test/morphs/test-system.tar exists
    AND file workspace/master/test/morphs/second-system.tar exists

However, we don't need to deploy every system defined in a cluster at
once. This is useful for cases such as having a cluster morphology for
deploying a whole distbuild network, and re-deploying only nodes that
have failed.

    GIVEN the files workspace/master/test/morphs/test-system.tar and workspace/master/test/morphs/second-system.tar are removed
    WHEN the user attempts to deploy test-system from cluster test-cluster.morph in branch master

A successful deployment will have morph exit successfully, and in the
case of tarball deployments, only the tarball for the system we asked
for will be created.

    THEN morph succeeded
    AND file workspace/master/test/morphs/test-system.tar exists
    AND file workspace/master/test/morphs/second-system.tar does not exist

Cluster morphs can contain "nested systems", i.e. systems which have
subsystems to deploy as part of them.

We need to add a subsystem to the cluster to test this.

    GIVEN a subsystem in cluster test-cluster.morph in branch master called test-system.sysroot
    AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master builds systems/test-system.morph
    AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master has deployment type: sysroot

We specify the location as a file path, this is relative to the parent
system's extracted rootfs, before it is configured.

    AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master has deployment location: var/lib/sysroots/test-system

The system which contains a nested system is deployed the same as
before, we don't need to mention the nested deployment.

    AND the file workspace/master/test/morphs/test-system.tar is removed
    WHEN the user attempts to deploy test-system from cluster test-cluster.morph in branch master
    THEN morph succeeded
    AND file workspace/master/test/morphs/test-system.tar exists
    AND tarball workspace/master/test/morphs/test-system.tar contains var/lib/sysroots/test-system/baserock

Morph will abort deployment if the system to deploy that is specified
on the command line is not defined in the morphology.

    WHEN the user attempts to deploy not-a-system from cluster test-cluster.morph in branch master
    THEN morph failed

It is not valid to deploy a nested system on its own. If it becomes
desirable to deploy a system that is identical to a system that already
exists but is nested in another, it should be redefined as a top-level
deployment.

    WHEN the user attempts to deploy test-system.sysroot from cluster test-cluster.morph in branch master
    THEN morph failed
    FINALLY the git server is shut down

Deploying branch-from-image produced systems
============================================

We have this nifty subcommand called branch-from-image, which can be
used to build the same thing as an existing image.

There's no special requirements for making the image reproducible.

    SCENARIO reproducing systems
    GIVEN a workspace
    AND a git server
    WHEN the user checks out the system branch called master
    AND the user builds the system systems/test-system.morph in branch master
    GIVEN a cluster called test-cluster.morph in system branch master
    AND a system in cluster test-cluster.morph in branch master called test-system
    AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph
    AND system test-system in cluster test-cluster.morph in branch master has deployment type: sysroot
    AND system test-system in cluster test-cluster.morph in branch master has deployment location: test-system
    WHEN the user attempts to deploy the cluster test-cluster.morph in branch master
    THEN morph succeeded
    AND file workspace/master/test/morphs/test-system exists

To reproduce an existing image, do a checkout with the extracted root
filesystem's /baserock directory as the `--metadata-dir` argument.

    WHEN the user attempts to check out the system branch from workspace/master/test/morphs/test-system called mybranch
    THEN morph succeeded
    AND the system branch mybranch is checked out

After it is checked-out, the system can be rebuilt.

    WHEN the user attempts to build the system systems/test-system.morph in branch mybranch
    THEN morph succeeded

Once it is rebuilt, it can be deployed.

    GIVEN a cluster called test-cluster.morph in system branch mybranch
    AND a system in cluster test-cluster.morph in branch mybranch called test-system
    AND system test-system in cluster test-cluster.morph in branch mybranch builds systems/test-system.morph
    AND system test-system in cluster test-cluster.morph in branch mybranch has deployment type: tar
    AND system test-system in cluster test-cluster.morph in branch mybranch has deployment location: test-system.tar
    WHEN the user attempts to deploy the cluster test-cluster.morph in branch mybranch
    THEN morph succeeded
    AND file workspace/mybranch/test/morphs/test-system.tar exists
    FINALLY the git server is shut down