| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Previously this resulted in a confusing traceback.
This is a quick fix. I'd rather insert the error in the buildcommand
module instead, but that code assumes the source being built is a
system in several places before it actually checks the kind. Those
would all need to be changed, or the code would need to reworked to
call _validate_root_kind() much earlier.
The Application.traverse_morphs() method is rather ugly anyway, so I'm
happy to add further ugliness to it for the time being.
|
| |
|
|
|
|
|
|
|
| |
The Source.__init__() function assumes that the artifact.split_rules
attribute is not None. Rather than complicating that code with error
checks, let's make it always be correct.
Avoids traceback from Source.__init__() when passing a cluster morph to
`morph build`.
|
| |\
| |
| |
| | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
gitdir._list_work_tree_files() needs to use os.relpath() instead of
direct string manipulation to avoid chopping off the first line of
every filename in cases where the base gitdir path string includes the
trailing /.
Unit test updated to catch this.
|
| | | |
|
| | | |
|
| |/
|
|
|
|
| |
We're clearly comparing two values for equality rather than
asserting that the type of an object is not False, since that
would make no sense at all.
|
| |\
| |
| |
| |
| | |
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
|
| | |
| |
| |
| |
| | |
'--help' when used with a subcommand will show the subcommand help.
Do not reflow the help text by using a custom formatter.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit introduces a new requirement: USERS MUST NOT HAVE SENSITIVE
DATA IN THEIR ENVIRONMENT. Otherwise it will be leaked into the system.
Note that configuration fields with 'PASSWORD' in their name are
stripped before writing the /baserock/deployment.meta file, so the
OpenStack OS_PASSWORD field is not leaked.
We want this so that we can run hooks at upgrade-time in the future.
These hooks might need to know how the system was configured and what
releaseuu it was. I'm not quite sure how we will define 'release' yet,
but by using `git tag` and `git describe` we are able to textually label
a time period in the history of the system's source code. We already
have the specific SHA1 of definitions.git stored in the system metadata,
so this should give us enough to be able to implement specific hooks
that work around any awkward upgrade complications we encounter in the
future.
|
| | |
| |
| |
| |
| |
| |
| | |
If the disk image was not yet created then the os.remove() call fails
and the original exception gets lost, causing confusion and sadness.
Also print status earlier on failure
|
| | |
| |
| |
| |
| | |
Most write extensions don't handle both initial deployments and upgrades
of a system.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
A write extension will have various kinds of sanity checks to do before
actually performing the write. The current architecture of 'morph
deploy' means that several minutes pass between the user starting the
command and the write extension actually executing. It would be
rage-inducing watching `morph deploy` spend 3 minutes unpacking a
system only to then abort due to a silly error such as forgetting the
--upgrade switch. Therefore it's better for now to split the sanity
checks out into separate extensions that can be run as soon as possible
and abort if the write extension is not going to be able to operate.
For now this will just be used to validate usage of the --upgrade flag
but in future checking connectivity to remote servers and the like
should be done here too.
|
| | |
| |
| |
| |
| | |
Now you can deploy an upgrade, set it to be the default version and reboot
into it all with one call to `morph deploy`.
|
| | |
| |
| |
| |
| | |
Also, be more flexible when parsing environment booleans -- convert to
lower case and match 0/1 and true/false as well as yes/no.
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
We now have a OS version manager tool in Baserock (in tbdiff.git). The
code to deploy a new base OS version should live there, to minimise
duplication between write extensions.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This messes up the baserock-system-config-sync tool. Systemd does not
require /etc/fstab to exist in any case.
I have bumped the 'system-compatibility-version' field in this commit
to trigger rebuilding all system artifacts.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The shared state directories defined in writeexts.py (/var, /home etc.)
are now separate Btrfs subvolumes that are mounted in place using fstab.
There are some warnings on mounting /var and /srv about the mountpoint
not being empty. Not yet investigated.
If a configure extension has already added / to the fstab, use the
device it chose rather than assuming /dev/sda. This is required for the
vdaboot.configure extension that we use for OpenStack deployments.
Similarly, if a configure extension has added an entry for a state
directory in /etc/fstab already, we don't replace it with a /state/xxx
directory. That's only done as a default behaviour.
|
| | | |
|
| |/
|
|
|
| |
We will need this file to enable a bootloader menu to choose between
OS after an upgrade.
|
| |
|
|
|
|
| |
After this, "./check --full" works.
Reviewed-by: Daniel Silverstone (on IRC)
|
| |
|
|
| |
This reverts commit a72c8dca6965d1ac239e4f0102f08fbf7fe59ac7.
|
| |
|
|
| |
This reverts commit ab0a83a09a93ca33aa402d9c4d3b916a48a1a882.
|
| |
|
|
| |
This reverts commit 329b81419be20e7b1f2651a47030186216044eec.
|
| |
|
|
|
|
|
| |
Add a command 'help-extensions' to list all extensions.
Add the ability to find help on an extension by calling
'morph help [extension name]'. This will then call
the extension with the '--help' option to obtain help text.
|
| | |
|
| |
|
|
|
| |
'--help' when used with a subcommand will show the subcommand help.
Do not reflow the help text by using a custom formatter.
|
| |
|
|
|
| |
This rips out any remaining order-preserving code and instead uses the
yaml dumper from morphloader.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This now means that the system morphology is not altered when chunks are
altered, so some tests had to change.
Since this uses the python warnings API, these warnings can be ignored
by running
python -W ignore:"stratum morphology" \
-W ignore:"system morphology" \
"$(which morph)" ...`
or turned into errors with
python -W error:"stratum morphology" \
-W error:"system morphology" \
"$(which morph)" ...`
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
This includes various changes allowing the repo and ref fields to be
missing; but also a change to the the component_key function, so that
instead of generating a string, it returns a tuple, since it's only
required to be a consistently hashable index.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
VirtualBox changed a command line option in 4.3 incompatibly, so we now
have to check the version number and change an option from --sataportcount
to --portcount if the version of VirtualBox running on the target is at
least 4.3
This turns the version into a tuple and compares it against another,
since it's more reliable than comparing strings, which will count '1.10'
as earlier than '1.2', and more convenient than comparing the digits
individually.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If two systems with the same name (e.g. different repo/ref) depend on the
same strata, then it will collide with systems which depend on different
artifacts from that stratum, but the same number of artifacts.
For example, if you checkout an existing branch and change the artifacts
used by one of its strata, then your local changes won't be built.
This is because the 'kids' field lists artifacts it depends on by their
cache-key, which is now no longer sufficient to uniquely identify
artifacts. The same number of artifacts issue is from it listing cache
keys multiple times.
The fix for this is to include the artifact name, so the 'kids' field is
now a list of dicts, with artifact name and cache key.
This is a dict rather than a tuple so that the generated /baserock
metadata is more readable.
|
| |
|
|
|
|
| |
For chunks the products field doesn't need to be hashed, since the
split-rules field is used instead, and includes the default rule set,
and for strata and systems it is handled by hashing the dependencies.
|
| | |
|
| | |
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Rationale
=========
This patch series implements the concept of stratum splitting.
For a long time we've had code to split a chunk into multiple artifacts,
however there's not been a way to split strata up, or systems select a
subset of the produced stratum artifacts to be included in the system.
This patch series implements the ability to split strata and have systems
include them in a way which still has the same behaviour if no rules
are specified, but with default rules that split chunk artifacts up into
various components, strata into runtime and development versions and has
systems include everything by default, but can be told to include less.
The default rules have chunk foo split up into -bins, -libs, -devel,
-doc, -locale and -misc.
These rules can be overridden in the chunk morphology by adding the new
'products' field, which lists match rules like the following:
products:
- artifact: libudev
include:
- (usr/)?lib(32|64)?/lubg?udev\..*
- artifact: udev
include:
- (usr/)?s?bin/udev*
- (usr/)?lib(32|64|exec)?/systemd/systemd-udevd
Strata are by default split into -runtime and -devel. -devel by default
contains chunks ending with -devel and -doc, -runtime contains everything
else.
Extra match rules can be added to a stratum similarly to chunks, but
instead of matching file names, they match artifact names.
products:
- artifact: core-python
include:
- "cpython-.*" # lazy shortcut to put all of cpython in this stratum
- "python-.*" # lazy shortcut to include all python chunks in
Additionally, in chunk specs, chunk artifacts may be assigned to stratum
artifacts, this takes precedence over products match rules in the stratum
and the default match rules. Assigning the chunk to `null` will discard
the chunk.
chunks:
...
- name: systemd
...
artifacts:
libudev: foundation-runtime
udev: foundation-runtime
systemd-doc: null
By default a system includes every produced artifact of every stratum
listed. Instead a subset can be specified in the stratum spec as follows:
name: tiny-system
strata:
- name: build-essential
...
artifacts:
- build-essential-runtime
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
For some reason we used to create a new Artifact object with the same
name as the Stratum morphology for our cache key.
This is non-sensical, since we already have an Artifact object and it
breaks splitting strata.
NOTE: cmdtest tests do not pass, since they list files and artifacts
produced, which has changed since the new default splitting rules were
added. The next patch fixes this, but was kept as a separate commit for
readability.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Filenames are now matched before chunks are constructed, so
bins.create_chunk now takes a list of relative file names.
bins.chunk_contents is gone, since this is now handled by passing
source.split_rules.partition the file names.
We now don't consider it to be a problem for directories to remain in
the DESTDIR after artifacts have been removed, since we need to handle
file matches implying their parent directories, and explicit matches
against directories.
NOTE: The bins_tests were broken in this patch, and are fixed in the
next. This was done to try and aid readability of the patch series.
Full functionality is still broken until stratum splitting is fixed.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
One important change is that the builds_artifacts field of Morphologies
is not used any more, since the split rules provide this information.
Another important change is that the ArtifactResolver now only returns
aritfacts that are required to build the root artifact, rather than
every artifact in the build. Previously there was no distinction.
This is required because when artifact splitting is in effect, some
artifacts may be produced, but not depended on by anything. This confuses
the BuildCommand, which expects to be able to find a single root artifact.
NOTE: This change breaks artifact construction until "Split chunk
morphologies according to new rules" and "Split Stratum artifacts
according to new rules", since systems and strata depend on artifacts
that weren't created.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Chunk artifacts need the [(artifact_name, [regular_expression])] so that
if the default split rules change, or the blending rule changes, then
an extra version field doesn't need to be added to the cache key computer.
This is for future plans to allow the split rules to be configurable
and allow us to more easily change them.
System and Stratum artifact computations don't need this, since those
splitting rules are already expressed in the dependencies information.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
This introduces a new artifactsplitrule module, which tries to provide a
nice abstraction over matching a sequence of things to a bunch of
outputs, to be used by both chunks splitting, for separating files out
into chunk artifacts, the stratum splitting, where chunks are
aggregated into stratum artifacts, and systems selecting the right
strata to go into the artifact.
|
