| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
| |
VirtualBox changed a command line option in 4.3 incompatibly, so we now
have to check the version number and change an option from --sataportcount
to --portcount if the version of VirtualBox running on the target is at
least 4.3
This turns the version into a tuple and compares it against another,
since it's more reliable than comparing strings, which will count '1.10'
as earlier than '1.2', and more convenient than comparing the digits
individually.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If two systems with the same name (e.g. different repo/ref) depend on the
same strata, then it will collide with systems which depend on different
artifacts from that stratum, but the same number of artifacts.
For example, if you checkout an existing branch and change the artifacts
used by one of its strata, then your local changes won't be built.
This is because the 'kids' field lists artifacts it depends on by their
cache-key, which is now no longer sufficient to uniquely identify
artifacts. The same number of artifacts issue is from it listing cache
keys multiple times.
The fix for this is to include the artifact name, so the 'kids' field is
now a list of dicts, with artifact name and cache key.
This is a dict rather than a tuple so that the generated /baserock
metadata is more readable.
|
| |
|
|
|
|
| |
For chunks the products field doesn't need to be hashed, since the
split-rules field is used instead, and includes the default rule set,
and for strata and systems it is handled by hashing the dependencies.
|
| | |
|
| | |
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Rationale
=========
This patch series implements the concept of stratum splitting.
For a long time we've had code to split a chunk into multiple artifacts,
however there's not been a way to split strata up, or systems select a
subset of the produced stratum artifacts to be included in the system.
This patch series implements the ability to split strata and have systems
include them in a way which still has the same behaviour if no rules
are specified, but with default rules that split chunk artifacts up into
various components, strata into runtime and development versions and has
systems include everything by default, but can be told to include less.
The default rules have chunk foo split up into -bins, -libs, -devel,
-doc, -locale and -misc.
These rules can be overridden in the chunk morphology by adding the new
'products' field, which lists match rules like the following:
products:
- artifact: libudev
include:
- (usr/)?lib(32|64)?/lubg?udev\..*
- artifact: udev
include:
- (usr/)?s?bin/udev*
- (usr/)?lib(32|64|exec)?/systemd/systemd-udevd
Strata are by default split into -runtime and -devel. -devel by default
contains chunks ending with -devel and -doc, -runtime contains everything
else.
Extra match rules can be added to a stratum similarly to chunks, but
instead of matching file names, they match artifact names.
products:
- artifact: core-python
include:
- "cpython-.*" # lazy shortcut to put all of cpython in this stratum
- "python-.*" # lazy shortcut to include all python chunks in
Additionally, in chunk specs, chunk artifacts may be assigned to stratum
artifacts, this takes precedence over products match rules in the stratum
and the default match rules. Assigning the chunk to `null` will discard
the chunk.
chunks:
...
- name: systemd
...
artifacts:
libudev: foundation-runtime
udev: foundation-runtime
systemd-doc: null
By default a system includes every produced artifact of every stratum
listed. Instead a subset can be specified in the stratum spec as follows:
name: tiny-system
strata:
- name: build-essential
...
artifacts:
- build-essential-runtime
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
For some reason we used to create a new Artifact object with the same
name as the Stratum morphology for our cache key.
This is non-sensical, since we already have an Artifact object and it
breaks splitting strata.
NOTE: cmdtest tests do not pass, since they list files and artifacts
produced, which has changed since the new default splitting rules were
added. The next patch fixes this, but was kept as a separate commit for
readability.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Filenames are now matched before chunks are constructed, so
bins.create_chunk now takes a list of relative file names.
bins.chunk_contents is gone, since this is now handled by passing
source.split_rules.partition the file names.
We now don't consider it to be a problem for directories to remain in
the DESTDIR after artifacts have been removed, since we need to handle
file matches implying their parent directories, and explicit matches
against directories.
NOTE: The bins_tests were broken in this patch, and are fixed in the
next. This was done to try and aid readability of the patch series.
Full functionality is still broken until stratum splitting is fixed.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
One important change is that the builds_artifacts field of Morphologies
is not used any more, since the split rules provide this information.
Another important change is that the ArtifactResolver now only returns
aritfacts that are required to build the root artifact, rather than
every artifact in the build. Previously there was no distinction.
This is required because when artifact splitting is in effect, some
artifacts may be produced, but not depended on by anything. This confuses
the BuildCommand, which expects to be able to find a single root artifact.
NOTE: This change breaks artifact construction until "Split chunk
morphologies according to new rules" and "Split Stratum artifacts
according to new rules", since systems and strata depend on artifacts
that weren't created.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Chunk artifacts need the [(artifact_name, [regular_expression])] so that
if the default split rules change, or the blending rule changes, then
an extra version field doesn't need to be added to the cache key computer.
This is for future plans to allow the split rules to be configurable
and allow us to more easily change them.
System and Stratum artifact computations don't need this, since those
splitting rules are already expressed in the dependencies information.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
This introduces a new artifactsplitrule module, which tries to provide a
nice abstraction over matching a sequence of things to a bunch of
outputs, to be used by both chunks splitting, for separating files out
into chunk artifacts, the stratum splitting, where chunks are
aggregated into stratum artifacts, and systems selecting the right
strata to go into the artifact.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
I think that it's confusing for both strata and chunk morphologies to
have a 'chunks' field, with the former listing sources and the latter
listing rules for splitting this source into artifacts.
The design for splitting strata has roughly the same idea, but operating
on chunk artifact names, rather than file names, so a name that can be
used for both was chosen.
Splits and artifacts weren't satisfactory names, so they're now called
'products'.
It was decided to break backwards compatibility of chunk morphologies
being able to specify 'chunks', since the format has changed, so extra
code would be required to translate the format, and the only users of
the 'chunks' field was the test suite, since there was no way to select
from the system, which chunk artifacts were included.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This simplifies logic for the ArtifactResolver, since it doesn't need to
have its own cache of (source, artifact_name) -> artifact and the
artifacts a source produces can be iterated directly with the artifacts
attribute, rather than having to iterate over the names and look up the
artifact object.
|
| | |
| |
| |
| |
| | |
This it convenient, as it allows the new validation code to validate
the old morphology class during the transition period.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When you attempt to build a stratum or chunk, the ArtifactResolver can
return multiple root artifacts, since the root source produces multiple
artifacts.
Rather than having the BuildCommand complain that there's multiple root
artifacts, it now validates all the produced artifacts too, since that
will validate the kinds of artifacts produced, and give a more useful
error message, that you're trying to build a stratum or chunk directly.
If all the produced artifacts validate, then an exception is raised to
signal that it got multiple artifacts, when it only expected one.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Later validation work causes the morphologies to be validated, when they
weren't previously. This would cause the test suite to not pass, since
the morphologies defined in the tests are malformed.
One common problem was tests that, instead of a name field, had the name
of the morpholgy in a field called "chunk".
There were a few cases of new fields being needed, since the tests were
written before they became mandatory.
The most interesting failure was a Source being created, which instead
of being passed a morphology object, was passed a string.
|
| |/
|
|
|
|
|
|
|
|
|
| |
This option lets the install-files config extension overwrite existing files.
A file will only be overwritten if the overwrite flag is specified for that file.
Since the overwrite arg is optionally prepended to the manifest line,
this patch should not break existing manifests
With this patch default config files can be replaced with project specific
config files
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
Previous fix only quoted URLs, which fixed petrify --no-git-update, but not the
whole problem, quoting ref and filename prevents other problems that may be
caused by non URL friendly characters.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
It now does not push branches as this is not necessary to locate the
artifact.
It still makes temporary build branches, since it is assumed that if
you have changes in your workspace, it's preferable for the deploy to
fail, rather than think you've deployed something you haven't.
|
| |
|
|
| |
The old build is still around for comparison.
|
| |
|
|
|
|
| |
This uses all the new APIs, so the code is shared across morphlib and
unit tested rather than everything being in one massive plugin that is
only black-box tested.
|
| |
|
|
|
|
|
| |
This is an abstraction on top of SystemBranchDirectories, providing the
ability to add uncommitted changes to the temporary build branch, push
temporary build branches and retrieve the correct repository URI and ref
to build the system.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Now it will optionally clean up on success based on a constructor
parameter.
It can be later cleaned up explicitly by calling close().
It is called close, rather than something more obvious, like cleanup(),
since it means the manager can be re-used with contextlib.closing().
This now means that using the Managers without a context manager is less
ugly, since you can explicitly call .close() in a finally block.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This was previously a private method of the branch and merge
plugin, but it's useful to other plugins, so has been moved to the
SystemBranchDirectory class, where everything else can get to it.
It has an unpleasant amount of coupling to other classes, but in a *good*
object oriented design it would either be a tiny module on its own,
or not exist and leave all its users to re-implement the same logic
multiple ways, so we've opted for a less clean, but more useful design.
It is left un-covered by the unit tests, since it requires a great deal
of instrumentation to test, at which point it may be best to leave it
to integration tests.
|
| |
|
|
| |
It saves some boilerplate.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bootstrap chunks don't make it into the final system, so there needs to
be an extra check for empty systems after the sources have been collected.
This was complicated slightly by the fact that if you try to build a chunk
directly you will have no strata in your sources, hence no non-bootstrap
chunks, but validation for having been told to build a chunk is best
handled later.
This amends the old yarns that depended on building a bootstrap chunk
and adds a new one that explicitly builds a system with bootstrap
chunks.
|
| |
|
|
|
|
|
|
|
|
| |
It doesn't currently make sense to build a system which contains no
strata. We may later add other fields, such as initramfs to contribute
to the system's artifact, but until then it's another bug to trip over.
This uses collections.Sequence for checking the type of the systems entry
in the morphology as a style choice, though it allows more flexibility
if the types in the parsed morphology change.
|
| |
|
|
|
|
|
|
| |
This was omittted from the MorphologyLoader due to cluster morphologies
being added at about the same time.
This bug escaped detection since the MorphologyLoader was not required
to deploy. It soon will be.
|
| |
|
|
|
| |
Dedicated exceptions allow more fine-grained exception catching and can
have extra data.
|
| |\ |
|
| | | |
|
| |\ \
| |/
|/|
| | |
'remotes/origin/baserock/ps/show-stratum-versions-in-build'
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This adds a LocalRefManager, which handles ref updates to local
repositories (i.e. your workspace).
It provides proxy methods for ref updates to a set of repositories.
If an exception occurs in the body of the context manager, the updates
will be rolled back to before the context manager was entered.
The purpose for using a LocalRefManager instead of making the changes to
the repositories directly, is to provide atomic updates to a set of refs
in a set of repositories, where all refs are updated, or none are.
This also adds a RemoteRefManager, which handles pushing branches to
remote repositories.
It provides a proxy push method, which will delete pushed branches, and
re-push deleted branches after the context manager exits.
Its purpose, instead of providing atomic updates to remote repositories,
is to provide temporary branches. This is because it is used to provide
temporary build branches. The difference between atomic update and
temporary push, is that the remote branches are deleted when the context
is left, rather than kept, as LocalRefManager does.
The RemoteRefManager currently cannot provide the same atomicity
guarantees as the LocalRefManager, so if there is a push between the
branch being created and the RemoteRefManager cleaning it up, that change
is lost without RemoteRefManager even knowing it existed.
Git 1.8.5 will add functionality to make this possible.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Remotes have a push method, which takes multiple RefSpecs, runs git push
using arguments derived from the set of refspecs, then returns the
push's result.
If it fails the push, it will return the result in the exception.
|
| | | |
|
| | |
| |
| |
| | |
Operations on remotes are now accessed through this proxy object.
|
| | | |
|
