| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
| |
This was the wrong response to the problem of accidentally checking-out
morph when trying to check out morphs. Now that it's called definitions,
this is irrelevent, and aborting a checkout when this check fails is the
wrong thing to do.
|
|
|
|
|
|
| |
This was originally used by `morph edit`, but since we removed the need
to run `morph edit system stratum` and could shorten it to `morph edit
chunk`, this function is no longer used.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The error message:
ERROR: Failed to determine the build system of repo file://foo/bar/baz at
ref 59713cf997385a094091443fdcce9d5c17313f39: was looking for
distbuild-system-x86-64.morph
is confusing since our system morph has nothing to do with build systems,
the fact that build system autodetection is executed when looking for
distbuild-system-x86-64.morph is an implementation detail that shouldn't be
exposed to the user.
This patch replaces this error message with:
ERROR: Couldn't find morphology: distbuild-system-x86-64.morph
This is still not ideal, since there are cases where we may not be able
find the morph because build system autodection has failed, but out of the
two user typos/mistakes are probably more likely.
Differentiating between user error and build system detection failure would
require more substantial changes.
This patch also splits the get_morphology function into two functions
_get_morphology_text, that actually fetches the morph text from the cache
or otherwise infers it from the build system,
and get_morphology which constructs a Morphology object from the text.
|
| |
|
| |
|
|
|
|
|
| |
This error message didn't report the cluster morph at fault.
We also fix a minor formatting issue.
|
|\
| |
| |
| |
| | |
Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When the given ref points to a specific commit, and it's already available in the
locally cached copy of repo, there's no need to update the repo.
If the ref points to a branch or tag, and the user didn't pass --no-git-update, the
locally cached copy of the repo will still be updated.
This should speed up many Morph commands.
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| | |
Instead of taking the name of a cluster morphology and zero or more
parameters for overriding the cluster morphology, morph deploy should
take the name of a cluster morphology and the names of zero or more
system deployments that are defined in the cluster morphology. If no
deployment names are given then all deployments are deployed.
|
| | |
|
| |
| |
| |
| |
| | |
The names of deployments in cluster morphologies will need to be unique
in order for the deployment of selected systems to work.
|
|/
|
|
|
|
| |
We want to supply the end of options option before
the SYSTEM_INTEGRATION_PATH so that the system integration path
doesn't get interpreted as an option if it happens to begin with a -
|
|\
| |
| |
| | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| |
| |
| |
| |
| |
| | |
There is code that assumes these exist in at least one place:
StagingArea.abort(). That code should be fixed, but we should also stop
deleting them every time we run 'morph gc'.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If btrfs is not present in the kernel we end up with strange output like
this:
Error creating disk image2014-06-10 16:00:40
[devel-system-x86_64-generic][my-raw-disk-image][rawdisk.write]Failure
to create disk image at /src/tmp/testdev.img
ERROR: Command failed: mount -o loop /src/tmp/testdev.img
/src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE
mount: mounting /dev/loop0 on
/src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE failed: Device or
resource busy
To avoid this confusing error, Morph should explicitly check first.
|
| |
| |
| |
| |
| |
| | |
After a failed attempt to connect to the controller node
the initiator will wait 30 seconds before attempting a reconnect,
if this reconnect fails the initiator gives up.
|
| | |
|
|\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Artifacts can have multiple parts; while this may not be an ideal
design, changing the format of artifacts has implications for backwards
compatibility.
We should transfer all parts at once and delete them all if we encounter
any errors, to reduce the change of getting the local artifact cache
into an inconsistent state.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Previously Morph would check if an artifact is present in the remote
artifact cache, then fetch the necessary files. If an error occured
during fetching, it would raise an error and abort.
Instead, we should just try and fetch the files and if anything fails,
move on to building locally. This avoids the situation where an error in
the remote cache makes local building impossible, which we experienced
recently.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Before:
ERROR: Failed to get metadata meta for the artifact
file:///src/ws-baserock-hawk/baserock/ps/build-system/baserock/baserock/definitions|refs/heads/baserock/builds/778b1a370a1f43c497c1354a2a949de1/56c9ec89d09240fd80faa7d2226b7eda|core|core-devel
from the artifact cache http://git.baserock.org:8080/
After:
ERROR: Failed to get metadata meta for the artifact
f896a081beacd4a99ded38d28b44fbf02970038fb53349265f85f8f3298ead9d.stratum.core-devel
from the artifact cache http://git.baserock.org:8080/
When debugging artifact cache issues, the information that's most
useful is the filename of the artfact.
|
| | |
| | |
| | |
| | | |
This saves from having to catch three separate exceptions.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
The deploy plugin is relying on existing code which respects the
'no-git-update' setting and updates gits if it is not set.
Since deploy can only work for built systems anyway, we can force this
True for deployments, to avoid wasting the user/caller's time.
|
|\ \ \
| |_|/
|/| |
| | | |
Reviewed-by: Sam Thursfield and Richard Ipsum
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
- All tests now pass
- The odd case of chunks with the same name but different repo URLs now
correctly informs the user of the multiple checkouts that were done.
- Tidyups
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Since we removed ref: fields from the system morph files, some of the logic
in morph edit is no longer needed. In particular, running
morph edit <system> <stratum>
is a no-op. This is because the version of <system> and <stratum> are now
implicit from the context of what we are doing. In other words we're always
working with the current version of <system> and <stratum> from the system
branch we are in.
Because of the complexity of morph's logic, we didn't notice this when
dropping the ref: fields, and we missed the opportunity to simplify our
logic for 'morph edit'
This patch aims to provide the simplest possible implementation of
morph edit:
morph edit <chunk>
It checks all strata for instances of <chunk>, and does what morph edit should
do for the instances it finds.
A later patch can add warnings to help users deal with situations where <chunk>
is not found, or is found more than once.
Also since this changes the syntax of morph, it breaks many of our tests. Later
patches will address that.
|
|/ /
| |
| |
| |
| |
| |
| |
| | |
Generally deployment temp dirs are removed by morph during deployment,
but in some cases deployment dirs may not be cleared up,
for example if morph gets a SIGKILL or something unexpected
happens that causes morph to terminate without having a chance
to cleanup.
|
|\ \
| | |
| | |
| | |
| | | |
Reviewed-by: Pedro Alvarez
Reviewed-by: Sam Thursfield
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This makes systems use the UUID of the disk in the fstab when there is
no pre-existing fstab entry for /.
This happens whether the system has an initramfs or not, since it should
be harmless, and by this point we're in userland, so can know what the
UUIDs are.
Passing the UUID to `complete_fstab_for_btrfs_layout` is optional, and
defaults to the old behaviour of using /dev/sda, since it is called
directly by some write extensions for doing upgrades, and upgrading
systems that use an initramfs will be part of a later patch.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
It's only ever called from the core of the write extensions library, so
we can change it to be mandatory in all cases.
install_extlinux is left with the uuid being optional and defaulting to
/dev/sda, since it is called directly from the rawdisk write extension
currently, and upgrades of systems that have an initramfs will be part
of a later patch series.
|
|\ \ \
| |/ /
|/| |
| | |
| | | |
Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
|
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The copy-artifacts and list-artifacts commands are mainly useful for
making releases. As part of the release process we copy artifacts for
the entire build graph of the release to the artifact cache on
trove.baserock.org, to provide Baserock users with ready-built
artifacts.
This part of the release process is now automated, and the automation
require the list-artifacts command to function as a 'plumbing' command.
The copy-artifacts command is no longer required. It can be replaced
with:
morph list-artifacts --quiet REPO REF MORPH | rsync --files-from=- $TARGET
The previous version of this plugin looked in the system artifact's
metadata for the list of artifacts. This is flawed as the final system
does not necessarily contain every build dependency. The new version of
the plugin calculates the build graph from source, using the same
process as the 'buildcommand' module. It also required looking in
Morph's artifact cache for the system artifact file to analyse.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If INITRAMFS_PATH is specified and the file exists, then the produced
kernel command line will use root=UUID=$uuid_of_created_disk rather than
root=/dev/sda, which may be incorrect.
Help files have been updated to mention the new option.
This leads to an unfortunate duplication of the path to the initramfs,
in both the location field of the nested deployment and the
INITRAMFS_PATH of the disk image creation.
However, an initramfs could be produced by a chunk and put in the same
place, so it doesn't make sense to couple the rawdisk and initramfs
write extensions to remove this duplication.
Similarly, there may be multiple valid initramfs in the rootfs e.g.
extlinux loads a hypervisor, which is Linux + initramfs, and the
initramfs then boots a guest Linux system, which uses a different
initramfs.
This makes it important to explicitly let the rootfs write extensions
know which to use, or not as the case may be.
util-linux's blkid is required, since the busybox version ignores the
options to filter its output, and parsing the output is undesirable.
Because syslinux's btrfs subvolume support is limited to being able to
use a non-0 default subvolume, the initramfs has to be copied out of
the run-time rootfs subvolume and into the boot subvolume.
This pushed the required disk space of a minimal system over the 512M
threshold because we do not have the userland tooling support to be able
to do a btrfs file contents clone.
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This creates a gzipped cpio archive that may be used as an initramfs.
It is hard-coded to use gzip to compress the initramfs, since it's the
most common way to do it.
This is unfortunate, since the busybox gzip utility only allows maximum
compression, which is rather slow and doesn't give progress reporting,
so you can easily think it's gotten stuck.
It's possible to use other compression formats, but they need the kernel
to be built with them supported, and in the case of lz4, unusual userland
tools to create it, since the version of lz4 supported in the kernel is
not what the standard lz4 tools produce.
|
|\
| |
| |
| |
| | |
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
shutil.move() does not preserve permissions, file modes, ownerships etc,
resulting in much confusion when prepopulating a non-root user during
deployment. This change to `mv` fixes that.
|
|/
|
|
|
|
|
| |
Add support to the VirtualBox write extension to notice if we are
doing a Vagrant Basebox installation and not do the clever network
setup we normally do to allow machines to talk to one another since
this confuses Vagrant quite a bit if it is left in.
|
|
|
|
|
|
|
|
|
| |
This is useful to build releases using distbuild. It avoids having the
SHA1 fields in the artifact metadata files pointing to commits that
exist only on temporary build branches. It also avoids file:// URLs in
the repo fields. Note that the repo URL still points to the Trove used
by the distbuild network, rather than being an upstream URL pointing to
git.baserock.org.
|
|\
| |
| |
| |
| |
| |
| | |
Reviewed by:
Lars Wirzenius
Sam Thursfield
Daniel Silverstone
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We could just set stdout to subprocess.PIPE
then read from the pipe, but then we won't get the output
till the command's finished and some commands take a long time.
Using the logfile kwarg a file will be created by tee
and the output will be written to it in 'real time'.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We use tee to write the output to a file as well as to stdout.
Using Popen it should be straight forward to send the output to a pipe and
then read that pipe and write to wherever.
At the moment morph uses cliapp's runcmd rather than Popen. cliapp's runcmd
is a blocking call, so we're not able to read from the pipe until the
command has completed, which prevents real time logging to a number of files.
One solution to this problem might be to spawn a thread which opens a pipe
to the command being executed, the thread then reads from the pipe and writes
to our collection of logfiles.
|
| | |
|
| | |
|