| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
| |
metadata_version is for when the format of the /baserock metadata files
changes.
This means it would make sense for it to either live with the code for
generating the metadata, or the cache key code so it lives with the rest
of the compatibility values.
Since the code for generating the metadata isn't in a nice module
anywhere, I've put it in the cachekeycomputer module.
|
|
|
|
|
|
|
|
|
|
| |
It was an odd thing to have, when Artifact objects are part of your
input.
Its purpose appears to have been to allow the build step to produce an
artifact called $morphology_name-rootfs, but since the split rules
decide that the artifact is called that anyway, the new_artifact step is
redundant.
|
| |
|
| |
|
|
|
|
|
| |
This helps debugging issues with rule matching, since SplitRules can be
print-statemented
|
|
|
|
|
|
|
|
| |
It rather peculiarly defines artifacts that have different cache keys,
but the same source.
This flies in the face of how real artifacts get cache keys, and our
ability to move the cache key to being per-source.
|
|
|
|
|
| |
We don't care if add_dependency causes there to be multiple dependents,
just that our artifacts are properly included.
|
|
|
|
|
|
|
| |
This pre-dates deployment, and if we need the kernel, we can always copy
it out of the rootfs.
It also uses new_artifact, which is a method I want to remove.
|
|
|
|
|
|
|
|
| |
This workaround fix http://bugs.python.org/issue12841
The code added in this patch is from tarfile.py from python 2.7.3.
Also only use the workaround for tarfile.makefile when the ptyhon
is smaller than 2.7.4
|
|
|
|
|
|
| |
It broke when we added /baserock/deployment.meta.
We didn't notice this because our test suite was looking at the artifact
produced by morph build, and listed on the terminal.
|
|
|
|
|
|
| |
The openstack.write extension was calling a nonexistent method
'check_location'. This metod was moved to openstack.check
in the commit ba7d1d1ed3bad002ce36e5d4adf4e3794625091a.
|
| |
|
|\
| |
| |
| |
| | |
Reviewed-by: Sam Thursfield
Reviewed-by: Lars Wirzenius
|
| |
| |
| |
| |
| |
| |
| |
| | |
If the credentials are wrong, then morph will fail before
attempting the OpenStack deployment.
To achieve that openstack.check will attempt to run
`glance image-list`.
|
| |
| |
| |
| | |
Found by Richard Maw.
|
| | |
|
| | |
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The KVM and VirtualBox deployments use sparse files for raw disk
images. This means they can store a large disk (say, tens or hundreds
of gigabytes) without using more disk space than is required for the
actual content (e.g., a gigabyte or so for the files in the root
filesystem). The kernel and filesystem make the unwritten parts of the
disk image look as if they are filled with zero bytes. This is good.
However, during deployment those sparse files get transferred as if
there really are a lot of zeroes. Those zeroes take a lot of time to
transfer. rsync, for example, does not handle large holes efficiently.
This change introduces a couple of helper tools (morphlib/xfer-hole
and morphlib/recv-hole), which transfer the holes more efficiently.
The xfer-hole program reads a file and outputs records like these:
DATA
123
binary data (exaclyt 123 bytes and no newline at the end)
HOLE
3245
xfer-hole can do this efficiently, without having to read through all
the zeroes in the holes, using the SEEK_DATA and SEEK_HOLE arguments
to lseek.
Using this, the holes take only take a few bytes each, making it
possible to transfer a disk image faster. In my benchmarks,
transferring a 100G byte disk image took about 100 seconds for KVM,
and 220 seconds for VirtualBox (which needs to more work at the
receiver to convert the raw disk to a VDI). Both benchmarks were from
a VM on my laptop to the laptop itself.
The interesting bit here is that the receiver (recv-hole) is simple
enough that it can be implemented in a bit of shell script, and the
text of the shell script can be run on the remote end by giving it to
ssh as a command line argument. This means there is no need to install
any special tools on the receiver, which makes using this improvement
much simpler.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than using `git ls-remote` every time to see if there are changes
at the remote end, use a local cache.
Git already solves this problem with its refs/remotes/$foo branch
namespace, so we can just use that instead.
In addition, the detection of which upstream branch to use has been
improved; it now uses git's notion of which the upstream branch of your
current branch is, which saves effort in the implementation, and allows
the name of the local branch to differ from that of the remote branch.
This now won't notice if the branch you currently have checked out had
commits pushed from another source, but for some use-cases this is
preferable, as the result equivalent to if you had built before the
other push.
It may make sense to further extend this logic to check that the local
branch is not ahead of the remote branch, instead of requiring them to
be equal.
|
|\
| |
| |
| |
| | |
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There's no need to log every time we look something up in a dict. This just
makes log files huge.
The CacheKeyComputer.compute_key() function still logs the first time it
calculates a cache key for an Artifact instance. This message now includes
the hex string that is used to identify the artifact.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously logging was disabled for Python deploy extensions.
We get a lot of useful information for free in the log file by
doing this: the environment will be written when the subprocess starts,
and if it crashes the full backtrace will be written there too. Subcommand
execution with cliapp.runcmd() will also be logged.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is achieved by passing it the write end of a pipe, so that the
extension has somewhere to write debug messages without clobbering either
stdout or stderr.
Previously deployment extensions could only display status messages on
stdout, which is good for telling the user what is happening but is not
useful when trying to do post-mortem debugging, when more information is
usually required.
This uses the asyncore asynchronous event framework, which is rather
specific to sockets, but can be made to work with file descriptors, and
has the advantage that it's part of the python standard library.
It relies on polling file descriptors, but there's a trick with pipes to
use them as a notification that a process has exited:
1. Ensure the write-end of the pipe is only open in the process you
want to know when it exits
2. Make sure the pipe's FDs are set in non-blocking read/write mode
3. Call select or poll on the read-end of the file descriptor
4. If select/poll says you can read from that FD, but you get an EOF,
then the process has closed it, and if you know it doesn't do that
in normal operation, then the process has terminated.
It took a few weird hacks to get the async_chat module to unregister
its event handlers on EOF, but the result is an event loop that is
asleep until the kernel tells it that it has to do something.
|
| |
| |
| |
| |
| |
| | |
The arguments to `morph deploy` can get quite long, any way we can make it
shorter and clearer is useful. We can also avoid having the strange
--no-upgrade flag in future.
|
|/ |
|
|\
| |
| |
| |
| | |
Reviewed-by: Francisco Redondo Marchena
Reviewed-by: Sam Thursfield
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The MorphologyFactory class will use a RemoteRepoCache to see if a
morphology file exists, and if it doesn't, uses a file listing to see
if it can detect what build-system is uses, hence what the default
morphology should be.
However, it was overly generic in what error cases it would accept as
the morphology not being found, so if the RemoteRepoCache was suddenly
un-resolvable for a brief period, then it would assume the morphology
didn't exist, and use the default one.
This happened to a user, and the result was a full rebuild.
So we now fix this by only raising the exception that means the file
didn't exist, if we got a HTTP 404.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The clone_into function is non-functional when you pass it a sha1 ref.
If you have a file:// URI then this doesn't get used, which is how it
slipped past the tests.
Reviewed-by: Lars Wirzenius +2
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We assumed that the sha1 of the tree of the commit of the ref we care
about was sufficient to cache, but `git replace` means that you need to
know the state of other branches too, since anything in refs/replace can
completely change what the tree you check-out is.
This behaviour can be disabled globally by setting
GIT_NO_REPLACE_OBJECTS, so we're going to do that.
If we need to integrate a project that uses git-replace to change the
contents of their git trees then we could support that by:
if any(refs/replace/*):
potentially_replacable_objects = [
`git rev-parse HEAD`,
`git rev-parse HEAD^{commit}`,
`git rev-parse HEAD^{tree}`]
potentially_replacable_objects.extend(
`git ls-tree -r HEAD | awk '{print $3}'`)
# NOTE: doesn't handle submodules, and you'd need to expand this
# set whenever you process a replacement
for object in refs/replace/*:
if basename(object) not in potentially_replacable_objects: continue
cache_key['replacements'][basename(object)] = `git rev-parse $object`
If we were to support this would need to traverse the tree anyway, doing
replacements, so we may as well use libgit to do the checkout anyway,
and list which replacements were used.
However, since the expected use-case of `git replace` is as a better way
to do history grafting, we're unlikely to need it, as it would only have
any effect if it replaced the commit we were using with a different one.
Rubber-stamped-by: Daniel Silverstone
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This avoids confusion when the user expected to be doing an initial
deployment and wasn't aware that a file with the same name as the
target already existed. Previously rawdisk.write would try to mount
the file and upgrade it.
Now we require the user to pass '--upgrade' when they intend to
upgrade, as with other deployment extensions.
|
|/
|
|
|
|
|
|
| |
This reduces the vast number of 'git config -z core.bare' which
we used to get a lot of, to one or two per run. It doesn't
save much time, but it does make logs less full of confusion.
Reviewed-By: Richard maw <richard.maw@codethink.co.uk> +2
|
| |
|
| |
|
| |
|
| |
|
| |
|
|\
| |
| |
| |
| | |
Reviewed by: Richard Maw <richard.maw@codethink.co.uk>
Merged by: James Thomas <james.thomas@codethink.co.uk>
|
| |\ |
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Remove the BOOTLOADER environment variable and instead favour
BOOTLOADER_CONFIG_FORMAT to set the desired bootloader format, and
BOOTLOADER_INSTALL to set the type of bootloader to install.
For example, since u-boot can boot using extlinux.conf files, it's
conceivable that someone might want to do CONFIG_FORMAT=extlinux.conf,
INSTALL=u-boot.
However, for most platforms you would want to set INSTALL to "none"
|
| |/ |
|
| |
| |
| |
| |
| |
| | |
Instead of leaving morph3 with a potentially confusing name, rename it
to `morphology` since it is now the only implementation of the
Morphology class.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This reverts commit 4124c50b8dc3dfb0ffb933153d0fe6385edf389c.
This commit is no longer needed now that morph2 is gone.
Conflicts:
morphlib/cachekeycomputer.py
|
| |
| |
| |
| | |
This commit removes the now unneeded morph2 and its associated tests.
|
| |
| |
| |
| |
| | |
This commit removes all use of morph2 from the unittests, replacing it
with morphloader/morph3. It also converts the test morphologies to YAML.
|
| |
| |
| |
| |
| | |
Rather than generating the text of a morphology which is later loaded,
generate a Morphology object and return that.
|
| |
| |
| |
| |
| |
| |
| |
| | |
This commit reworks morphologyfactory to stop using morph2. It removes
the validation that is already handled by morphloader, and uses
morphloader to load the morphology rather than using morph2. The unit
tests are also changed to turn the example morphologies into YAML, and
also to check for the correct exceptions.
|
| |
| |
| |
| |
| |
| | |
Rather than having a `get_commands` method to obtain missing commands
from the build system when they are needed, get the commands when
loading a morphology.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit stops morphloader setting the `morph` field in chunk specs,
and also makes it set defaults for the prefix and build-mode fields. Not
setting the `morph` field is necessary as its presence in chunk specs is
used by `traverse_morphs` to mean that the morphology file is in the
definitions repository, not the chunk source repository. If we set a
default value here, we end up looking for files which do not exist.
|
| |
| |
| |
| |
| |
| |
| |
| | |
Currently, morphloader sets the morph field in chunk specs by default.
A later commit will remove this behaviour, so morphset needs to stop
expecting the `morph` field to exist in morphology specs. However,
stratum specs don't have a `name` field so we can't expect to be able to
find that either.
|