| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\
| |
| |
| |
| | |
Reviewed-by: Lars Wirzenius
Reviewed-by: Pedro Alvarez
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Parts of the morphology go into the name of the staging area, so it
helps to convert them into a str, so later attempts to join it with
another string don't result in a unicode string.
pyfilesystem insists that file paths must be unicode. It is incorrect,
but we passed something unicode compatible in in the first place, so we
can get away with converting it back to a bytestring.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
| | | |
|
| |/
|
|
|
| |
This will make it easier to determine what is wrong if the controller
daemon is run with a bad controller host address.
|
| |\
| |
| |
| |
| |
| |
| | |
'origin/baserock/richardmaw/S11284/morphologies-by-path-v4'
Reviewed-by: Sam Thursfield
Reviewed-by: Lars Wirzenius
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We want to move our morphologies in our definitions repository into
subdirectories, so they're more organised.
We'd prefer to only refer to morphologies by file path, rather than a
name that loosely corresponds to the file path, but we need to support
that for backwards compatibility until we can move all of our
morphologies into the definitions repository.
However, since we want to eventually remove this, and we want to ensure
that file paths work, we change the yarn tests to use file paths.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We used to check whether a file existed before trying to read it. We
used to be able to get away with only looking at the top-level
directory, which made using ls-files before trying to cat-file it
better.
Unfortunately, we need to look at the files in subdirectories now, so
this no longer works.
We could make it include files in subdirectories, but for repositories
with many files, you would end up reading a file listing longer than
the morphology, so even in the slow case of needing to read the entire
morphology file, it would be faster to attempt to read the file first.
So now we beg forgiveness rather than asking permission.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This was noticed because our definitions.git had a dangling symlink,
so it failed to construct the temporary build branch.
We shouldn't process symlinks as morphologies either, since either we
make symlinked morphologies a first-class behaviour, and validate that
the link points inside the repository, which is a lot of work for
something we don't and probably won't need; or we can just assume that
we deal with the morphology this is linked to correctly anyway.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We want to use file paths to locate morphologies now, so the old model
of get a list of names and hand it those back to get the contents
doesn't really make sense any more.
This abstraction initially came about as one idea I had for moving
morphologies out of the source tree was to put them in something like
git notes, where it's possible to look up information for one commit in
another ref in the repository, at which point this abstraction would
have been flexible enough to handle that as well as in the
However, moving the chunk morphologies into the definitions repository
has other benefits too, so it makes more sense to be honest about using
filenames in the API.
It remains as a single point where we can put the logic for knowing which
files in a repository look like morphologies, but if we need to remove
any further functionality, it should be replaced by a single function.
|
| | |
| |
| |
| |
| | |
There's a lot of them, it's too much of a pain to enumerate them all, so
it's convenient to provide a hierachy and catch the base exceptions.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
`morph merge` only worked for a small subset of cases, and has been left
to bit-rot, since we don't use it.
`morph tag` is just a `git tag` when we have petrified definitions
repository. We don't use it, nor do we need it, so it can go away rather
than take up valuable development time fixing it when requirements
change.
`old-foo` have all been superceded by newer versions and are no-longer
used.
|
| | |
| |
| |
| |
| |
| |
| | |
This was the wrong response to the problem of accidentally checking-out
morph when trying to check out morphs. Now that it's called definitions,
this is irrelevent, and aborting a checkout when this check fails is the
wrong thing to do.
|
| | |
| |
| |
| |
| |
| | |
This was originally used by `morph edit`, but since we removed the need
to run `morph edit system stratum` and could shorten it to `morph edit
chunk`, this function is no longer used.
|
| | |
| |
| |
| | |
This reverts commit ad65450c145e3283f95a40397280fc8b938a04c0.
|
| | |
| |
| |
| | |
This reverts commit a6488c42e79177576d8c73927dac7c1e86311122.
|
| | | |
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | | | |
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When overriding a default chunk splitting rule, the use 'break' would
mean that some of the other default chunk split rules might be ignored.
This isn't ever what you want.
I also clarified the current behaviour in a comment. I think in future
we should add a mechanism to extend the default rules, as well as the
current mechanism which allows only replacing them. But that is a
separate issue.
|
| |\ \
| |/
|/|
| |
| | |
Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I was getting the following error when running the 'do-release.py'
script:
ERROR:root:Command failed: morph --quiet --trove-host=git.baserock.org list-artifacts baserock:baserock/definitions sam/auto-release minimal-system-x86_32-generic
ERROR: Ref d67a0e110187abd560a1de63fa172894a52839d5 is an invalid reference for repo git://git.baserock.org/delta/linux
The commit that it wants did actually exist in git.baserock.org and the
logs show that Morph hadn't done a `git remote update` to get the commit
locally.
This turned out to be because it'd had already looked up a different ref
in linux.git. It hadn't needed to run 'git remote update', but it *had*
added linux.git to a "don't need to update this repo again" list. Oops!
I've moved the code in question to the cachedrepo module so that the repo
object keeps that of whether it should be updated. The bug is now fixed.
As a side effect this patch fixes the spurious 'Updating repo
file:///...' messages, which were printed even though repos with
file:/// URLs are never updated.
|
| | |
|
| |\
| |
| |
| |
| |
| |
| | |
'origin/baserock/richardipsum/improve_getmorph_errmsg'
Reviewed-by: Lars Wirzenius
Reviewed-by: Emmet Hikory
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The error message:
ERROR: Failed to determine the build system of repo file://foo/bar/baz at
ref 59713cf997385a094091443fdcce9d5c17313f39: was looking for
distbuild-system-x86-64.morph
is confusing since our system morph has nothing to do with build systems,
the fact that build system autodetection is executed when looking for
distbuild-system-x86-64.morph is an implementation detail that shouldn't be
exposed to the user.
This patch replaces this error message with:
ERROR: Couldn't find morphology: distbuild-system-x86-64.morph
This is still not ideal, since there are cases where we may not be able
find the morph because build system autodection has failed, but out of the
two user typos/mistakes are probably more likely.
Differentiating between user error and build system detection failure would
require more substantial changes.
This patch also splits the get_morphology function into two functions
_get_morphology_text, that actually fetches the morph text from the cache
or otherwise infers it from the build system,
and get_morphology which constructs a Morphology object from the text.
|
| |/ |
|
| | |
|
| |
|
|
|
| |
This error message didn't report the cluster morph at fault.
We also fix a minor formatting issue.
|
| |\
| |
| |
| |
| | |
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
I found an issue in distbuild where the controller was stuck in a busy
loop where it was continually writing to a closed socket. With 'strace'
I saw write(), SIGPIPE, write(), SIGPIPE, ad infinitum. I got this much
of a Python backtrace using GDB:
distbuild.socketsrc.SocketEventSource.write()
distbuild.sockbuf.SocketBuffer._flush()
distbuild.sm.StateMachine.handle_event()
I didn't manage to get further. However, I suspect one of the state
machine transitions may be creating an event loop instead of correctly
handling the error.
The log file was quiet at this point, the last entries were:
2014-06-19 08:57:36 INFO There seems to be nothing to build
2014-06-19 08:57:36 INFO Requested artifact is built
2014-06-19 08:57:36 DEBUG InitiatorConnection: sent to 10.24.1.215:53818: {'mess
age': 'Need to build 0 artifacts', 'type': 'build-progress', 'id': 790629564}
2014-06-19 08:57:36 DEBUG Notifying initiator of successful build
2014-06-19 08:57:36 DEBUG MainLoop.remove_state_machine: <BuildController at 0xb
6c554c, request-id InitiatorConnection-93>
2014-06-19 08:57:36 DEBUG InitiatorConnection: sent to 10.24.1.215:53818: {'type
': 'build-finished', 'id': 790629564, 'urls': [u'http://hawkdevtrove:8080/1.0/ar
tifacts?filename=861f640923494ca3626bbd65655b350ce1bebea4c0bf7a57693bc06ed122cef
4.system.devel-system-x86_32-chroot-rootfs']}
2014-06-19 08:57:36 DEBUG InitiatorConnection: 10.24.1.215:53818: closing: <Json
Machine at 0xc6cb22c: socket 10.24.1.164:7878 -> 10.24.1.215:53818, max_buffer 1
6384>
2014-06-19 08:57:36 DEBUG MainLoop.remove_state_machine: <InitiatorConnection at 0xc6cbcec: remote 10.24.1.215:53818>
2014-06-19 08:57:36 DEBUG MainLoop.remove_state_machine: <JsonMachine at 0xc6cb22c: socket 10.24.1.164:7878 -> 10.24.1.215:53818, max_buffer 16384>
2014-06-19 08:57:36 DEBUG MainLoop.remove_state_machine: <SocketBuffer at 0xc6cbe2c: socket None max_buffer 16384>
This commit should improve matters a little: in future the log file will show
the ID of the SocketEventSource object and error we hit when calling its
write() function.
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When the given ref points to a specific commit, and it's already available in the
locally cached copy of repo, there's no need to update the repo.
If the ref points to a branch or tag, and the user didn't pass --no-git-update, the
locally cached copy of the repo will still be updated.
This should speed up many Morph commands.
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed by: Richard Maw <richard.maw@codethink.co.uk>
Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Instead of taking the name of a cluster morphology and zero or more
parameters for overriding the cluster morphology, morph deploy should
take the name of a cluster morphology and the names of zero or more
system deployments that are defined in the cluster morphology. If no
deployment names are given then all deployments are deployed.
|
| | | | |
|
| |/ /
| |
| |
| |
| | |
The names of deployments in cluster morphologies will need to be unique
in order for the deployment of selected systems to work.
|
| |/
|
|
|
|
| |
We want to supply the end of options option before
the SYSTEM_INTEGRATION_PATH so that the system integration path
doesn't get interpreted as an option if it happens to begin with a -
|
| |\
| |
| |
| |
| |
| |
| |
| |
| | |
Several reviews; all were happy with the code but not all were sure that
this is the correct approach to take. I have merged, because we need this
change to make Baserock releases easier. We can revisit it in the future
if we want to provide semantic versioning for Morph, instead of our
current "report the version of the containing OS image instead"
approach.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This still (ab)uses 'git describe', in order to get an '-unreproducible'
postfix when the tree has uncommitted changes.
Previously we tagged morph.git with the name of the corresponding
Baserock release that it was to be part of. This made the Baserock
release process awkward. We needed to first create and test the
release images, then (once we knew the version of Morph we were
releasing was good) tag morph.git, and then build all the release
images again so that the new tag would be picked up and the output of
`morph --version` would be correct.
|
| |\ \
| | |
| | |
| | | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | | |
| | |
| | |
| | |
| | |
| | | |
There is code that assumes these exist in at least one place:
StagingArea.abort(). That code should be fixed, but we should also stop
deleting them every time we run 'morph gc'.
|
| | |/
|/| |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If btrfs is not present in the kernel we end up with strange output like
this:
Error creating disk image2014-06-10 16:00:40
[devel-system-x86_64-generic][my-raw-disk-image][rawdisk.write]Failure
to create disk image at /src/tmp/testdev.img
ERROR: Command failed: mount -o loop /src/tmp/testdev.img
/src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE
mount: mounting /dev/loop0 on
/src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE failed: Device or
resource busy
To avoid this confusing error, Morph should explicitly check first.
|
| |\ \
| | |
| | |
| | |
| | |
| | | |
Reviewed by:
Lars Wirzenius
Richard Maw
|
| | | |
| | |
| | |
| | |
| | |
| | | |
After a failed attempt to connect to the controller node
the initiator will wait 30 seconds before attempting a reconnect,
if this reconnect fails the initiator gives up.
|
| | | | |
|
