| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
| |
This adds a new optional field to system morphologies:
"configuration-extensions".
The deployment plugin relies heavily on code from the branch and
merge plugin. This needs to be eventually fixed by refactoring
the codebase so that the shared code is in morphlib and not in
plugins. However, doing that is beyond the scope of adding a
deployment plugin.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For Morph to work it's necessary for the staging area to contain a
full FHS heirarchy. However, if the staging area is empty we want to
give the user an obvious error. Previously we would say something like:
OSError: [Errno 2] No such file or directory: '/tmp/ccache'
This makes it seem like the problem is with ccache. With this patch,
we will instead get as far as:
/bin/sh: No such file or directory
This should make the real problem clearer.
Also, aborting while trying to mount the ccache directory causes the
/proc and /dev/shm mounts to persist which is fixed by this patch.
|
| | |
|
| |\
| |
| |
| | |
Merge commit fixes up a copyright year and removes EOL whitespace
|
| | |
| |
| |
| |
| | |
This can go away when we have made a release with yaml in it, and its
staging filler.
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Current versions of build-essential in the Baserock morphologies
don't contain distcc, but Morph tries to use distcc anyway. Since
useful operation of distcc requires setting up a distcc network,
there's no reason for it to be on by default.
This change fixes bootstrap, and means that users of the
build-essential staging filler no longer have to set no-distcc=true
manually.
In the future, cliapp will grow automatic boolean negation, which
will allow us to turn this setting into a 'distcc' setting that
defaults to false rather than the current rather ugly double negative.
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Invalid text changed to be something that doesn't parse as YAML
either, and catch convert the YAMLError to the expected exception.
Ideally there wouldn't be any `#pragma: no cover`s, but I could not
trigger these code paths.
|
| | |
| |
| |
| |
| | |
Tests are currently broken, one because invalid JSON
can be valid YAML, and coverage is incomplete.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This way everyone who needs json or an OrderedDict doesn't have to worry
about it.
Currently morph2 is the only one that needs this complex behaviour,
other users of the json module don't need object_pairs_hook.
|
| |\ \ |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
This can be used to rebuild an artifact with the exact same version
of morph. Eventually there may be a tool to do this, but it allows
for expert diagnosis until that point.
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When morph is built, it writes various version information from git
into morphlib.
When morphlib is loaded it attempts to read this version information.
if it cannot be found then it checks whether morphlib is being run from
inside a git checkout, if it is then it reads the information that way.
If it isn't in a git checkout then it raises an exception as builds made
in such a fashion are not reproducible.
The git version information retained is:
1. The output of git describe
This is a relatively human-friendly way of knowing a version and
gives a reasonably short output string.
This will end with `-unreproducible` if there were uncommitted changes.
2. The commit sha1, so the exact part of Morph's history can be found
3. The tree sha1, so if the branch has been rebased rather than
merged such that the commit is lost, you may still be able to find
it, though it requires a git-wizard to check it out
4. The branch of morph, so that it's easier to see if the
Further possible changes to increase reproducibility include:
1. Not allowing `python setup.py build` if there are uncommitted changes
2. Failing to run with uncommitted changes (recommended against since it
will just annoy developers who are making changes to morph, and make
them commit just to shut it up, then destroy the history later)
Requiring an extra flag to build in this case may work better.
3. Reading the uncommitted changes into a tree object and including
that would allow it to be recovered if the tree was later committed.
4. Checking whether the commit has been pushed upstream as well.
Too annoying to work.
|
| |\ \
| | |
| | |
| | |
| | |
| | | |
'origin/baserock/richardmaw/temp-build-ref-commit-fix'
Broke a long line to make test suite pass.
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Before it would determine which files are changed by comparing your
working tree to
This gets even weirder, since it's effectively comparing your working
tree, the last commit on the temporary build ref, and the last commit
of your HEAD, so committing a change to remove a file isn't noticed,
because it was in the temporary build branch.
It could also cause a strange case of a file being both added and
untracked.
Now the working tree and HEAD are compared, and committed on top of
the temporary build ref.
Better handling of the status output is still required, a deleted
file is treated identically to an added one if it is only removed
in the index.
It would also be nicer to keep the user's index, since they may have
added or removed files from it.
|
| | | |
|
| |/
|
|
|
|
|
|
| |
After installing libraries into the staging area, we need to run
ldconfig to update the shared library cache.
I presume that this wasn't required until now because the cache in
the staging filler tarball was enough.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
ssh://git.baserock.org/baserock/baserock/morph
This includes the following fixups:
- altering the bootstrap script to install ordereddict and simplejson.
- Adding a comment to clarify that it is intentional to use simplejson
if collections does not have OrderedDict
- Amending the copyright years to include 2013
|
| | |
| |
| |
| |
| | |
As we use ordererdict for python < 2.7 as well
See 37f8d84d7ab356169bf7c04bdecb39a61b85c88d
|
| |/ |
|
| |\
| |
| |
| | |
Fixed a copyright year to make ./check happy.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
This patch makes morph store its ccaches by the url's basename instead
of using the whole url. This fixes the problem of morph choking when it
tries to mount a directory with a colon in its path, and also makes
all clones of one repository use the same cache (probably). This will
cause projects with the same name to use the same cache, but that is
not likely to cause serious problems.
|
| |/
|
|
|
|
|
|
| |
glob.glob() can return results in any order. In practice it varies
at least according to the file system being used. This means that test
results may be different. To avoid this, we should always use
sorted(glob.iglob()) instead, so that the code always behaves in the
same way given the same inputs.
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
The code took some refactoring. The core functionality is now all inside
one function with make_available() separate, as this is used other places.
The code is still far from perfect, but will hopefully be rewritten to
use the new abstractions of system branches etc. soon
|
| |/
|
|
|
| |
This ensures that we don't try to build the same stratum from two different
refs.
|
| |
|
|
| |
Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is set up so that each individual project repository has its own
ccache, all under one defined directory.
The top-level ccache directory is added as the setting
'compiler-cache-dir', and defaults to $cachedir/ccache.
When a build is performed, this will bind-mount a project's ccache into
the /tmp/ccache of the staging-area and set up the environment variables
so that ccache will be used (if appropriate executables are installed
to /usr/lib/ccache in the staging-area).
In addition, this removes code for ccache-remotedir, as it is unrelated
to this implementation of ccache.
Reviewed-by: Jannis Pohlmann <jannis.pohlmann@codethink.co.uk>
Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| |
|
|
|
|
|
|
|
|
|
| |
The committer information in the environment used to run git in morph
tag is not needed. In morph build it makes sense as morph commits without
the user knowing. With morph tag, it's the user that decides to create
the commit and tag.
There is something weird going on, where morph tag may end up generating
commits with different SHA1s on different machines. The full log output
in the morph tag tests might help investigate what happens.
|
| |\
| |
| |
| | |
Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| | |
This cuts down on the number of HTTP requests made while computing th
build order (> create source pool > traverse morphs). In an example
system I was building this patch reduced the requests from 501 to 195,
reducing the time to building the first chunk from 123s to 18s.
|
| |\ \
| | |
| | |
| | | |
into baserock/merge-queue
|
| | | |
| | |
| | |
| | |
| | | |
This was done to ensure tests.branching/branch-fails-if-branch-exists
always passes, but also seems like the right approach in general.
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously some code used `git show-ref`, which is wrong -- given two
refs named 'alpha/master' and 'master', `git show-ref master` will
return both, sorted alphabetically. This can lead to build failures,
etc. due to refs resolving to the wrong SHAs.
We should also use `git rev-parse --verify` to verify SHA1s, which
we previously did with `git rev-list`.
Finally, `git rev-parse --verify` is more than twice as fast as
`git show-ref`.
|
| | | |
|
| |\ \
| |/
|/| |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In order to make releases and freeze system branches entirely, we
need to be able to 100% petrify a system branch (that is, resolve
ALL refs into SHA1s) and tag this state to be able to check it out
again later.
This is essentially what "morph tag" does. It takes a tag name
and an arbitrary amount of arguments to "git tag", petrifies all
morphologies of the current system branch behind the scenes, creates
a dangling commit and attaches an annotated tag to it.
Petrifying in this case means that all refs used for chunks are
resolved into commit SHA1s. For stratum and system morphologies,
the refs are replaced by the name of the tag that's being created.
The "tag" command also supports tagging when stratum morphologies
are spread across multiple repositories. In this case, it will
include all statum morphologies from other repos in the tag commi
in the branch root repo. The references to these morphologies are
updated so that they point to the branch root repo and the tag
being created.
This commit also adds a few tests for "morph tag" to verify that
all this works.
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This changes the cache key generation so that it will ignore unimportant
fields of the morphology, e.g. description, build-depends, chunks.
description is unimportant because it does not affect building, and
build-depends/chunks are unimportant because they are already considered
|
| | |
| |
| |
| |
| | |
The lookup method will only find the first entry, but previously
would allow later entries to be found by iterating.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Before this change, if a stratum had dependencies listed, then
morph would add that stratum to the list of things to build
multiple times, resulting in more things being built.
This was not a major problem, since it would just result in more cache
lookups than necessary, but lookups are more expensive on remote
artifact caches, and anything attempting to find which artifact would
be the result of a build would get confused by there being many
top-level artifacts with no dependencies.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
The test relied on undesired behaviour, since it would create
the same source, since the sha1 is not used in the sourcepool's key.
Now it creates a new source pool and replaces the source it is being
compared to.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit c859d70d86423a52bc7053abf64e9ca21f62a487, reversing
changes made to 3085a672d1e8b5177be33f0463385de72a0ef5bf.
Unfortunately, hardlinking and linux-user-chroot both break builds
in various ways. Hardlinking breaks the bootstrap process by creating
symlinks like /usr/libexec that can then not be overwritten with real
files by install scripts from morphologies like gcc.
linux-user-chroot caused problems by breaking privileged operations such
as chgrp and CAP_SETFCAP. As a consequence, chunks like util-linux and
libcap can no longer be built.
|
| |
|
|
| |
This also makes coverage.py be silent on Debian squeeze.
|
| | |
|
