| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
|
|
|
| |
This should help clear the way for the eventual removal of deprecated
workspace code and related commands: morph checkout etc
Change-Id: I66298c4f9f790848962bd70ab434bef96cf96cb9
|
|
|
|
|
|
| |
These must have been missed when we removed the edit command.
Change-Id: Ia0eca7eab50dec34f2e02193d3ff676b0e78b33c
|
|
|
|
|
|
|
| |
Static analysis showed this code is not being used, and we can always
bring it back if necessary through git.
Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
|
|
|
|
|
|
|
|
|
|
|
| |
fixes
ERROR: [Errno 2] No such file or directory:
'/usr/lib/python2.7/site-packages/morphlib/schemas/defaults.json-schema'
when running morph build
Change-Id: I440a274f9eff4fe86450064538ebc1ed36945ea2
|
|
|
|
|
|
|
| |
Tests were broken since https://gerrit.baserock.org/#/c/1418/ was
merged.
Change-Id: I4e5a139d84946514634214938990cca3c86f4512
|
|
|
|
|
|
|
| |
YBD has been doing this for a while and seems like it is faster as a
result.
Change-Id: I8f95a53195cdbc2c75c06a8abe9eb089a84b1c1b
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently Morph enforces that chunk A must be defined before anything
that build-depends on it. YBD doesn't enforce that.
Definitions format at <http://wiki.baserock.org/definitions/current/>
doesn't mention ordering currently. I propose that we make Morph be
permissive about ordering, like YBD is, and update the spec to mandate
no restrictions on ordering.
Since behaviour was previously undefined, making Morph be more
permissive about this shouldn't require a new version number of the
definitions format.
I still think we need to make sure stratum .morph files are ordered
logically, but that is in the realm of 'code style', it shouldn't be
being enforced by a build tool.
Change-Id: I425f2e5b9dfb62e4a26ed11f5c50e3978a0dd1a6
|
|
|
|
|
|
|
|
|
|
|
| |
Accordingly to [1], the listdir function (called by os.walk) returns a list of
filepaths on an arbitrary order. Sort the list of file paths to produce
always the same list.
[1]: https://docs.python.org/2/library/os.html
Change-Id: I4bb9842b1722f27a8becb9c50391cda089bb0a33
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When attempting to get a repo with no commit at the ref we want,
we get a 'No such file or directory' error, which is confusing,
this is because we attempt to remove the directory we plan to clone
into, but if there's no commit at the ref we want then we don't
get as far as running git clone, so the directory doesn't exist,
and we fail to remove it.
This commit improves the error message by catching the InvalidRefError
separately.
Change-Id: Ibca20dd995d858713e56ab834daa97a8297cafc8
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit makes it an error for a system to contain duplicate chunk sources,
example error message below,
ERROR: Multiple `syslinux' chunks detected:
upstream:syslinux|d715b39c0801ecea5e52f9029cea7c76320f93cf|strata/bsp-x86_both-tools/syslinux.morph|syslinux
upstream:syslinux|2aab8555987b547b617cbb887e61083fece01541|strata/bsp-x86_64-generic/syslinux.morph|syslinux
Multiple `nasm' chunks detected:
upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_64-generic/nasm.morph|nasm
upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_both-tools/nasm.morph|nasm
Change-Id: I1d1539a46ce6eb098d3a559295ab9a08d6d2865c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a new 'Defaults' class to represent definitions defaults
The Python 'jsonschema' module is used to validate the contents of the
Defaults file. This module is already included in Baserock 'build' and
'devel' reference systems by way of the 'openstack-common' stratum.
This commit embeds a copy of the JSON-Schema schema for the DEFAULTS
file. I think the canonical location of this schema should be in the
reference definitions.git, for now. In future, the schemas should maybe
have their own repos. Either way, Morph should embed a copy for the time
being so that we are sure the schema matches how Morph expects to parse
the file.
Morph's automated tests are all updated to use definitions version 7.
I removed most of the tests for built-in build systems, because the
built-ins themselves are no longer part of Morph. Only the mechanism for
defining them needs to be tested now.
Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
|
|
|
|
| |
Change-Id: I493fced8cf2664283923f6f41097ca991d3fc3de
|
|
|
|
|
|
|
|
| |
Some tests were removed in commit f4d3cbaf1d6e5c3bdbb1b2e2 but it seems
there are still code remnants that we don't need any more. There may be
more -- I haven't done a thorough job of this.
Change-Id: I0d7151402365f9e6f23a93a1d30df0198de9ef02
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When preparing the build directory for a chunk, Morph clones the Git repo
being built (and any submodules) into the staging area.
Instead of using `git clone --no-hardlinks`, the
morphlib.git.copy_repository() function uses 'cp -a' plus some faffing
to achieve the same thing a bit faster.
The 'cp -a' command will preserve the ownership and permissions of the
cached copy. These may not match up with which user is doing the build.
I found that files in my Git cache were owned by UID 1002. This caused
chunks that needed to access the .git directory at build time to fail
with strange errors. Worse, it would trigger a bug in Git[1] that leads
to a fork bomb, which would cause Linux to freeze up completely. This
occured even though I was building as `root`, because of the way
`linux-user-chroot` drops certain privileges: presumably, setting
SECBIT_NOROOT leads to CAP_DAC_OVERRIDE being unset.
To avoid this bug, the code now ensures the copied .git repos are owned
by the user and group who ran `morph`.
Another way to fix this would be to change the
morphlib.gits.copy_repository() function to use `git clone
--no-hardlinks`. This is what YBD does. I found that there is a slight
speed benefit to using the current code ... these are results of cloning
repos from the cache with the 2 methods:
With 'cp -a' plus a chown:
upstream:bison: 4.54 seconds average (10 results)
upstream:ybd: 0.13 seconds average (10 results)
upstream:linux: 40.51 seconds average (10 results)
With 'git clone --hardlinks':
upstream:bison: 6.23 seconds average (10 results)
upstream:ybd: 0.11 seconds average (10 results)
upstream:linux: 43.36 seconds average (10 results)
Test code is: <https://gist.github.com/ssssam/833e0ef8d04fb1fb6ff3>.
Ideally we would fix `git clone --no-hardlinks` to be faster, but we
may as well keep the existing code for the time being.
[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750687
Change-Id: Ieea87322ea7b7f62975b9480f877755665656217
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Loading .morph files is becoming a bit more complicated, as we need to
deal with the VERSION file, and possibly soon with a DEFAULTS file as
well.
The logic of loading and parsing .morph files is done either in the
sourceresolver module, or the morphloader module. This change means that
all users of the latter module can use the get hold of a
MorphologyLoader instance with VERSION already parsed. If DEFAULTS is
added then it is also simple to parse DEFAULTS.
Change-Id: Ib33756e9dbd078e38f12dd7f776c89584b178959
|
|
|
|
|
|
|
| |
Using app.status() produces nicer logs as the messages will have the
status prefix prepended.
Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
|
|
|
|
|
|
|
|
|
| |
Replace cliapp.shell_quote() with pipes.quote() to produce the chroot
scripts. Despite pipes.quote() being deprecated, it produces much less
escaping than cliapp.shell_quote(). This method should be replaced with
shlex.quote() when migrating to Python 3.
Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
|
|
|
|
| |
Change-Id: I276a00bb9117aaa2d4fcc7e187b846eedae65591
|
|
|
|
|
|
| |
... instead the more common autogen[.sh]
Change-Id: I3d797581708ffaefd1a1d4880253cfc2a3f5aa85
|
|
|
|
|
|
|
|
|
|
| |
Commit a4d8098b229592db40565747b0444b518bf8a6eb removed the `morph
foreach` command, which broke some `morph edit` tests. I don't think
we are ready to remove `morph edit` yet (documentation still mentions
it) but we can remove these tests, which require `morph foreach` to
work.
Change-Id: Id372760f69f31a46d1d872ee36ec6fdfb324ce34
|
|
|
|
|
|
|
|
|
| |
Looks like,
2015-07-05 16:08:10 [Build 1/304] [stage1-binutils] Fetching to local cache: artifact stage1-binutils-misc
stage1-binutils-misc[##################### ] 51.9/73.0 MB
Change-Id: Ib10f1cfaa0c1df80ae605ecfeb5b706c8d46c4a4
|
|
|
|
| |
Change-Id: Iedefd1745e85e1f30d45ee8d26b315760a542a0b
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pass the Source to the staging area constructor so that we don't need to
pass it as a parameter when we call some StagingArea methods. Also move
the creation of the build and destdir directories to the constructor so
that we can get rid of the chroot_open() and chroot_close() methods.
Also, provide API to retrieve the relative locations for buildir and
destdir.
Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
|
|
|
|
| |
Change-Id: Iad95af65bd5c528d2e72f5b2ffa80a01152f50ff
|
|
|
|
|
|
|
|
| |
We're beginning a transition away from workspaces,
foreach in particular is possibly one of the least used commands,
and is only mentioned in tutorials explicitly marked as 'old'.
Change-Id: I2c95ce689bb5550bf50feb6b55be6c58671c4c4a
|
|
|
|
| |
Change-Id: Icec7e948c60decefe5a8bab6141bc89603c265b7
|
|
|
|
| |
Change-Id: I6c45763fb4d9f304695c1894625ced36bda36f89
|
|
|
|
|
|
|
|
|
| |
This change, suggested in story #16, causes the testing suite to use
the user-defined tempdir in /etc/morph.conf if it exists or continue
using /tmp if not. This will prevent the test suite from filling drives
that are not intended to be used for development work.
Change-Id: I94edabe4ea1eca7a55ece7424ece7bbd4b68c513
|
|
|
|
| |
Change-Id: I9e5b1a70946a87eb29009df2def3bd98ecc7ad2a
|
|
|
|
|
|
|
|
| |
ExtUtils::MakeMaker is preferred, Module::Build was meant to replace it
but essentially wasn't good enough, some projects still use Module::Build
though.
Change-Id: I124ee7b33f32167302e9bcb5299f6422f4fc346e
|
|
|
|
|
|
| |
Install to default locations rather than overriding.
Change-Id: Ieccd06afdf623eee0d09b8d29d40d147dc8c5e7c
|
|
|
|
|
|
| |
There are several perl build systems.
Change-Id: I17b5d61b7bffbdbcb9944e9e41449bec7a69c527
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Improve error messages and remove redundant information
* Stop displaying the containerised settings, as they are not very
helpful to debug a build failure
* Display the error code
* Change system integration commands logging to consistent with build
commands logging.
* Provide a log file in the staging directory for each chunk build
failure or system integration failure.
* Provide a chroot script in the staging directory for each chunk build
failure or system integration build failure. This script allows the
user to enter an environment similiar to the one where the build
failed.
* Fix a bug where the system integration commands stdout was being
omitted when these commands failed.
* Fix a bug where the error_message_for_containerised_commandline() was
being called with an empty error message, as the stderr was being
directed to stdout.
Chunk build failure output before the this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands
build failed
# configure
# # echo "configure-commands stdout"
configure-commands stdout
# # echo "configure-commands stderr" >&2
configure-commands stderr
# build
# # echo "build commands stdout"
build commands stdout
# # echo "build commands stderr" >&2
build commands stderr
# # echo "some more sdout"
echo "some more stderr" >&2
echo $PATH
foo
some more sdout
some more stderr
/tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
sh: foo: not found
ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c
'echo "some more sdout"
echo "some more stderr" >&2
echo $PATH
foo
':
Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball',
'/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True,
'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths':
['/src/tmp/staging/tmpglc5Sz/test.build',
'/src/tmp/staging/tmpglc5Sz/test.inst',
'/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc',
'/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz',
'cwd': '/test.build'}
Error output:
$>
Chunk build failure output after this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands
### CONFIGURE-COMMANDS ###
+ echo configure-commands stdout
configure-commands stdout
+ echo configure-commands stderr
configure-commands stderr
### BUILD-COMMANDS ###
+ echo build commands stdout
build commands stdout
+ echo build commands stderr
build commands stderr
+ echo some more sdout
some more sdout
+ echo some more stderr
some more stderr
+ echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
/tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
+ foo
sh: foo: not found
ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi
t_code=127)
$>
System integration build failure output before this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the
system integration commands
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while
running system integration commands
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while
building system
ERROR: Command failed:
baserock/system-integration/00-test-test-misc-0002:
Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'),
('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root':
'/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'}
Error output:
+ foo
baserock/system-integration/00-test-test-misc-0002: line 1: foo: not
found
$>
System integration build failure output after this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the
system integration commands
+ echo stdout
stdout
+ echo stderr
stderr
+ foo
baserock/system-integration/00-test-test-misc-0002: line 1: foo: not
found
ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration
commands failed (exit_code=127)
$>
Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stop moving staging areas of failed builds from the 'staging' directory
to the 'failed' directory. Moving staging areas make it very difficult
to debug build failures on the build essential chunks, as the paths set
on the configure scripts and some environment variables (e.g.
STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area.
This change will also make it easier to create scripts that chroot n
environment similiar to the one where the build failure occurred.
To make it still possible to safely do a build an run `morph gc` in
parallel, we use flock(2) to control access to the staging area
directory.
Also, move the `test_supports_non_isolated_mode` test into a different
class, as it requires a different SetUp() routine (the staging area is
contructed with different parameters).
Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
|
|
|
|
| |
Change-Id: I2e1ae43da6e7dd0d7ddcb51d3cd9bc1794bb80a5
|
|
|
|
|
|
|
| |
I need to use this outside of the 'sourceresolver' module. Also, the
'sourceresolver' module is too big.
Change-Id: I523bcc9555193b7369768441b72f1059e6adde5c
|
|
|
|
|
|
| |
Just for future debuggers.
Change-Id: Ie915a9f47fbf3ca56d6e75b816eaa5a4fb36bc88
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of logging every command for ourselves, use the shell X-ray
mode. As side effect, each command argument will now be printed out in
its evaluated form, which aids debugging.
Also, improve the visual separation between the different build steps,
and display those steps with finer granularity.
Change-Id: I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8
|
|
|
|
|
|
|
| |
This is probably a sign that something is broken, but it's better to
cope with it rather than crashing.
Change-Id: I4e9c40d0b7db031ea271143938db61c4ad70a888
|
|
|
|
|
|
|
|
|
| |
When I spotted that artifacts contained a .git directory I was worried
that it might indicate a Morph bug. It turned out to be due to the way
the 'test shell' used in the Yarn tests works. This change should save
anyone else from similar confusion.
Change-Id: I97b57eff7aa947506d4062a76e9803398e46c2b6
|
|
|
|
|
|
|
|
|
|
|
|
| |
The system-integration field in a chunk .morph file defines which of the
chunk artifacts the integration script should live. If you list an
artifact that doesn't exist, the integration script will not go anywhere
and the commands won't run.
Ideally system-integration commands will be become a bit more simple to
use, but hopefully this warning helps in the meantime.
Change-Id: I10a068e79eb46f98f0bd308a5caf3c445dda22ed
|
|
|
|
|
|
|
| |
Chunk commands will currently block if they read from stdin.
To fix this we set stdin to /dev/null so that the first read returns EOF.
Change-Id: I143396f4c443c098b2880c5925e6f0907cb4fb17
|
|
|
|
|
|
| |
They were all run onto the same line before with no linebreaks.
Change-Id: Ibdff91a23221034a3d345542f1268f8863cf4515
|
|
|
|
| |
Change-Id: I891d1b13ed0581b293fe6b09b3cc73af8fd81d67
|
|
|
|
|
|
|
|
|
|
|
| |
The previous release of the Baserock reference systems (baserock-15.25
tag) used Baserock definitions version 3, so this version of Morph can
still build the last release of the Baserock reference systems.
In the course of fixing up the cmdtest tests to use version 5
definitions, I removed a couple of bits of redundant code.
Change-Id: If9d480aa966d2ea5939cf04351f3d488cccea67f
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This makes more sense to me conceptually: the idea of the 'manual' build
system is you specify *everything* manually.
However, I made this change to work around a problem in the Yarn test
suite. If the definitions used for testing are set to version 5 or
newer, `morph build` will try to run 'strip' and 'objcopy', which fails
because they don't exist in the test definitions.
We could solve this by setting 'strip-commands: []' in the test chunk
.morph files, but this triggers an error further down where `morph edit`
fails with 'Field strip-commands not allowed in morphology
test-chunk.morph'. This bug is caused by the conditional way that
strip-commands are currently implemented in Morph. When we drop support
for versions of the definitions format older than V5, the bug will go
away, and the `morph edit` command is deprecated in any case, so I think
it makes sense to work around it instead of trying to fix it.
Change-Id: Ib5124d72466a77cf3f28ed3e14cc4c231bdce4c4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
tar will preserve ownership and permissions by default
if running as the superuser, whereas git doesn't track ownership,
the owner is set to the user running the clone.
This is a reproducibility hole since the resulting repo may differ
depending on whether the repo was obtained from a tarball or via a git
clone, but morph considers the repos to be equivalent.
Running tar with --no-same-owner ensures the files
written by tar will be owned by the user extracting the archive.
This only fixes half the problem,
lorry may also need to be modified to ensure files in the
tarball have the same mode as those in the git repo.
Change-Id: I849d054bc4574f9c0bbcfd608914306e52b7a22e
|
|
|
|
| |
Change-Id: Ife16546fc90e26919672f6658b30084bafa42f94
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This test, tests for a case that produces unhelpful results.
It asserts that if we don't have the repository locally, and we try to
fetch it remotely and the file doesn't exist, that we should have no
morphology.
What we'd actually like is for it to fetch the repository and try that
locally, at which point it would succeed.
The reason the flawed previous behaviour (that this test asserted to be
correct) failed to cause problems for us, is that in the normal course
of resolving refs, we would always either have found it remotely or
cached it locally, when we were doing ls_tree or resolve_ref operations.
Change-Id: I287eb1e5bc9c916b31f306b4f5f24b0a6d3f0559
|
|
|
|
|
|
|
| |
If we did end up trying to load a morphology that wasn't defined, we'd
get an unhelpful error about the yaml being invalid.
Change-Id: If1b182f1100430424a0fe62bd8938a88c1b9a11a
|