| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If the destdir path returned when creating a staging area is a unicode
string, then when attempting to `os.walk(destdir)`, it will encounter
unicode errors if there are file paths in the destdir that are not
representable as unicode strings.
For various as-yet unknown reasons, when building stage-2 eglibc it
produces file paths that are not unicode compatible.
There was previously a patch to fix this issue with regards to creating
the metadata files, but it did not fix all the issues, because the build
at the time was local rather than distributed.
This is failing during a distributed build because morphologies are
serialised into json, and during deserialisation their string values are
left as unicode.
Rather than doing the byte-string conversion during deserialisation, I
have chosen to do it when the contents of the morphology are used,
because it's only at the point where it's used to create a file path,
that it matters whether it's unicode or not.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
We use tee to write the output to a file as well as to stdout.
Using Popen it should be straight forward to send the output to a pipe and
then read that pipe and write to wherever.
At the moment morph uses cliapp's runcmd rather than Popen. cliapp's runcmd
is a blocking call, so we're not able to read from the pipe until the
command has completed, which prevents real time logging to a number of files.
One solution to this problem might be to spawn a thread which opens a pipe
to the command being executed, the thread then reads from the pipe and writes
to our collection of logfiles.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Make the message "Installing chunk..." be chatty, i.e., only displayed
when the user turns verbosity higher. Most of the time the chunk is
already unpacked in the cache, so installing it takes a fraction of
a second.
Add a new message, at default verbosity, when a chunk needs to be
unpacked. This can take a while, so a message is appropriate, so the
user knows what is happening.
|
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| | |
Reviewed-by: Lars Wirzenius
|
| | |
| |
| |
| |
| | |
Somehow this was working on x86 even though it had no ability to
write to tempdirs, but on ARM it wasn't working.
|
| | |
| |
| |
| |
| | |
This changes the semantics of the use_chroot flag to instead mean
chrooting to /
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This improves upon the logic by allowing subdirectories to be
marked as writable.
This is not needed in its state here, but it will be built upon.
It also does not attempt to make symlinks read-only, since the symlink
resolution is done before chrooting, so there will be dangling links,
which cause linux-user-chroot to fail during the bootstrap.
This also uses the --chdir option of linux-user-chroot instead of
running a shell script to cd and run the command.
|
| |/ |
|
| |
|
|
|
|
|
|
|
| |
This uses the same logic as when a build fails, so it's been
consolidated into `StagingArea.abort()`.
You could argue that if a build fails before any commands are run,
then there's nothing interesting to see, but it will be useful if the
hardlink/tarball extract algorithm fails in some corner case.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Paul had file system problems which led to a partially constructed
chunk hardlink cache, which caused later builds to fail, since they
got a partially extracted chunk.
This patch fixes the case where the failed extract caused unreproducible
builds, but it's possible to corrupt the hardlink cache in other ways.
Read-only btrfs subvolumes would fix this, but either tie us further to
btrfs, or complicates the codebase with fallback logic.
|
| |
|
|
|
|
| |
Now, inside the temporary directory we will have the following
subdirectories: chunks, staging, failed and deployments. The failed
directory will contain the staging areas of failed builds.
|
| | |
|
| | |
|
| |
|
|
|
| |
If for example /bin is a symlink, we shouldn't and can't mount the host
system's /bin over the top.
|
| |
|
|
|
|
|
|
|
| |
Morph no longer supports setting the prefix using the --prefix
argument / setting. This was only used in tests and during bootstrap.
If a chunk build-depends on a chunk within a stratum which has
a custom prefix, that prefix is appended to the PATH in the build
environment.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Allowed values:
staging: build with a staging chroot (default)
test: build with the host's tools
bootstrap: build with the host's tools, and do not include this
chunk in the final stratum artifact
In the past, 'normal mode' has been used to describe building a chunk
with the host's tools. We don't want that mode to ever be used,
because it is a huge hole in reproducability, but we need to keep it
around to avoid making Morph's cmdtest suite depend on Baserock.
Hopefully naming it 'test' should discourage potential abusers.
It is unfortunate that the build tests now take a separate code path
compared to real-world usage of Morph. However, this is necessary to
avoid a circular dependency between Morph's test suite and the
build-essential stratum in Baserock.
We do whole-build testing of Baserock, too, so the 'staging' code path
is still tested outside of Morph. However, testing a staging area
requires populating it with at minimum a working shell, and this is a
bit too complex to go in Morph's test suite.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reorganise the build_artifact() and build_artifacts() functions to
allow more complex work when setting up chunk builds in
build_artifact().
The staging area now holds the BuildEnvironment object (the
environment variables that should be set during build). This makes sense
because all build commands should be run inside the staging area and
therefore through the StagingArea object.
The BuildEnvironment object is now considered immutable after it is
created. The environment is used in cache key computation when
computing what artifacts are required; if it changes after that point
we risk either computing different artifact keys for the same artifact
or missing data in the cache key that should be included in the hash.
Better to force changes into a separate 'extra_env' variable.
|
| |
|
|
|
| |
This involved having a staging area with split personalities
and was generally a bit ugly.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
When we hardlink the staging area tree, we were checking if the
target file already exists with os.path.exists. Unfortunately, this
follows a symlink, and in this case, we don't want to: if the
symlink target exists, and is a symlink, we want to remove the symlink
only, and we don't care about its target.
The target file may exists because of overlapping files in chunks.
Reported-By: Paul Sherwood
|
| | |
|
| |
|
|
|
|
| |
Instead of repeatedly unpacking tarballs into a succession of staging-
areas, it will unpack each tarball only once, then hardlink all the files
into the staging-area instead.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit c859d70d86423a52bc7053abf64e9ca21f62a487, reversing
changes made to 3085a672d1e8b5177be33f0463385de72a0ef5bf.
Unfortunately, hardlinking and linux-user-chroot both break builds
in various ways. Hardlinking breaks the bootstrap process by creating
symlinks like /usr/libexec that can then not be overwritten with real
files by install scripts from morphologies like gcc.
linux-user-chroot caused problems by breaking privileged operations such
as chgrp and CAP_SETFCAP. As a consequence, chunks like util-linux and
libcap can no longer be built.
|
| |
|
|
| |
This also makes coverage.py be silent on Debian squeeze.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The patch gives two things.
Improves morph build time by reusing decompressed files of
chunks/stage-fillers with hardlinks from the chroot. Rather than
decompressing each time into each chroot.
Original: real 5h 17m 47s
Hardlink: real 2h 52m 27s
It uses linux-user-chroot to create the chroot and make all but
the basics readonly.
|
| |
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
| | |
|
| |
|
|
|
| |
This import was forgotten when the artifact extract code was moved
into bins, so it could be used in builder
|
| |
|
|
|
| |
Before only the staging area had symbolic links handled, now
strata and systems will have the same logic.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
We had a problem where something in the bootstrap would fail because
two different chunk artifacts had a path (/tools/lib64) exist
as a directory exist as a symlink (to /tools/lib) versus a directory,
and morph confusing things.
The new logic should handle this better.
|
| |
|
|
|
|
| |
I really don't like this, but it causes the bootstrap to fail
because /bin/bash exists for compatibility and the stratum
also creates /bin/bash.
|
| |
|
|
|
|
|
| |
chroot may not be installed to /usr/sbin/chroot, this breaks bootstrap
because chroot is installed in /tree/tools/sbin.
PATH should be set, so putting the absolute path in should be unnecessary
|
| | |
|
| | |
|
| |
|
|
|
| |
The -x to sh didn't actually produce any output to help in my
debugging, so I remove it now.
|
| |
|
|
|
|
| |
We'll get back to linux-user-chroot later, but to do that,
we need to sort out how we deal with the install-commands
without needing real or fake root for them.
|
| |
|
|
|
|
| |
When we are not using --staging-chroot (e.g., bootstrap), we need to
specify the temporary directory separately, because it needs to obey
things like $TMPDIR.
|
| | |
|
| |
|
