| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
| |
Rather than having a `get_commands` method to obtain missing commands
from the build system when they are needed, get the commands when
loading a morphology.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With os.walk, if the target of the link doesn't exist, or it is a link
to a file, it ends up in the basenames list.
If it is a link to a directory, it goes in the subdirs list.
There's a bug in the subdirsymlinks check, in that it checks if the
wrong file is a symlink, so it never returns them.
This was missed, since we did not have the cross bootstrap in CI.
This is not eligible for our yarn tests, since to trigger this would
require changes to the host system's rootfs, so it's a system-level test.
To test this properly requires putting the cross bootstrap in CI.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
|
|
|
|
|
| |
We want to supply the end of options option before
the SYSTEM_INTEGRATION_PATH so that the system integration path
doesn't get interpreted as an option if it happens to begin with a -
|
|
|
|
|
|
|
|
|
| |
We could just set stdout to subprocess.PIPE
then read from the pipe, but then we won't get the output
till the command's finished and some commands take a long time.
Using the logfile kwarg a file will be created by tee
and the output will be written to it in 'real time'.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The scripts will be created into the folder:
/baserock/sytem-integration/
The system integration commands have the following
syntax:
name: linux
kind: chunk
...
install-commands:
...
system-integration:
linux-libs:
00-depmod:
- depmod -a
70-more-integration:
- touch /baserock/FILE
- |
for FOLDER in $(ls /)
do
echo "$FOLDER"
done
linux-misc:
70-more-integration:
- echo "Hello world"
In this concrete example, the following files will be created:
$DESTDIR/baserock/system-integration/
00-depmod-linux-libs-0000:
#!/bin/sh
set -xev
depmod -a
70-more-integration-linux-libs-0000
#!/bin/sh
set -xev
touch /baserock/FILE
70-more-integration-linux-libs-0001
#!/bin/sh
set -xev
for FOLDER in $(ls /)
do
echo "$FOLDER"
done
70-more-integration-linux-misc-0000
#!/bin/sh
set -xev
echo "Hello world"
|
|
|
|
|
|
|
|
| |
This messes up the baserock-system-config-sync tool. Systemd does not
require /etc/fstab to exist in any case.
I have bumped the 'system-compatibility-version' field in this commit
to trigger rebuilding all system artifacts.
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Rationale
=========
This patch series implements the concept of stratum splitting.
For a long time we've had code to split a chunk into multiple artifacts,
however there's not been a way to split strata up, or systems select a
subset of the produced stratum artifacts to be included in the system.
This patch series implements the ability to split strata and have systems
include them in a way which still has the same behaviour if no rules
are specified, but with default rules that split chunk artifacts up into
various components, strata into runtime and development versions and has
systems include everything by default, but can be told to include less.
The default rules have chunk foo split up into -bins, -libs, -devel,
-doc, -locale and -misc.
These rules can be overridden in the chunk morphology by adding the new
'products' field, which lists match rules like the following:
products:
- artifact: libudev
include:
- (usr/)?lib(32|64)?/lubg?udev\..*
- artifact: udev
include:
- (usr/)?s?bin/udev*
- (usr/)?lib(32|64|exec)?/systemd/systemd-udevd
Strata are by default split into -runtime and -devel. -devel by default
contains chunks ending with -devel and -doc, -runtime contains everything
else.
Extra match rules can be added to a stratum similarly to chunks, but
instead of matching file names, they match artifact names.
products:
- artifact: core-python
include:
- "cpython-.*" # lazy shortcut to put all of cpython in this stratum
- "python-.*" # lazy shortcut to include all python chunks in
Additionally, in chunk specs, chunk artifacts may be assigned to stratum
artifacts, this takes precedence over products match rules in the stratum
and the default match rules. Assigning the chunk to `null` will discard
the chunk.
chunks:
...
- name: systemd
...
artifacts:
libudev: foundation-runtime
udev: foundation-runtime
systemd-doc: null
By default a system includes every produced artifact of every stratum
listed. Instead a subset can be specified in the stratum spec as follows:
name: tiny-system
strata:
- name: build-essential
...
artifacts:
- build-essential-runtime
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
For some reason we used to create a new Artifact object with the same
name as the Stratum morphology for our cache key.
This is non-sensical, since we already have an Artifact object and it
breaks splitting strata.
NOTE: cmdtest tests do not pass, since they list files and artifacts
produced, which has changed since the new default splitting rules were
added. The next patch fixes this, but was kept as a separate commit for
readability.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Filenames are now matched before chunks are constructed, so
bins.create_chunk now takes a list of relative file names.
bins.chunk_contents is gone, since this is now handled by passing
source.split_rules.partition the file names.
We now don't consider it to be a problem for directories to remain in
the DESTDIR after artifacts have been removed, since we need to handle
file matches implying their parent directories, and explicit matches
against directories.
NOTE: The bins_tests were broken in this patch, and are fixed in the
next. This was done to try and aid readability of the patch series.
Full functionality is still broken until stratum splitting is fixed.
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I think that it's confusing for both strata and chunk morphologies to
have a 'chunks' field, with the former listing sources and the latter
listing rules for splitting this source into artifacts.
The design for splitting strata has roughly the same idea, but operating
on chunk artifact names, rather than file names, so a name that can be
used for both was chosen.
Splits and artifacts weren't satisfactory names, so they're now called
'products'.
It was decided to break backwards compatibility of chunk morphologies
being able to specify 'chunks', since the format has changed, so extra
code would be required to translate the format, and the only users of
the 'chunks' field was the test suite, since there was no way to select
from the system, which chunk artifacts were included.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It is better to test whether a straum is empty here, since it will be
noticed earlier in the build, as soon as the morphologies are loaded,
rather than after they have all been parsed.
It is also conceptually nicer to put it here, since the morphologyfactory
was written to perform this kind of validation.
On a more practical note, the validation is moved here so that the test
for this error isn't masked by the test for no build dependencies.
To ensure tests still pass, we alter the stratum morphology used by
other unit tests to no longer be empty, and add an empty one to test.
|
|
|
|
|
|
|
|
|
| |
This uses the same logic as when a build fails, so it's been
consolidated into `StagingArea.abort()`.
You could argue that if a build fails before any commands are run,
then there's nothing interesting to see, but it will be useful if the
hardlink/tarball extract algorithm fails in some corner case.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cross-bootstrap is a way to build baserock on an architecture that
does not currently have Baserock. It can be used by `morph
cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact
that can be used as a root filesystem with a basic build environment
with a script named `native-bootstrap` which will build and install
every chunk in the system.
If done with a devel system, this will give you a suitable environment
for building a proper Baserock system.
This does not currently provide a kernel for the target architecture.
Apart from adding the cross-bootstrap plugin, it also makes the
following changes:
* Moves the lit of valid_archs into morphlib (instead of locally-scoped
in MorphologyFactory)
* BuildCommand takes an extra argument, build_env
* split BuildCommand's get_artifact_object into create_source_pool and
resolve_artifacts (plus changes things that use get_artifact_object to
use the new way)
* setup_mounts finds out whether to do so by whether build_mode is
'staging', instead of by whether the setting 'staging-chroot' is true.
* Makes ChunkBuilder's get_sources use the
morphlib.builder2.extract_sources() method, and moved
set_mtime_recursively into morphlib.builder2, since it's not currently
used anywhere else.
* moved ChunkBuilder's get_commands into the Morphology class (plus
changes to anything that used get_commands)
|
|
|
|
|
|
| |
I blithely replaced runcmd(['mv', src_dir, dest_dir]) with
os.rename(src_dir, dest_dir), without realising that mv does
extra work when the target is a directory that already exists.
|
| |
|
|
|
|
|
| |
Also clarify how the directory containing it gets created
to avoid a bit of code duplication.
|
| |
|
|
|
|
|
|
| |
We now only support one system-kind, so there's no need for the
factory stuff, and at this point it only serves to obfuscate
and complexify.
|
| |
|
|
|
|
|
| |
Also delete the infrastructure for disk-based system-kinds
from builder2.py.
|
|
|
|
| |
It's just 1 system call instead of at least 3.
|
|
|
|
|
|
| |
Now, inside the temporary directory we will have the following
subdirectories: chunks, staging, failed and deployments. The failed
directory will contain the staging areas of failed builds.
|
|\
| |
| |
| |
| |
| |
| | |
Reviewed-by: Richard Maw
I, Lars, note that we have an old version of CoverageTestRunner
in Baserock. The new version hides the spurious logging messages.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It is almost never a good idea to catch all exceptions, and then do
nothing about them. This patch logs all caught exceptions so that the
user has some possibilty to debug what is happening.
Also, make ./check check for bare excepts and fail the test suite if it
finds anything.
|
| | |
|
| | |
|
| | |
|
|/ |
|
|
|
|
|
| |
Removed trailing comma, corrected copyright year, used #pragma: no cover
to exclude the exception class form coverage and allow ./check full to run.
|
|
|
|
|
| |
Added new cliapp.AppException class to raise error when attempting to
build an empty stratum (no dependencies) and raised it appropriately.
|
|
|
|
|
| |
This is necessary so that the build-essential stratum doesn't end up
containing three different compilers.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reorganise the build_artifact() and build_artifacts() functions to
allow more complex work when setting up chunk builds in
build_artifact().
The staging area now holds the BuildEnvironment object (the
environment variables that should be set during build). This makes sense
because all build commands should be run inside the staging area and
therefore through the StagingArea object.
The BuildEnvironment object is now considered immutable after it is
created. The environment is used in cache key computation when
computing what artifacts are required; if it changes after that point
we risk either computing different artifact keys for the same artifact
or missing data in the cache key that should be included in the hash.
Better to force changes into a separate 'extra_env' variable.
|
|
|
|
|
|
| |
That means that bootstrapping Baserock is currently not possible with
this branch of Morph, but there's no reason it cannot be bootstrapped
using an older version of Morph instead.
|
|\
| |
| |
| |
| | |
Conflicts:
morphlib/morph2.py
|
| |
| |
| |
| |
| |
| |
| |
| | |
We already have configure-commands. Add pre-configure-commands and
post-configure-commands. Likewise for build-command, test-commands,
and install-commands.
Added-to-pacify: Rob Kendrick
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Within a linux-user-chroot, we do not want to allow arbitrary code to
create device nodes, but still want it to be possible to create a device
node. This commit creates and handles the 'devices' field in a chunk
morphology, which takes:
* filename. A string, e.g. "/dev/null"
* uid. The ID of the user the file belongs to, e.g. 0 for root.
* gid. The ID of the group the file belongs to, e.g. 0 for root.
* type. A string of either 'c' for a character device or 'b' for a block
device.
* major. The device's major number.
* minor. The device's minor number.
* permissions. A string of the octal number that would be passed to chmod
e.g. '0777'
|
|/
|
|
|
|
| |
Instead of repeatedly unpacking tarballs into a succession of staging-
areas, it will unpack each tarball only once, then hardlink all the files
into the staging-area instead.
|
|
|
|
|
|
|
|
|
| |
This is required for reproducing the morphologies that created the
artifacts.
It is possible, but expensive, to `unexpand` a url, provided your
repository aliases are the same as the morphology's, but this may
not always be the case.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For Morph to work it's necessary for the staging area to contain a
full FHS heirarchy. However, if the staging area is empty we want to
give the user an obvious error. Previously we would say something like:
OSError: [Errno 2] No such file or directory: '/tmp/ccache'
This makes it seem like the problem is with ccache. With this patch,
we will instead get as far as:
/bin/sh: No such file or directory
This should make the real problem clearer.
Also, aborting while trying to mount the ccache directory causes the
/proc and /dev/shm mounts to persist which is fixed by this patch.
|
|
|
|
|
|
| |
This can be used to rebuild an artifact with the exact same version
of morph. Eventually there may be a tool to do this, but it allows
for expert diagnosis until that point.
|
|
|
|
|
|
|
|
|
| |
This patch makes morph store its ccaches by the url's basename instead
of using the whole url. This fixes the problem of morph choking when it
tries to mount a directory with a colon in its path, and also makes
all clones of one repository use the same cache (probably). This will
cause projects with the same name to use the same cache, but that is
not likely to cause serious problems.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is set up so that each individual project repository has its own
ccache, all under one defined directory.
The top-level ccache directory is added as the setting
'compiler-cache-dir', and defaults to $cachedir/ccache.
When a build is performed, this will bind-mount a project's ccache into
the /tmp/ccache of the staging-area and set up the environment variables
so that ccache will be used (if appropriate executables are installed
to /usr/lib/ccache in the staging-area).
In addition, this removes code for ccache-remotedir, as it is unrelated
to this implementation of ccache.
Reviewed-by: Jannis Pohlmann <jannis.pohlmann@codethink.co.uk>
Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit c859d70d86423a52bc7053abf64e9ca21f62a487, reversing
changes made to 3085a672d1e8b5177be33f0463385de72a0ef5bf.
Unfortunately, hardlinking and linux-user-chroot both break builds
in various ways. Hardlinking breaks the bootstrap process by creating
symlinks like /usr/libexec that can then not be overwritten with real
files by install scripts from morphologies like gcc.
linux-user-chroot caused problems by breaking privileged operations such
as chgrp and CAP_SETFCAP. As a consequence, chunks like util-linux and
libcap can no longer be built.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The patch gives two things.
Improves morph build time by reusing decompressed files of
chunks/stage-fillers with hardlinks from the chroot. Rather than
decompressing each time into each chroot.
Original: real 5h 17m 47s
Hardlink: real 2h 52m 27s
It uses linux-user-chroot to create the chroot and make all but
the basics readonly.
|
|
|
|
|
|
|
| |
gzip files can store the name and date of the uncompressed original, and
this is displayed in some GUI archive managers, or when gunzip -N is used.
In our case the original filename is something like 'tmpXsgaY15', which
is useless information and serves only to confuse.
|