| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
| |
The helper class, Factory, has unit tests, which is why it's currently
separate. It may later get integrated with BlobBuilder, or the other
way around.
Classes that don't have unit tests are marked out of coverage.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
If the remove method is insufficient (because it gets run as non-root,
but some of the stuff needs root to remove them), then we need to fix
the method, not replace it with other things.
The Tempdir.clear method was not used anywhere, so YAGNI and removed it.
|
| |
|
|
|
|
| |
We don't do releases, so having the version number in the source is
misleading. If and when we start doing numbered releases, it's easy
to add back. However, YAGNI and all that.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Also, change builder.py to always obey the --max-jobs setting, unless
a morphology has a max-jobs field.
The defaults have been chosen so that they work for everyone equally
well. It may be useful to have a local mirror and then set the options
to point there, but it's not reasonable to try to guess such things,
so the defaults can be adapated to that.
Collect the defaults into one place so they're easier to overview.
The cliapp interface for adding settings is verbose enough that the
defaults were getting buried.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
This we a proof-of-concept, and we want to explore various options of
how to do this, so let's not keep the code in morph. The code also has
no tests, so it is already likely to have bit-rotted.
If we decide we want to, we can always resurrect it from git history.
|
| | |
|
| |
|
|
|
|
| |
This adds options --staging-filler and --staging-chroot.
The wisdom of these options needs to be re-considered at
some point, but for now they're OK.
|
| |
|
|
| |
This reverts commit 63cab3e0e2f550a0bfe470d018063a8b91a759bd.
|
| | |
|
| | |
|
| |
|
|
|
|
| |
This will not affect or overwrite TMPDIR, so all other parts of morph
and Python (like the multiprocessing library) will still use TMPDIR
(e.g. /tmp) for temporary files and directories.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This change requires the latest version of cliapp to be installed.
Rather than creating an archive from the cached git repositories,
moving this tarball archive into the cache and then extracting it from
there into the build directory, we now directly extract the sources into
the build directory.
We also recursively extract all submodules into the corresponding paths
inside the build directory. (Recursing has not been tested yet,
however.)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This requires build-single to take a dependency context tuple when
building chunks of a stratum. This context tuple is the surrounding
stratum which is used to construct the dependency graph in the worker
and then do a breadth-first search to collect all dependencies that need
to be added to the staging area.
Implementing this required a few hash/eq changes in Blob, Morphology
and Treeish as well as a few adjustments in the corresponding unit
tests.
|
| |
|
|
|
|
| |
Computing the cache ID will not update all dependencies, just some
of them (e.g. the chunk repos a stratum morph needs). Resolving
the dependency graph, however, will update everything.
|
| |
|
|
|
| |
...while at the same time making sure that all dependencies are marked
for staging prior to building the blob itself.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
All BuildWorker subclasses are likely to make use of the multiprocessing
library anyway, so most of the functionality can be shared.
All workers now have an ident (e.g. "user@hostname" or "local") which
is used by the BuildController to generate enumerated names for them
(e.g. "user@hostname-1", "user@hostname-2" or "local-1" and "local-2"),
which makes it easier to identify who does what.
The RemoteBuildWorker now runs "ssh HOSTNAME fakeroot morph" for
building stuff remotely.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit introduces four new classes:
BuildController:
* takes an app instance and a tempdir
* allows to add BuildWorker objects
* provides a build() method that takes a set of blobs and
a build order that is then built by assigning work to the
build workers as needed
* the build() method takes care of polling the workers for
their state, moving them between busy and idle states
reliably, collect and print their output in a non-confusing
order, and makes sure to wait for all workers to finish
before processing the next group in the build order.
* at this point, when waiting for one or more workers to
become idle to assign them another blob to build, the
controller always picks the worker that has been idling
for the longest period of time. this can be changed later.
BuildWorker:
* base class for all worker classes
* takes a name and an app instance
* has a idle_since datetime property
* provides a build() method that takes a Blob object and builds
it in whatever way the subclasses implement it
* provides a check_complete(timeout) method that checks whether
the worker has finished building the blob yet or not
LocalBuildWorker:
* worker class for local builds that don't go through SSH
* it uses morphlib.execute.Execute to run morph in a child process
in build()
* at the moment, this class executes "./morph" instead of "morph"
as it assumes the user to run morph from its source tree. obviously,
this will have to be fixed later.
RemoteBuildWorker:
* doesn't implement anything yet, will be used for distributing
work to other machines running morph via SSH
Notes:
* At the moment, there is a degree of undesired redundancy when
building a stratum in a worker, as this will cause the worker to
rebuild all its dependencies. This will have to be fixed as it is
avoidable and wastes a lot of time and processing power.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
This affects pretty much every part of morph, so this might not be fully
working and stable yet.
This commit also introduces the "update-gits" command that can be used
to update all cached repositories from the list of base URLs. The tree
walk when resolving the Treeish objects in Builder.get_cache_id() is a
bit similar to what we do in BuildDependencyGraph, maybe we can merge
that one day.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
The "show-dependencies" command takes a series of build tuples and dumps
the resulting dependency graph (including strata and chunks at the
moment) to the standard output. It also dumps the resulting build order
which is a list of groups. These groups indicate which chunks and strata
can be built in parallel and are not dependent on each other.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Instead, assume the whole build will be run, by the user, under
fakeroot or sudo (the latter for system image builds). This allows
us to run all non-system-image-build tests without root access
at all.
We now always create the cache directory, if missing, even if
we're running as root.
We no longer run ldconfig if ld.so.conf is missing. It is missing
during our tests, but is (now) created by fhs-dirs for real builds.
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Then fix all the places where this broke things because they didn't have
the required permissions.
This is a potential security risk of course, it would be preferable if
the install commands were run in a fakeroot session and only the final
image building was run as root, but it needs to work first
|
| |/ |
|
| |\ |
|
| | | |
|
| |/
|
|
|
| |
This is simpler than giving each separate little thing we need from
there.
|
| |
|
|
|
|
|
| |
We don't create it if running as root, since that would mean a typo
creates a directory in a place that may be awkward.
Anyone running builds as root can go create the directory themselves.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Previously, we've been able to get morphs for strata and systems
directly from files, with uncommitted changes. No more.
|
| |
|
|
|
| |
It now uses a systematic approach for building needed components,
and for decided when something has already been built.
|
