| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This requires build-single to take a dependency context tuple when
building chunks of a stratum. This context tuple is the surrounding
stratum which is used to construct the dependency graph in the worker
and then do a breadth-first search to collect all dependencies that need
to be added to the staging area.
Implementing this required a few hash/eq changes in Blob, Morphology
and Treeish as well as a few adjustments in the corresponding unit
tests.
|
| | |
|
| |
|
|
|
| |
...while at the same time making sure that all dependencies are marked
for staging prior to building the blob itself.
|
| | |
|
| |
|
|
| |
This allows worker machines to be configured via ~/.bash_profile.
|
| | |
|
| | |
|
| |
|
|
|
|
| |
This functionality would ideally be implemented in cliapp.Settings, as
it needs access to internals of that class. However, I'll drop it here
for now.
|
| |
|
|
|
|
| |
Also, distinguish between running "sudo" and "fakeroot" depending on
whether or not a remote worker builds a system image. For some reason,
sudo in SSH does not seem to work here, so we need to figure that out.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
All BuildWorker subclasses are likely to make use of the multiprocessing
library anyway, so most of the functionality can be shared.
All workers now have an ident (e.g. "user@hostname" or "local") which
is used by the BuildController to generate enumerated names for them
(e.g. "user@hostname-1", "user@hostname-2" or "local-1" and "local-2"),
which makes it easier to identify who does what.
The RemoteBuildWorker now runs "ssh HOSTNAME fakeroot morph" for
building stuff remotely.
|
|
|
This commit introduces four new classes:
BuildController:
* takes an app instance and a tempdir
* allows to add BuildWorker objects
* provides a build() method that takes a set of blobs and
a build order that is then built by assigning work to the
build workers as needed
* the build() method takes care of polling the workers for
their state, moving them between busy and idle states
reliably, collect and print their output in a non-confusing
order, and makes sure to wait for all workers to finish
before processing the next group in the build order.
* at this point, when waiting for one or more workers to
become idle to assign them another blob to build, the
controller always picks the worker that has been idling
for the longest period of time. this can be changed later.
BuildWorker:
* base class for all worker classes
* takes a name and an app instance
* has a idle_since datetime property
* provides a build() method that takes a Blob object and builds
it in whatever way the subclasses implement it
* provides a check_complete(timeout) method that checks whether
the worker has finished building the blob yet or not
LocalBuildWorker:
* worker class for local builds that don't go through SSH
* it uses morphlib.execute.Execute to run morph in a child process
in build()
* at the moment, this class executes "./morph" instead of "morph"
as it assumes the user to run morph from its source tree. obviously,
this will have to be fixed later.
RemoteBuildWorker:
* doesn't implement anything yet, will be used for distributing
work to other machines running morph via SSH
Notes:
* At the moment, there is a degree of undesired redundancy when
building a stratum in a worker, as this will cause the worker to
rebuild all its dependencies. This will have to be fixed as it is
avoidable and wastes a lot of time and processing power.
|
