| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Although in theory a worker should only ever have one job at once, in
practice this assumption doesn't hold, and can cause serious confusion.
The worker (implemented in the JsonRouter class) will actually queue up
exec-request messages and run the oldest one first. I saw a case where,
due to a build not being correctly cancelled, the
WorkerConnection.current_job attribute got out of sync with what the
worker was actually building. This lead to an error when trying to fetch
the built artifacts, as the controller tried to fetch artifacts for
something that wasn't actually built yet, and everything got stuck.
To prevent this from happening, we either need to remove the
exec-request queue in the worker-daemon process, or make the
WorkerConnection class cope with multiple jobs at once. The latter seems
like the more robust approach, so I have done that.
Another bug this fixes is the issue where, if the 'Computing build
graph' (serialise-artifact) step of a build completes on the controller
while one of its WorkerConnection objects is waiting for artifacts to
be fetched by the shared cache from the worker, the build hangs. This
would happen because the WorkerConnection assumed that any
HelperResponse message it saw was the result of its request, so would
send a _JobFinished before caching had actually finished if there was
an unrelated HelperResponse received in the meantime. It now checks
the request ID of the HelperResponse before calling the code that is
now in the new _handle_helper_result_for_job() function.
Change-Id: Ia961f333f9dae77405b58c82c99a56e4c43e1628
|
|
|
|
|
|
|
| |
Rather than generating IDs for each job, identify them by what artifact
is going to be built. Artifact cache IDs need to be unique in any case.
Change-Id: I37a0277931c45a8fb6e37ae7c2a6a942ae732fdd
|
|
|
|
|
|
|
| |
This is a bit more comprehensive than the previous approach of using
public instance attributes, and I find it easier to reason about.
Change-Id: I2942ecf53c95e29893dc0982d38aec689ebfa614
|
|
|
|
|
|
|
| |
The intention is to allow workers to use this class for job tracking, in
addition to the controller.
Change-Id: I355861086764476b383266bab7e850af5e05bc54
|
|
|
|
| |
Change-Id: I77a8a3aab887f5d14a372690502df3fdeba6db10
|
|
|
|
| |
Change-Id: If9d92d7c75b9c4276b69c482c076c6fc1d4ccbbf
|
|
|
|
| |
Change-Id: I1df58c33987597d4aa5a8eb241b4de4ac72fe250
|
|
|
|
|
|
|
| |
"the user gets the repo" does not set exit-morph, so it is not valid to
check whether it exited successfully.
Change-Id: I05e2d5c1919eee6b714269642eb9c39bcf578bbc
|
|
|
|
|
|
|
| |
There's no API to do it in one push yet, but we can send a delete for
all the branches that *did* commit.
Change-Id: I671e9384b84657a3e9034d62818caa0ac0d8de1e
|
|
|
|
| |
Change-Id: I500cb81fd0f133bd9f4e76d46bc0ff8a4f57fe50
|
|
|
|
| |
Change-Id: I5dec13df6c28eeb4e8c83ec41fb4bd119e2eebb1
|
|
|
|
|
|
|
| |
87f8dbefda89bf6cb9e4b88f23a5317b054da0d4 added a method that used
_gitdir, but the patch to change it to gitdir was merged afterwards.
Change-Id: Ibd9bff73a0fe69b3c1c2ff6acd02df6cea4a13de
|
|
|
|
|
|
| |
os.symlink will fail if the origin file/link already exist
Change-Id: I8175c8dce699e55c3e39e35dfd45c0c19b8bd96d
|
|
|
|
| |
Change-Id: Iaab620f3d9ebc037fe024db933b03e8f40ca40a4
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit allows the specification of one or more strata/chunks
in a deployment entry in a cluster morphology to deploy instead of
the full system if --partial is set. These are listed in a
'partial-deploy-components' field in each deployment definition.
The components must be in the system, and this only works for
tarball or sysroot deployments. It SHOULD NOT be used when
deploying production systems, as it has a number of limitations.
Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714
|
|
|
|
| |
Change-Id: I09e9b17ef2e0fb94dbf5a96dca91062d64433add
|
|
|
|
| |
Change-Id: I4e7888cbff2e4708154538f8f0a48aeaa1a8a811
|
|
|
|
|
|
|
|
|
|
| |
Currently, attempting to distbuild a component which is not in
the given system or doesn't exist at all will cause the full
system to be built, rather than an error raised. This is because
the logic which checks that all components were found is
completely nonsensical. This commit makes it actually check the
right thing.
Change-Id: Ide4d7e3fa5f71e433f3a7b7c8c387fe594c92e43
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a `morph generate-manifest-csv` command which creates
a CSV manifest for a system containing license information,
upstream URL and a guess at the version of each component in a
system.
The existing `morph generate-manifest` is also renamed to
`morph generate-manifest-genivi` to improve the distinction
between the two commands which generate a manifest.
The basis of this commit was done by
Michael Drake <michael.drake@codethink.co.uk>.
Change-Id: I712de11eaa6e1564c5bae5f5f396e5e77484972f
|
|
|
|
| |
Change-Id: I501632c4d7acde06e391df3752275a3afc334ba6
|
|
|
|
|
|
|
|
|
|
| |
If a user had the wrong setting for artifact-cache-server then they
could build a system with 'morph distbuild', then be told it wasn't
built by 'morph deploy'. The logs do not show any info about which
cache server was used, so it looks like a Morph bug. This change should
make such a configuration problem much easier to spot.
Change-Id: I932ca3c5caa46990f9bcd77d75b3862f49e0dfb8
|
|
|
|
|
|
|
| |
commit-tree is usually sufficient, but I have a need to be able to
create a commit directly.
Change-Id: I80ba63eb9601aa1190554bb07522465ffb2cb5d9
|
|
|
|
| |
Change-Id: Ife3a809cbf241d8d5a283d7f010ccb5e4d7ea292
|
|
|
|
|
|
| |
It's useful to be able to use this.
Change-Id: Ib32d27ddb637f1b8ed683bdd8ec1db108529a163
|
|
|
|
| |
Change-Id: I048687305054f08b0d8e01daa1366170b3eec323
|
|
|
|
| |
Change-Id: If1b156a309236956d7a35136db35a8fd0ab3ad71
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Once building starts we close the json machine on the initiator,
but we may have received build progress events between processing
our build-started event and closing the json machine, since there
is not a nice way to tell the different types of build progress
apart (they all use BuildProgress) we will ignore all BuildProgress
messages for now.
A possible fix for this is to introduce GraphProgress messages so that
we can report the building of the graph without reporting other types
of BuildProgress ("Waiting for worker" or "Transferring artifact to cache")
that we're not interested in.
Note that we will still report build failures or build success, so if
there's a mistake in the definitions this will be reported before the
detach can occur, similarly if the system is already built this will
be reported before the detach happens.
Change-Id: Ia006ccfba826d2c91f4dea6c028ecdcb5a2b02d6
|
|
|
|
| |
Change-Id: Icfc3d1aa125196e208d7ac35f43f06c5f5a21ba4
|
|
|
|
|
|
|
|
|
| |
Adds a command to get the status of all recently ran distbuilds
for a given server (e.g. Running, Finished, Failed, Cancelled),
so as to tell if a build running via distbuild-start has finished
or otherwise exited without going through the server's log files
Change-Id: I5ce9fe54ae7b1bd8fe3e0d629f615042be8827ed
|
|
|
|
|
|
|
|
|
|
|
| |
Add command for distbuild-start to build_plugin in morphlib,
and create a boolean parameter to inform the initiator whether
to disconnect the controller and leave the build running remotely.
Add distbuild-cancel command to parse currently-running distbuild
build-request IDs and cancel the one matching the given argument
Change-Id: I458a5767bb768ceb2b4d8876adf1c86075d452bd
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently, the distbuild-list-jobs command will fail if morph is
outdated (i.e. protocol version for client and distbuild network
don't match); a protocol_version field has been added to the
list-jobs request message to fix this.
Moved version check outside build-request message to reduce duplication
in new functions.
Generalised the list-request output to reduce duplication for any
further additions that may require a message output.
Change-Id: I28e733cbfe8c89e8c11427df5d40ab275abd313c
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since commit 154a760fb884cee14c2604b8bfbe52b0e7c0d4b1
morph suports a new architecture (armv5)
This is effectivly a change in the definitions format,
as old morph doesnt recognizse this architecture and
will fail if a system with this architecture is added
to definitions
Change-Id: I2adc3a4b103b35c8ca5ec6d4034b797fd10f2485
|
|
|
|
|
|
| |
This fixes morph check.
Change-Id: I5f9c8f5e73c1976b623f9a75f97b5e9aa1887c44
|
|
|
|
|
|
|
|
| |
Do not remove this code for now as its possible that new versions
of morph build old sytems without the install-essential-files
extension
Change-Id: I5e49d0bc6baee73bedf16469d922822e76286bc1
|
|
|
|
|
|
|
| |
This is meant to add essential system files like
/etc/profile, /etc/os-release ...
Change-Id: I3d67b3a452b32205c5d3c7303d128bda80ce75de
|
|
|
|
| |
Change-Id: Ifdaa92c209a4ca488c4447911bef9b1bf7d61438
|
|
|
|
| |
Change-Id: I1c79ce68a7a7534d36a9e83210e18a58e7b648e8
|
|
|
|
|
|
|
|
|
| |
If a `morph get-repo` command hits an error or is interrupted while the
'git clone' operation is in progress, the partial clone is left around.
When the user tries to run the command again, they hit an error saying
the directory already exists.
Change-Id: Id4bb57f2136049ede0ea900e7e359255617427ec
|
|
|
|
|
|
|
|
| |
Also add support to allow building compatible architectures on armv7
machines, as per the rationale in the comment in
_validate_architecture().
Change-Id: Ie0fe4002523f1f92f576bac8b654d4ea6fad5cf2
|
|
|
|
| |
Change-Id: Ie0ac708946c3b1d65d2022638c02d6a2d288ad9d
|
|
|
|
| |
Change-Id: I0c175649bd98067ea69d471ebe6880aac5a25598
|
|
|
|
|
|
|
|
|
|
| |
Commit bd788c7219d8b ("Implement partial builds") caused the
original_ref metadata to be lost in some cases. This meant that the
metadata for built systems would have original_ref set to a SHA1
hex string, instead of the original named ref that the system was
built from.
Change-Id: Id31654ad60b411206ccc7ee70b46f005baa209ef
|
|
|
|
| |
Change-Id: I3c4b4f687703ab665841a3527044d0f669135513
|
|
|
|
| |
Change-Id: Ibf7cf1f81998678f9354d77f52e54344294e89f7
|
|
|
|
|
|
|
| |
Calculate the build graph and find the artifact which is referred to
by the ArtifactReference which was deserialised.
Change-Id: Icbef90932979fc1a18daf0dedac16f4429f56349
|
|
|
|
|
|
|
|
| |
We no longer serialise entire artifacts, so the output of deserialise_artifact
is an ArtifactReference. This commit changes stuff in distbuild to know how to
deal with that rather than an Artifact.
Change-Id: I79b40d041700a85c25980e3bd70cd34dedd2a113
|
|
|
|
|
|
|
|
|
|
| |
The controller no longer needs to know everything about an artifact
as the workers can calculate the build graph themselves quickly.
This reduces the amount of data which needs to be serialised by
serialise-artifact, making the yaml dump quicker.
Change-Id: I6bd0bed14c2efb2f499e9d6f0a97e6188353121a
|
|
|
|
|
|
|
|
|
|
|
| |
Its going to be more likely that a baserock user is more interesed
in the build log than actually the debug output from morph.
And also the intuitive option would be to use -v for this.
--verbose/-v: show build output
--debug/-d: show morph debug output
Change-Id: I1fb99034dc8680a5f168f6306724663aea33ebc5
|
|
|
|
|
|
|
|
|
| |
This is mostly to check that the 'cancel entire subprocess tree' works
as expected. Revert that patch and the test fails.
There are also some tweaks included in this commit.
Change-Id: If297522e6589ebb3a07dac66a39eb243789e53aa
|
|
|
|
|
|
|
| |
This check is intended for local builds, not distbuilds. Much less disk
space is needed to distbuild.
Change-Id: Iaff77805119129cbf6584de418ef97c7f1b82ad8
|