| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
|
|
|
|
| |
System artifacts won't change because of changes in the configuration
extensions. These extensions run at deployment time, therefore, they
are ignored for generating the cache-key.
Change-Id: I64f28605630bda4c27748197aef84809b4b1f86b
|
|
|
|
|
|
|
| |
Switching to the keystoneclient python api
gives us a more reliable means of detecting auth failure.
Change-Id: I5f734bbfe5568c855f524a3448357f7cf46ab254
|
|
|
|
|
|
|
|
|
|
|
| |
--upgrade being passed on to subsystem deployments broke attempting to
perfom an ssh-rsync upgrade with a system that had a sysroot in it.
Therefore it is appropriate to prevent --upgrade being inherited like
this, as it is seldom what is wanted, and if needed, it can be set in
the cluster, or on the command-line separately.
Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Concievably, you may want to force a cluster to behave as an upgrade
regardless of whether --upgrade is set or not.
This would allow two use cases that are not currently possible:
1. To force a cluster to be upgrade-only, which would make our
instructions for the upgrade-devel.morph cluster simpler.
2. To allow a cluster to deploy then immediately upgrade.
One potential use for this would be to create a disk image with
multiple versions on it.
This also works as a work-around for --upgrade implying UPGRADE=yes for
subsystem deployments, but that should be fixed separately.
Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
|
|
|
|
|
|
|
|
| |
Specifying --upgrade with the upgrade command is redundant,
but it's not a contradiction, so we can safely ignore --upgrade
if it happens to be specified with the upgrade command.
Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
|
|
|
|
|
|
|
|
|
|
| |
We use 'git config' format config files outside Git repos, so it's
useful to have a helper class independent of the GitDirectory class.
This allows us to use it in the sysbranchdir.open() function to remove
a hack.
Change-Id: Ifa5e87f404d10666c98b9469079b7925d16becf6
|
|
|
|
|
|
|
|
| |
It's sometimes useful to use --snapshot when debugging the yarn tests,
and `./check --yarns --snapshot` is far easier to type and remember
than the required yarn command.
Change-Id: I3f9c9599dd8418f28e572e484f3515c6ab004572
|
|
|
|
| |
Change-Id: I8e5726cd1c16b7ef3c614e647f3ccfe50bf80ffe
|
|
|
|
| |
Change-Id: I7c993aee22f8438c28566acce01c524d9b315951
|
|
|
|
| |
Change-Id: Iee611887c54f8b8b67504ae2ba0cd78be703e455
|
|
|
|
| |
Change-Id: I54d1a03ba37f7589783f41c02482c0ac1821508d
|
|
|
|
|
|
|
|
|
|
|
| |
If there is no $trove-id/local-config/lorries repository, and similarly if
there is no baserock/local-config/lorries repository on the Trove defined
by 'trove-host', the generate-manifest-csv command will fail with a
NoRemote error from trying to clone a non-existent repository. This commit
fixes this and instead outputs a warning if either of the repositories
cannot be found.
Change-Id: I8edd8484e45d8fc91021a9be34966c13d2beb705
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Old message:
ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not
present in the artifact cache.
Please ensure that mason-system-x86_64-generic-rootfs is built before
deployment, and the artifact-cache-server
(http://cache.baserock.org:8080/) is the correct one.
New message:
ERROR: Deployment failed as
45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs
is not present in the artifact cache.
Please ensure that mason-system-x86_64-generic-rootfs is built before
deployment, and the artifact-cache-server
(http://cache.baserock.org:8080/) is the correct one.
Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
|
|
|
|
| |
Change-Id: I16680439b131e63d30eeff91814a1af643af6246
|
|
|
|
| |
Change-Id: I01a60d4ec187d5fab060f40947d97aa97013f7a7
|
|
|
|
|
|
| |
Logging build output makes the controller logs difficult to read.
Change-Id: I5b81ff9359ada969e964328eb1c2624ab6b9375a
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A fix for the install-files extension has been merged in
commit 93b034f045a2c7443122d7b082ad4460339907d0
This effectively make a new version of definitions if we want to
use this (be able overwrite symlinks), in our definitions
We will use this to create a symlink to /usr/lib/os-release from
/etc/os-release using the install-essential-files extension
Change-Id: I615df922f2e720da411221e61bb838c8c5c63fef
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We found a distbuild controller stuck in a busy loop, with the logs
full of the same error message repeated:
... _flush(): Exception 'IOError: [Errno 32] Broken pipe' from sock.write()
We suspect this came about because the initiator disconnected without
sending an EOF. The initiator was in a VM on a laptop so it seems
possible that the host OS turned off the wireless adaptor without giving
the VM a chance to close its connections gracefully.
The busy loop is because nothing in the SocketBuffer class handles the
SocketError events queued by the _flush() method. Unhandled events are
ignored. So the SocketBuffer stays in 'w' state without ever shifting
any data and never returns. Adding transitions to handle the SocketError
event will fix the problem.
If a socket error happens now in the same scenario, it will be handled
as if the initiator disconnected.
Change-Id: I0f6834f7186a01ca2bc74aef899a4cccbc891e51
|
|
|
|
|
|
|
|
|
| |
Create an InitiatorCommand class that accepts message_type and
status_text parameters to be used by the distbuild-list-jobs,
distbuild-status and distbuild-cancel commands to send request
messages to the distbuild network
Change-Id: Ib686dcd7c370d802b612e9aaa1e3df76f0275fae
|
|
|
|
| |
Change-Id: I6cb572266f4d0be7e08c46b2adef3a701f582353
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out I completely broke Morph's handling of recursive submodules
in commit 0855c357e74e6dd7a.
We didn't notice because nothing in the reference systems actually uses
recursive submodules. However, building the baserock/tlsa/mason2 branch
of definitions.git triggered the bug due to the
delta:python-packages/gitpython chunk, which contains
delta:python-packages/gitdb which in turn contains
delta:python-packages/smmap.
The error is this:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 290, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/build_plugin.py", line 291, in build
self._build(source_pool, filename, component_names=component_names)
File "/src/morph/morphlib/plugins/build_plugin.py", line 316, in _build
bc.build_in_order(component)
File "/src/morph/morphlib/buildcommand.py", line 296, in build_in_order
self.cache_or_build_source(s, build_env)
File "/src/morph/morphlib/buildcommand.py", line 316, in cache_or_build_source
self.build_source(source, build_env)
File "/src/morph/morphlib/buildcommand.py", line 336, in build_source
self.fetch_sources(source)
File "/src/morph/morphlib/buildcommand.py", line 400, in fetch_sources
self.lrc.ensure_submodules(source.repo, source.sha1)
File "/src/morph/morphlib/localrepocache.py", line 308, in ensure_submodules
if (submod.url, submod.commit) not in done:
AttributeError: 'tuple' object has no attribute 'url'
This commit fixes the breakage and the baserock/tlsa/mason2 branch of
definitions.git now builds again.
Change-Id: Id24ac40f4670a60655e84953bcfd84d8f77e1da9
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes an issue in `morph list-artifacts` and probably other commands:
ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts
repo, ref, system_filename)
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system
status_cb=self.app.status)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool
definitions_original_ref=original_ref)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs
definitions_absref, definitions_tree, visit)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children
definitions_queue = collections.deque(system_filenames)
TypeError: 'NoneType' object is not iterable
The issue is that the sourceresolver.create_source_pool() function did
not work correctly if passed a single filename. This was a regression from
commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0.
In order to have a logical function prototype, the create_source_pool()
function now only takes a list of systems, instead of taking either a
single system or a list.
Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
|
|
|
|
|
|
|
| |
This patch fixes an error where we can end up calling int(None) when
we try to send an error response for a malformed message.
Change-Id: Id3ee3298cfb6a5cb32e35fdc5916dab1e4c87a03
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cancelling a distbuild with ctrl+c no longer cancels the build
itself. This commit adds some output explaining what should be
done to cancel the build as well as the local process.
This commit also fixes a bug where the BuildStarted event would
be sent each time a chunk finished building, since it was being
sent in _queue_worker_builds. This is fixed by adding a new
function to be called when the build graph annotation is
complete which sends BuildStarted and then calls
_queue_worker_builds, which no longer sends the BuildStarted
event.
Change-Id: I26ddea2c9080887f449e87004411ddffe4e583b7
|
|
|
|
|
|
|
|
|
| |
Currently jobs may continue running after exec-cancel is sent if
exec-response takes a while to be sent back. This commit makes the
job's state be set to 'failed' when exec-cancel is sent, so that
the wait for exec-response doesn't matter.
Change-Id: I858d9efcba38c81a912cf57aee2bdd8c02cb466b
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit 75ef3e9585091b463b60d2981b3b7283a2ea8eab.
It turns out that the JobQueue may need to handle more than one
build of the same artifact at once, as one may be in the process
of being cancelled when another build of the same artifact is
requested. So they do need an ID separate from the artifact ID.
Change-Id: Ifa0c06987795a4aebdadbd9927de27919377b0a2
|
|
|
|
| |
Change-Id: I27f5319721aa3e996c186f92a3c2296d6df4bedb
|
|
|
|
|
|
|
| |
We no longer serialise whole artifacts, so it doesn't make sense
for things to still refer to serialise-artifact and similar.
Change-Id: Id4d563a07041bbce77f13ac71dc3f7de39df5e23
|
|
|
|
| |
Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous script creates new directory for the chunk it is going
to build without checking whether the directory exists or not.
It will fail back if the directory it attempted to creat exists.
So if build failed, you always need to remove all .inst directories and
let the native-bootstrap script build from the beginning.
This patch improves this, and you can run the native bootstrap script
direct after resumed without loss previous progress.
A condition was added to determine whether previous native bootstrap
script was terminated when it was building current chunk.
As .build directory for certain chunk only exists during building
phase of itself, it was created when started build, and cleaned up
after building finished. If .inst for certain chunk exists,
the .build directory doesn't, the building of the chunk should
be succeed in previous build.
The second go of native-bootstrap will skip all successful chunks
and start where it left off.
Change-Id: I91ae213ecc8c98808efdfd969624291e70f7e0fe
|
|
|
|
| |
Change-Id: I674c39149aad82c07c85d2db3207280b91dfa292
|
|
|
|
| |
Change-Id: I95fbfcb2ed6a8ffdd946d36eacc030b4ae1b9b21
|
|
|
|
|
|
|
| |
Adds distinct message types to give us more flexibility over message
handling now that we have multiple initiator types with different requirements.
Change-Id: Ib2af8736b83d66ef20a8e37591ca68c9441b6497
|
|
|
|
|
|
|
|
| |
This fixes an issue with distbuild-status and distbuild-cancel crashing
due to their appropriate Initiator classes not handling 'build-failed'
messages
Change-Id: Ia35c8e14a30e3a9bdea1e44f7726181db75dfbe5
|
|
|
|
| |
Change-Id: If3f6abdaab6518e77da911bfe1952c8ffe4bda34
|
|
|
|
| |
Change-Id: Ia644ddfaa5138f0ad459099cf26f51b545a9f9ca
|
|
|
|
| |
Change-Id: If74c97ccd81aa4d92ef247d2be59282f9552d4a1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `morph anchor`, `morph build-morphology` and a potential `morph
diff` command would all benefit from having a unified way to parse the
argv for the systems it must operate on, especially in the case of the
potential `morph diff`, which needs to be able to handle being given two
sets of systems.
`morph anchor` may make use of it now by passing the list of systems to
the Source resolver, but `morph build-morphology` would have to iterate
over the systems and graph each independently.
Change-Id: I91ab4764ffca3aa16f144f89f68f37cc21b9f643
|
|
|
|
|
|
|
| |
Remove extra job set line as self._current_job no longer exists
in worker_build_scheduler.py
Change-Id: I8849742587f11f83ebba64f48eaf97fac83e6589
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The existing Source resolution code handles resolution of multiple
systems sufficiently.
It is not appropriate to then take this source pool and attempt to
create a build graph from it though, as the logical structure of the
input of what we want to build, and the logical structure of what we
will produce are conflated in the Source object.
If we do not intend to create a build graph from the Source Pool we
generate, then it is an appropriate data structure that may be used to
analyse changes in the input to a build.
Change-Id: If8e4a726f16f8aca000e59ecbbeb7d926cc08391
|
|
|
|
|
|
|
|
|
|
| |
Passing a single ref is still accepted, but if you have multiple refs
you need to check from the same repository, it is more appropriate to do
it in one call to get_updated_repo, as otherwise there will be
unnecessary output about it not needing to be updated in multiple
places.
Change-Id: I194d7c0e3e84c4917518ba37672b508505c71b8e
|
|
|
|
| |
Change-Id: I0e0b8d352eb4ef1ab6c50e0ba0162263d9bac09d
|
|
|
|
|
|
|
|
|
|
| |
Previously it would not attempt to make commits it needed locally
available if the commit was available on the remote repo cache.
Now it will do the update if the commit is not available locally, and
will obey --no-git-update.
Change-Id: I80f1e351ce334641e2ef733fa4c9a6ab967f9b67
|
|
|
|
|
|
|
| |
This is useful for representing lists of items in status or exception
messages.
Change-Id: I530eecdc311ac77fca9922dab063f550ea810c06
|
|
|
|
| |
Change-Id: I60e808d7f42890dac2e1470a994e1a31a92401e7
|
|
|
|
|
|
| |
The * should not be in quotes.
Change-Id: Ieebdc7532ba1bff5ba9742f72440ed00b0c0de2a
|
|
|
|
|
|
|
|
|
|
|
|
| |
If a stratum build-depends on itself, the build graph calculation
gets stuck in an infinite loop as it adds the same stratum to the
queue of morphologies to inspect over and over again.
This commit causes MorphologyLoader.validate_stratum to raise an
error if a stratum contains itself in it's build-depends, as
depending on itself makes no sense and will cause the above problem.
Change-Id: I76df5b7d63d010ae3b17f72bfa39b273e74279dd
|
|
|
|
|
|
| |
This fixes an error caused by not enough parameters being given
to the InitiatorBuildCommand constructor in distbuild-morphology.
Change-Id: I133bd2f267fd06cfe88a1cbf4711cc79ad00d209
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If the initiator sends an invalid build-request message, it will now
exit with the following sort of error:
ERROR: Failed to build baserock:baserock/definitions
f2d78e9b7221bca65cba53af3f3b50d50d90628f
systems/build-system-x86_64.morph: Invalid build-request message. Check
you are using a supported version of Morph. This distbuild network uses
protocol version 2.
Previously, the controller would log an error to its log file, but it
would not send any response to the initiator so the initiator would
hang forever.
Behaviour is the same as before for the case where the initiator sends a
build-request message with the wrong protocol version: the initiator
will exit with an error message.
Change-Id: I94fdee02bc701d4a679a0261b3c46dbdf14cfcaf
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit 807e6a90876c5469d242 changed the behaviour of sysroot.write to
avoid deleting the contents of the sysroot. This was done so if you
accidentally set 'sysroot=/' it wouldn't delete your whole system.
It turns out that SDK deployments like
clusters/sdk-example-cluster.morph depended on the contents of the
directory being deleted. The system
armv7lhf-cross-toolchain-system-x86_64.morph has a bunch of files
installed by the cross-toolchain in
/usr/armv7lhf-baserock-linux-gnueabi/sys-root. Previously sysroot.write
would delete these, but since commit 807e6a90876c5469d242 it would
fail with several errors like:
mv: can't rename '/src/tmp/deployments/usr/armv7l.../sys-root/sbin'
If we use 'cp -a' instead of 'mv' then it is slower to deploy, but there
are no errors.
I am still unsure why files from the cross-toolchain system are
installed and then deleted. Although this patch fixes the immediate
issue, I don't know if it's the right thing to do. It seems better to
not install those files in the first place, if we do not need them.
This commit also removes the check for the sysroot target location being
empty. This doesn't work, because it runs /before/ the system being
deployed is unpacked.
Change-Id: I10671c2f3b2060cfb36f880675b83351c6cdd807
|