| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
| |
Change-Id: I01b3132c9fb27f1c0277ffe5eb0166e489312e38
|
|
|
|
|
|
| |
It's too much of a mouthful to use a qualified import.
Change-Id: Ide5695e49fd6188b520fecce9d6b076fd7280b8e
|
|
|
|
| |
Change-Id: Ib8281c4030c42b61e298d6cc52f150a02384ed96
|
|
|
|
| |
Change-Id: I895776bfd57ca6a14997c5535355b259d827b3a6
|
|
|
|
| |
Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
|
|
|
|
|
|
|
| |
We commonly create a temporary directory and clean it up after we're
done. This can be encapsulated into a context manager.
Change-Id: Ie50de5c3954141a3e7fd40e4627fb743287ef61f
|
|
|
|
|
|
|
|
| |
We cache based on the name, but to get the name we acutally loaded the
build system, so it's a waste of effort to cache by the name only to
reload it later.
Change-Id: Ia77e1c73d27005cd4cc6a3bb5553b44e8dfc75c1
|
|
|
|
|
|
| |
Sometimes we don't want a separate mount namespace
Change-Id: I9f2645f552c69090612aecd45104a0320cd436c2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For a cluster morph,
- morph: systems/devel-system-x86_64-generic.morph
deploy:
self:
type: ssh-rsync
location: root@127.0.0.1
ATTACK_KITTENS:
The yaml parser will assign the value None
to the element with key 'ATTACK_KITTENS',
sanitize_environment blindly calls str() on each value in the dictionary,
the result is that our extensions receive an 'ATTACK_KITTENS' environment
variable with the string value 'None', this is not useful and confuses anyone
trying to do basic validation in their extension.
This patch makes it so that any None types are replaced with the empty string,
which seems a more sensible value for this case.
Change-Id: I734d33e9304f0b7d4d590e464e350d2e5b4720d3
|
|
|
|
|
|
|
|
| |
System artifacts won't change because of changes in the configuration
extensions. These extensions run at deployment time, therefore, they
are ignored for generating the cache-key.
Change-Id: I64f28605630bda4c27748197aef84809b4b1f86b
|
|
|
|
|
|
|
| |
Switching to the keystoneclient python api
gives us a more reliable means of detecting auth failure.
Change-Id: I5f734bbfe5568c855f524a3448357f7cf46ab254
|
|
|
|
|
|
|
|
|
|
|
| |
--upgrade being passed on to subsystem deployments broke attempting to
perfom an ssh-rsync upgrade with a system that had a sysroot in it.
Therefore it is appropriate to prevent --upgrade being inherited like
this, as it is seldom what is wanted, and if needed, it can be set in
the cluster, or on the command-line separately.
Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Concievably, you may want to force a cluster to behave as an upgrade
regardless of whether --upgrade is set or not.
This would allow two use cases that are not currently possible:
1. To force a cluster to be upgrade-only, which would make our
instructions for the upgrade-devel.morph cluster simpler.
2. To allow a cluster to deploy then immediately upgrade.
One potential use for this would be to create a disk image with
multiple versions on it.
This also works as a work-around for --upgrade implying UPGRADE=yes for
subsystem deployments, but that should be fixed separately.
Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
|
|
|
|
|
|
|
|
| |
Specifying --upgrade with the upgrade command is redundant,
but it's not a contradiction, so we can safely ignore --upgrade
if it happens to be specified with the upgrade command.
Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
|
|
|
|
|
|
|
|
|
|
| |
We use 'git config' format config files outside Git repos, so it's
useful to have a helper class independent of the GitDirectory class.
This allows us to use it in the sysbranchdir.open() function to remove
a hack.
Change-Id: Ifa5e87f404d10666c98b9469079b7925d16becf6
|
|
|
|
|
|
|
|
| |
It's sometimes useful to use --snapshot when debugging the yarn tests,
and `./check --yarns --snapshot` is far easier to type and remember
than the required yarn command.
Change-Id: I3f9c9599dd8418f28e572e484f3515c6ab004572
|
|
|
|
| |
Change-Id: I8e5726cd1c16b7ef3c614e647f3ccfe50bf80ffe
|
|
|
|
| |
Change-Id: I7c993aee22f8438c28566acce01c524d9b315951
|
|
|
|
| |
Change-Id: Iee611887c54f8b8b67504ae2ba0cd78be703e455
|
|
|
|
| |
Change-Id: I54d1a03ba37f7589783f41c02482c0ac1821508d
|
|
|
|
|
|
|
|
|
|
|
| |
If there is no $trove-id/local-config/lorries repository, and similarly if
there is no baserock/local-config/lorries repository on the Trove defined
by 'trove-host', the generate-manifest-csv command will fail with a
NoRemote error from trying to clone a non-existent repository. This commit
fixes this and instead outputs a warning if either of the repositories
cannot be found.
Change-Id: I8edd8484e45d8fc91021a9be34966c13d2beb705
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Old message:
ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not
present in the artifact cache.
Please ensure that mason-system-x86_64-generic-rootfs is built before
deployment, and the artifact-cache-server
(http://cache.baserock.org:8080/) is the correct one.
New message:
ERROR: Deployment failed as
45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs
is not present in the artifact cache.
Please ensure that mason-system-x86_64-generic-rootfs is built before
deployment, and the artifact-cache-server
(http://cache.baserock.org:8080/) is the correct one.
Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
|
|
|
|
| |
Change-Id: I16680439b131e63d30eeff91814a1af643af6246
|
|
|
|
| |
Change-Id: I01a60d4ec187d5fab060f40947d97aa97013f7a7
|
|
|
|
|
|
| |
Logging build output makes the controller logs difficult to read.
Change-Id: I5b81ff9359ada969e964328eb1c2624ab6b9375a
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A fix for the install-files extension has been merged in
commit 93b034f045a2c7443122d7b082ad4460339907d0
This effectively make a new version of definitions if we want to
use this (be able overwrite symlinks), in our definitions
We will use this to create a symlink to /usr/lib/os-release from
/etc/os-release using the install-essential-files extension
Change-Id: I615df922f2e720da411221e61bb838c8c5c63fef
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We found a distbuild controller stuck in a busy loop, with the logs
full of the same error message repeated:
... _flush(): Exception 'IOError: [Errno 32] Broken pipe' from sock.write()
We suspect this came about because the initiator disconnected without
sending an EOF. The initiator was in a VM on a laptop so it seems
possible that the host OS turned off the wireless adaptor without giving
the VM a chance to close its connections gracefully.
The busy loop is because nothing in the SocketBuffer class handles the
SocketError events queued by the _flush() method. Unhandled events are
ignored. So the SocketBuffer stays in 'w' state without ever shifting
any data and never returns. Adding transitions to handle the SocketError
event will fix the problem.
If a socket error happens now in the same scenario, it will be handled
as if the initiator disconnected.
Change-Id: I0f6834f7186a01ca2bc74aef899a4cccbc891e51
|
|
|
|
|
|
|
|
|
| |
Create an InitiatorCommand class that accepts message_type and
status_text parameters to be used by the distbuild-list-jobs,
distbuild-status and distbuild-cancel commands to send request
messages to the distbuild network
Change-Id: Ib686dcd7c370d802b612e9aaa1e3df76f0275fae
|
|
|
|
| |
Change-Id: I6cb572266f4d0be7e08c46b2adef3a701f582353
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out I completely broke Morph's handling of recursive submodules
in commit 0855c357e74e6dd7a.
We didn't notice because nothing in the reference systems actually uses
recursive submodules. However, building the baserock/tlsa/mason2 branch
of definitions.git triggered the bug due to the
delta:python-packages/gitpython chunk, which contains
delta:python-packages/gitdb which in turn contains
delta:python-packages/smmap.
The error is this:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 290, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/build_plugin.py", line 291, in build
self._build(source_pool, filename, component_names=component_names)
File "/src/morph/morphlib/plugins/build_plugin.py", line 316, in _build
bc.build_in_order(component)
File "/src/morph/morphlib/buildcommand.py", line 296, in build_in_order
self.cache_or_build_source(s, build_env)
File "/src/morph/morphlib/buildcommand.py", line 316, in cache_or_build_source
self.build_source(source, build_env)
File "/src/morph/morphlib/buildcommand.py", line 336, in build_source
self.fetch_sources(source)
File "/src/morph/morphlib/buildcommand.py", line 400, in fetch_sources
self.lrc.ensure_submodules(source.repo, source.sha1)
File "/src/morph/morphlib/localrepocache.py", line 308, in ensure_submodules
if (submod.url, submod.commit) not in done:
AttributeError: 'tuple' object has no attribute 'url'
This commit fixes the breakage and the baserock/tlsa/mason2 branch of
definitions.git now builds again.
Change-Id: Id24ac40f4670a60655e84953bcfd84d8f77e1da9
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes an issue in `morph list-artifacts` and probably other commands:
ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts
repo, ref, system_filename)
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system
status_cb=self.app.status)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool
definitions_original_ref=original_ref)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs
definitions_absref, definitions_tree, visit)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children
definitions_queue = collections.deque(system_filenames)
TypeError: 'NoneType' object is not iterable
The issue is that the sourceresolver.create_source_pool() function did
not work correctly if passed a single filename. This was a regression from
commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0.
In order to have a logical function prototype, the create_source_pool()
function now only takes a list of systems, instead of taking either a
single system or a list.
Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
|
|
|
|
|
|
|
| |
This patch fixes an error where we can end up calling int(None) when
we try to send an error response for a malformed message.
Change-Id: Id3ee3298cfb6a5cb32e35fdc5916dab1e4c87a03
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cancelling a distbuild with ctrl+c no longer cancels the build
itself. This commit adds some output explaining what should be
done to cancel the build as well as the local process.
This commit also fixes a bug where the BuildStarted event would
be sent each time a chunk finished building, since it was being
sent in _queue_worker_builds. This is fixed by adding a new
function to be called when the build graph annotation is
complete which sends BuildStarted and then calls
_queue_worker_builds, which no longer sends the BuildStarted
event.
Change-Id: I26ddea2c9080887f449e87004411ddffe4e583b7
|
|
|
|
|
|
|
|
|
| |
Currently jobs may continue running after exec-cancel is sent if
exec-response takes a while to be sent back. This commit makes the
job's state be set to 'failed' when exec-cancel is sent, so that
the wait for exec-response doesn't matter.
Change-Id: I858d9efcba38c81a912cf57aee2bdd8c02cb466b
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit 75ef3e9585091b463b60d2981b3b7283a2ea8eab.
It turns out that the JobQueue may need to handle more than one
build of the same artifact at once, as one may be in the process
of being cancelled when another build of the same artifact is
requested. So they do need an ID separate from the artifact ID.
Change-Id: Ifa0c06987795a4aebdadbd9927de27919377b0a2
|
|
|
|
| |
Change-Id: I27f5319721aa3e996c186f92a3c2296d6df4bedb
|
|
|
|
|
|
|
| |
We no longer serialise whole artifacts, so it doesn't make sense
for things to still refer to serialise-artifact and similar.
Change-Id: Id4d563a07041bbce77f13ac71dc3f7de39df5e23
|
|
|
|
| |
Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous script creates new directory for the chunk it is going
to build without checking whether the directory exists or not.
It will fail back if the directory it attempted to creat exists.
So if build failed, you always need to remove all .inst directories and
let the native-bootstrap script build from the beginning.
This patch improves this, and you can run the native bootstrap script
direct after resumed without loss previous progress.
A condition was added to determine whether previous native bootstrap
script was terminated when it was building current chunk.
As .build directory for certain chunk only exists during building
phase of itself, it was created when started build, and cleaned up
after building finished. If .inst for certain chunk exists,
the .build directory doesn't, the building of the chunk should
be succeed in previous build.
The second go of native-bootstrap will skip all successful chunks
and start where it left off.
Change-Id: I91ae213ecc8c98808efdfd969624291e70f7e0fe
|
|
|
|
| |
Change-Id: I674c39149aad82c07c85d2db3207280b91dfa292
|
|
|
|
| |
Change-Id: I95fbfcb2ed6a8ffdd946d36eacc030b4ae1b9b21
|
|
|
|
|
|
|
| |
Adds distinct message types to give us more flexibility over message
handling now that we have multiple initiator types with different requirements.
Change-Id: Ib2af8736b83d66ef20a8e37591ca68c9441b6497
|
|
|
|
|
|
|
|
| |
This fixes an issue with distbuild-status and distbuild-cancel crashing
due to their appropriate Initiator classes not handling 'build-failed'
messages
Change-Id: Ia35c8e14a30e3a9bdea1e44f7726181db75dfbe5
|
|
|
|
| |
Change-Id: If3f6abdaab6518e77da911bfe1952c8ffe4bda34
|
|
|
|
| |
Change-Id: Ia644ddfaa5138f0ad459099cf26f51b545a9f9ca
|
|
|
|
| |
Change-Id: If74c97ccd81aa4d92ef247d2be59282f9552d4a1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `morph anchor`, `morph build-morphology` and a potential `morph
diff` command would all benefit from having a unified way to parse the
argv for the systems it must operate on, especially in the case of the
potential `morph diff`, which needs to be able to handle being given two
sets of systems.
`morph anchor` may make use of it now by passing the list of systems to
the Source resolver, but `morph build-morphology` would have to iterate
over the systems and graph each independently.
Change-Id: I91ab4764ffca3aa16f144f89f68f37cc21b9f643
|
|
|
|
|
|
|
| |
Remove extra job set line as self._current_job no longer exists
in worker_build_scheduler.py
Change-Id: I8849742587f11f83ebba64f48eaf97fac83e6589
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The existing Source resolution code handles resolution of multiple
systems sufficiently.
It is not appropriate to then take this source pool and attempt to
create a build graph from it though, as the logical structure of the
input of what we want to build, and the logical structure of what we
will produce are conflated in the Source object.
If we do not intend to create a build graph from the Source Pool we
generate, then it is an appropriate data structure that may be used to
analyse changes in the input to a build.
Change-Id: If8e4a726f16f8aca000e59ecbbeb7d926cc08391
|
|
|
|
|
|
|
|
|
|
| |
Passing a single ref is still accepted, but if you have multiple refs
you need to check from the same repository, it is more appropriate to do
it in one call to get_updated_repo, as otherwise there will be
unnecessary output about it not needing to be updated in multiple
places.
Change-Id: I194d7c0e3e84c4917518ba37672b508505c71b8e
|