| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
| |
Change-Id: I8d9330eb5f72a8f3f106aae4387eb8321f6126e5
|
|
|
|
|
|
|
|
|
| |
This code is a rework from changes done by:
- Tiago Gomes <tiago.gomes@codethink.co.uk>
https://storyboard.baserock.org/#!/story/86
Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
|
|
|
|
| |
Change-Id: I603b5172902b5249874ea3bb90de59db5a908185
|
|
|
|
|
|
|
| |
Use an optional parameter to control whether the defaults should be set
instead of a separate function.
Change-Id: Idbbd5a08e8b16d8e01bb9539274092978b64f6f0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's not really any reason you'd want to use the RemoteRepoCache
class except as a workaround for the slow speed of some LocalRepoCache
operations, so I can't see this ruining anyone's day.
The main reason for doing this is so we can simply the sourceresolver
code. One reason that the sourceresolver class is so hopelessly
complicated is that it right now has to use two incompatible interfaces
for Git repo caches.
I've taken the opportunity to detangle the RepoCache class from the
App class. Now all of the configuration for the RepoCache class is
passed into the constructor explicitly. This makes the class usable from
outside Morph:
resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[])
repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver)
Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For a long time the CachedRepo class has basically been a wrapper around
the GitDir class, but with a few extra methods that don't really even
belong there.
It is now a tiny class in the localrepocache module which just keeps
track of a few extra attributes. All other functionality is provided
by the gitdir module.
This commit also removes the `git clone` approach for copying repos out
of the cache. The alternative approach implemented by
git.copy_repository() was slightly faster when I tested, so for now we
should use that everywhere. Longer term we should find out why this is
quicker than `git clone`, and fix Git itself to be fast.
Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
|
|
|
|
|
|
|
|
|
|
| |
This allows us to simplify a couple of places.
I cannot think of a single situation where'd you want to get the cached
copy of a repo, but not update it. Think about it -- the repo might be
*years* behind the upstream remote.
Change-Id: I60340c7fb33e7bfe871ad30c0a9322a7202548e2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For some reason I had a deployment .morph file that specified:
deploy-defaults:
This resulted in the deploy_defaults variable having value None, which
lead to a crash further down:
2016-02-19 12:03:52 Deciding on task order
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 299, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 574, in upgrade
self.deploy(args)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 370, in deploy
env_vars, deployments)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 403, in deploy_cluster
parent_location='')
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 462, in deploy_system
env_vars, deployment_filter, parent_location)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 483, in deploy_system_with_source_pool
system_id, env_vars, deploy_defaults, deploy_params)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 59, in configuration_for_system
deploy_params.items() +
AttributeError: 'NoneType' object has no attribute 'items'
This change makes the code work as expected.
Change-Id: I667f4142667be31797cac7c8994d35a404119cca
|
|
|
|
| |
Change-Id: Ifb2bc2e2084806fe70b6db96828a390d3082288f
|
|
|
|
| |
Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
|
|
|
|
|
|
| |
RIP.
Change-Id: I6aac995415c5d67c60687367697173be52cd2bde
|
|
|
|
|
|
|
| |
Static analysis showed this code is not being used, and we can always
bring it back if necessary through git.
Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When attempting to get a repo with no commit at the ref we want,
we get a 'No such file or directory' error, which is confusing,
this is because we attempt to remove the directory we plan to clone
into, but if there's no commit at the ref we want then we don't
get as far as running git clone, so the directory doesn't exist,
and we fail to remove it.
This commit improves the error message by catching the InvalidRefError
separately.
Change-Id: Ibca20dd995d858713e56ab834daa97a8297cafc8
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Loading .morph files is becoming a bit more complicated, as we need to
deal with the VERSION file, and possibly soon with a DEFAULTS file as
well.
The logic of loading and parsing .morph files is done either in the
sourceresolver module, or the morphloader module. This change means that
all users of the latter module can use the get hold of a
MorphologyLoader instance with VERSION already parsed. If DEFAULTS is
added then it is also simple to parse DEFAULTS.
Change-Id: Ib33756e9dbd078e38f12dd7f776c89584b178959
|
|
|
|
| |
Change-Id: I276a00bb9117aaa2d4fcc7e187b846eedae65591
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pass the Source to the staging area constructor so that we don't need to
pass it as a parameter when we call some StagingArea methods. Also move
the creation of the build and destdir directories to the constructor so
that we can get rid of the chroot_open() and chroot_close() methods.
Also, provide API to retrieve the relative locations for buildir and
destdir.
Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
|
|
|
|
|
|
|
|
| |
We're beginning a transition away from workspaces,
foreach in particular is possibly one of the least used commands,
and is only mentioned in tutorials explicitly marked as 'old'.
Change-Id: I2c95ce689bb5550bf50feb6b55be6c58671c4c4a
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stop moving staging areas of failed builds from the 'staging' directory
to the 'failed' directory. Moving staging areas make it very difficult
to debug build failures on the build essential chunks, as the paths set
on the configure scripts and some environment variables (e.g.
STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area.
This change will also make it easier to create scripts that chroot n
environment similiar to the one where the build failure occurred.
To make it still possible to safely do a build an run `morph gc` in
parallel, we use flock(2) to control access to the staging area
directory.
Also, move the `test_supports_non_isolated_mode` test into a different
class, as it requires a different SetUp() routine (the staging area is
contructed with different parameters).
Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
|
|
|
|
|
|
|
|
| |
This was causing:
UnboundLocalError: local variable 'original_ref' referenced before assignment
Change-Id: I46e4b5d527d9ac5480a3d1fe5e6d631f0b7279dc
|
|
|
|
|
|
|
| |
This makes it more useful and saves people from poking around in the
cache in order to dig up build logs.
Change-Id: I8e062c5c32b01aca0df54e1974ead3c3b3134cc3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This makes `morph get-chunk-details`, `morph get-repo` and `morph
show-build-log` work in any Git checkout of definitions.git. These
are all of the commands outside of the deprecated
branch_and_merge_plugin that take notice of system branches.
The DefinitionsRepo.relative_path_to_chunk() function is changed a
bit. It actually only existed for the `get-repo` command to use, but
it turns out that it did the wrong thing by returning a long path for
the repo (e.g. baserock/baserock/fhs-dirs) instead of a short one
(fhs-dirs). The latter is less typing, and is the behaviour expected
by the `get-repo` yarn tests, so it now does that.
Change-Id: I430b540b3b0f309cf7018e0b8236f0e8a9042d89
|
|
|
|
|
|
|
| |
Previously callers to DefinitionsRepoWithApp.source_pool() could
override the user's value, but that's not actually used anywhere.
Change-Id: I43b53e7b9fc937886c8e6e95947e5e2b6776d085
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously it would print nothing and then exit with success. Better
to tell the user that an error occurred.
Example of the new error:
ERROR: No build log for artifact
6fb39673b8f9a1c9848063f5132aa958ffa75f2be61a8dde68ebb11f5a3c4a5f found
on cache server http://cache.baserock.org:8080/
Change-Id: Icf8ceef60eb497ff90e00391d442a394e3d76f10
|
|
|
|
|
|
|
|
| |
Previously if you tried to view the log on an x86_32 machine for a
chunk that was build on ARM, it'd give a spurious error saying "Are you
trying to cross-build?"
Change-Id: I19c3781e0951ecb4161f06fd8b3a0c8ea5bfef9d
|
|
|
|
|
|
|
|
|
|
| |
This allows you to run `morph deploy` from any Git checkout of
definitions.git.
This also changes `morph help-extensions` to use the DefinitionsRepo
class, as the code overlaps.
Change-Id: I64c48f59c8ec5aebc7169f3b4b4abbb759bd0b9a
|
|
|
|
|
|
|
|
|
| |
This allows using `morph build`, `morph distbuild` and `morph
distbuild-start` from any Git checkout of a definitions.git repo, so
nobody needs to use `morph checkout` or `morph branch` if they don't
want to.
Change-Id: I5fdfae0f8bec1953893e26f0d227e289da11fa84
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After moving deployment extensions into the definitions repository, but
they need access to shared code, so deployment extensions need to import
code which lives in definitions. However morph runs a copy of them in a
temporary directory.
Version 5 will allow deployment extensions to depend only on code which
is also in definitions and the Python standard library, though they may
also include code from elsewhere as described in PYTHONPATH.
Change-Id: I998f01830656a5118bba1e579f649ec98f5f82c6
|
|
|
|
| |
Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
|
|
|
|
|
|
|
|
|
|
|
| |
--upgrade being passed on to subsystem deployments broke attempting to
perfom an ssh-rsync upgrade with a system that had a sysroot in it.
Therefore it is appropriate to prevent --upgrade being inherited like
this, as it is seldom what is wanted, and if needed, it can be set in
the cluster, or on the command-line separately.
Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Concievably, you may want to force a cluster to behave as an upgrade
regardless of whether --upgrade is set or not.
This would allow two use cases that are not currently possible:
1. To force a cluster to be upgrade-only, which would make our
instructions for the upgrade-devel.morph cluster simpler.
2. To allow a cluster to deploy then immediately upgrade.
One potential use for this would be to create a disk image with
multiple versions on it.
This also works as a work-around for --upgrade implying UPGRADE=yes for
subsystem deployments, but that should be fixed separately.
Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
|
|
|
|
|
|
|
|
| |
Specifying --upgrade with the upgrade command is redundant,
but it's not a contradiction, so we can safely ignore --upgrade
if it happens to be specified with the upgrade command.
Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
|
|
|
|
|
|
|
|
|
|
|
| |
If there is no $trove-id/local-config/lorries repository, and similarly if
there is no baserock/local-config/lorries repository on the Trove defined
by 'trove-host', the generate-manifest-csv command will fail with a
NoRemote error from trying to clone a non-existent repository. This commit
fixes this and instead outputs a warning if either of the repositories
cannot be found.
Change-Id: I8edd8484e45d8fc91021a9be34966c13d2beb705
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Old message:
ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not
present in the artifact cache.
Please ensure that mason-system-x86_64-generic-rootfs is built before
deployment, and the artifact-cache-server
(http://cache.baserock.org:8080/) is the correct one.
New message:
ERROR: Deployment failed as
45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs
is not present in the artifact cache.
Please ensure that mason-system-x86_64-generic-rootfs is built before
deployment, and the artifact-cache-server
(http://cache.baserock.org:8080/) is the correct one.
Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
|
|
|
|
|
|
|
|
|
| |
Create an InitiatorCommand class that accepts message_type and
status_text parameters to be used by the distbuild-list-jobs,
distbuild-status and distbuild-cancel commands to send request
messages to the distbuild network
Change-Id: Ib686dcd7c370d802b612e9aaa1e3df76f0275fae
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes an issue in `morph list-artifacts` and probably other commands:
ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts
repo, ref, system_filename)
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system
status_cb=self.app.status)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool
definitions_original_ref=original_ref)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs
definitions_absref, definitions_tree, visit)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children
definitions_queue = collections.deque(system_filenames)
TypeError: 'NoneType' object is not iterable
The issue is that the sourceresolver.create_source_pool() function did
not work correctly if passed a single filename. This was a regression from
commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0.
In order to have a logical function prototype, the create_source_pool()
function now only takes a list of systems, instead of taking either a
single system or a list.
Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
|
|
|
|
|
|
|
| |
We no longer serialise whole artifacts, so it doesn't make sense
for things to still refer to serialise-artifact and similar.
Change-Id: Id4d563a07041bbce77f13ac71dc3f7de39df5e23
|
|
|
|
| |
Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous script creates new directory for the chunk it is going
to build without checking whether the directory exists or not.
It will fail back if the directory it attempted to creat exists.
So if build failed, you always need to remove all .inst directories and
let the native-bootstrap script build from the beginning.
This patch improves this, and you can run the native bootstrap script
direct after resumed without loss previous progress.
A condition was added to determine whether previous native bootstrap
script was terminated when it was building current chunk.
As .build directory for certain chunk only exists during building
phase of itself, it was created when started build, and cleaned up
after building finished. If .inst for certain chunk exists,
the .build directory doesn't, the building of the chunk should
be succeed in previous build.
The second go of native-bootstrap will skip all successful chunks
and start where it left off.
Change-Id: I91ae213ecc8c98808efdfd969624291e70f7e0fe
|
|
|
|
| |
Change-Id: If74c97ccd81aa4d92ef247d2be59282f9552d4a1
|
|
|
|
|
|
|
|
|
|
| |
Previously it would not attempt to make commits it needed locally
available if the commit was available on the remote repo cache.
Now it will do the update if the commit is not available locally, and
will obey --no-git-update.
Change-Id: I80f1e351ce334641e2ef733fa4c9a6ab967f9b67
|
|
|
|
|
|
| |
This fixes an error caused by not enough parameters being given
to the InitiatorBuildCommand constructor in distbuild-morphology.
Change-Id: I133bd2f267fd06cfe88a1cbf4711cc79ad00d209
|
|
|
|
| |
Change-Id: If9d92d7c75b9c4276b69c482c076c6fc1d4ccbbf
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit allows the specification of one or more strata/chunks
in a deployment entry in a cluster morphology to deploy instead of
the full system if --partial is set. These are listed in a
'partial-deploy-components' field in each deployment definition.
The components must be in the system, and this only works for
tarball or sysroot deployments. It SHOULD NOT be used when
deploying production systems, as it has a number of limitations.
Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714
|
|
|
|
| |
Change-Id: I09e9b17ef2e0fb94dbf5a96dca91062d64433add
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a `morph generate-manifest-csv` command which creates
a CSV manifest for a system containing license information,
upstream URL and a guess at the version of each component in a
system.
The existing `morph generate-manifest` is also renamed to
`morph generate-manifest-genivi` to improve the distinction
between the two commands which generate a manifest.
The basis of this commit was done by
Michael Drake <michael.drake@codethink.co.uk>.
Change-Id: I712de11eaa6e1564c5bae5f5f396e5e77484972f
|
|
|
|
| |
Change-Id: I501632c4d7acde06e391df3752275a3afc334ba6
|
|
|
|
|
|
|
|
|
|
| |
If a user had the wrong setting for artifact-cache-server then they
could build a system with 'morph distbuild', then be told it wasn't
built by 'morph deploy'. The logs do not show any info about which
cache server was used, so it looks like a Morph bug. This change should
make such a configuration problem much easier to spot.
Change-Id: I932ca3c5caa46990f9bcd77d75b3862f49e0dfb8
|
|
|
|
|
|
|
|
|
| |
Adds a command to get the status of all recently ran distbuilds
for a given server (e.g. Running, Finished, Failed, Cancelled),
so as to tell if a build running via distbuild-start has finished
or otherwise exited without going through the server's log files
Change-Id: I5ce9fe54ae7b1bd8fe3e0d629f615042be8827ed
|
|
|
|
|
|
|
|
|
|
|
| |
Add command for distbuild-start to build_plugin in morphlib,
and create a boolean parameter to inform the initiator whether
to disconnect the controller and leave the build running remotely.
Add distbuild-cancel command to parse currently-running distbuild
build-request IDs and cancel the one matching the given argument
Change-Id: I458a5767bb768ceb2b4d8876adf1c86075d452bd
|
|
|
|
|
|
|
|
|
| |
If a `morph get-repo` command hits an error or is interrupted while the
'git clone' operation is in progress, the partial clone is left around.
When the user tries to run the command again, they hit an error saying
the directory already exists.
Change-Id: Id4bb57f2136049ede0ea900e7e359255617427ec
|