summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Make install-essential-files set the correct env_varbaserock/adamcoldrick/exts-in-subdirsAdam Coldrick2015-06-011-1/+1
|
* Stop assuming that the extensions are in the root of definitionsAdam Coldrick2015-06-011-1/+11
|
* Do not call str() on None types in sanitize_envRichard Ipsum2015-05-291-1/+1
| | | | | | | | | | | | | | | | | | | | | | | For a cluster morph, - morph: systems/devel-system-x86_64-generic.morph deploy: self: type: ssh-rsync location: root@127.0.0.1 ATTACK_KITTENS: The yaml parser will assign the value None to the element with key 'ATTACK_KITTENS', sanitize_environment blindly calls str() on each value in the dictionary, the result is that our extensions receive an 'ATTACK_KITTENS' environment variable with the string value 'None', this is not useful and confuses anyone trying to do basic validation in their extension. This patch makes it so that any None types are replaced with the empty string, which seems a more sensible value for this case. Change-Id: I734d33e9304f0b7d4d590e464e350d2e5b4720d3
* Ignore the configuration extensions when generating the cache-keyPedro Alvarez2015-05-291-0/+3
| | | | | | | | System artifacts won't change because of changes in the configuration extensions. These extensions run at deployment time, therefore, they are ignored for generating the cache-key. Change-Id: I64f28605630bda4c27748197aef84809b4b1f86b
* Use keystoneclient python api in openstack.checkRichard Ipsum2015-05-291-29/+28
| | | | | | | Switching to the keystoneclient python api gives us a more reliable means of detecting auth failure. Change-Id: I5f734bbfe5568c855f524a3448357f7cf46ab254
* deploy: Don't imply --upgrade in subsystemsRichard Maw2015-05-271-2/+5
| | | | | | | | | | | --upgrade being passed on to subsystem deployments broke attempting to perfom an ssh-rsync upgrade with a system that had a sysroot in it. Therefore it is appropriate to prevent --upgrade being inherited like this, as it is seldom what is wanted, and if needed, it can be set in the cluster, or on the command-line separately. Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
* morph deploy: Allow overriding UPGRADE in definition or on command lineRichard Maw2015-05-271-4/+11
| | | | | | | | | | | | | | | | | | | Concievably, you may want to force a cluster to behave as an upgrade regardless of whether --upgrade is set or not. This would allow two use cases that are not currently possible: 1. To force a cluster to be upgrade-only, which would make our instructions for the upgrade-devel.morph cluster simpler. 2. To allow a cluster to deploy then immediately upgrade. One potential use for this would be to create a disk image with multiple versions on it. This also works as a work-around for --upgrade implying UPGRADE=yes for subsystem deployments, but that should be fixed separately. Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
* Ignore --upgrade flag rather than exitingRichard Ipsum2015-05-271-4/+0
| | | | | | | | Specifying --upgrade with the upgrade command is redundant, but it's not a contradiction, so we can safely ignore --upgrade if it happens to be specified with the upgrade command. Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
* Move 'git config' wrapper into its own classSam Thursfield2015-05-223-18/+75
| | | | | | | | | | We use 'git config' format config files outside Git repos, so it's useful to have a helper class independent of the GitDirectory class. This allows us to use it in the sysbranchdir.open() function to remove a hack. Change-Id: Ifa5e87f404d10666c98b9469079b7925d16becf6
* check: Allow yarns to be run with --snapshotAdam Coldrick2015-05-201-1/+9
| | | | | | | | It's sometimes useful to use --snapshot when debugging the yarn tests, and `./check --yarns --snapshot` is far easier to type and remember than the required yarn command. Change-Id: I3f9c9599dd8418f28e572e484f3515c6ab004572
* yarns: Fix typo in cross-bootstrap yarnAdam Coldrick2015-05-202-3/+3
| | | | Change-Id: I8e5726cd1c16b7ef3c614e647f3ccfe50bf80ffe
* yarns: Add a test for `morph list-artifacts`Adam Coldrick2015-05-202-0/+18
| | | | Change-Id: I7c993aee22f8438c28566acce01c524d9b315951
* yarns: Add a test for `morph certify`Adam Coldrick2015-05-202-0/+19
| | | | Change-Id: Iee611887c54f8b8b67504ae2ba0cd78be703e455
* yarns: Add a test for `morph generate-manifest-csv`Adam Coldrick2015-05-202-1/+19
| | | | Change-Id: I54d1a03ba37f7589783f41c02482c0ac1821508d
* generate-manifest-csv: Catch NoRemote error when cloning lorries repoAdam Coldrick2015-05-201-9/+20
| | | | | | | | | | | If there is no $trove-id/local-config/lorries repository, and similarly if there is no baserock/local-config/lorries repository on the Trove defined by 'trove-host', the generate-manifest-csv command will fail with a NoRemote error from trying to clone a non-existent repository. This commit fixes this and instead outputs a warning if either of the repositories cannot be found. Change-Id: I8edd8484e45d8fc91021a9be34966c13d2beb705
* Give exact filename when an artifact is not found in the cache.Sam Thursfield2015-05-191-2/+3
| | | | | | | | | | | | | | | | | | | | | Old message: ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. New message: ERROR: Deployment failed as 45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
* Use protocol to validate incoming requestsRichard Ipsum2015-05-192-23/+49
| | | | Change-Id: I16680439b131e63d30eeff91814a1af643af6246
* Disable WC exec-output messages in log by defaultRichard Ipsum2015-05-181-1/+3
| | | | Change-Id: I01a60d4ec187d5fab060f40947d97aa97013f7a7
* Disable logging of build output by defaultRichard Ipsum2015-05-182-5/+13
| | | | | | Logging build output makes the controller logs difficult to read. Change-Id: I5b81ff9359ada969e964328eb1c2624ab6b9375a
* Bump supported definitions version to version 4Javier Jardón2015-05-181-1/+1
| | | | | | | | | | | | | A fix for the install-files extension has been merged in commit 93b034f045a2c7443122d7b082ad4460339907d0 This effectively make a new version of definitions if we want to use this (be able overwrite symlinks), in our definitions We will use this to create a symlink to /usr/lib/os-release from /etc/os-release using the install-essential-files extension Change-Id: I615df922f2e720da411221e61bb838c8c5c63fef
* distbuild: Handle errors from socketSam Thursfield2015-05-154-3/+25
| | | | | | | | | | | | | | | | | | | | | | | We found a distbuild controller stuck in a busy loop, with the logs full of the same error message repeated: ... _flush(): Exception 'IOError: [Errno 32] Broken pipe' from sock.write() We suspect this came about because the initiator disconnected without sending an EOF. The initiator was in a VM on a laptop so it seems possible that the host OS turned off the wireless adaptor without giving the VM a chance to close its connections gracefully. The busy loop is because nothing in the SocketBuffer class handles the SocketError events queued by the _flush() method. Unhandled events are ignored. So the SocketBuffer stays in 'w' state without ever shifting any data and never returns. Adding transitions to handle the SocketError event will fix the problem. If a socket error happens now in the same scenario, it will be handled as if the initiator disconnected. Change-Id: I0f6834f7186a01ca2bc74aef899a4cccbc891e51
* distbuild: Condense Initiator class to remove unnecessary duplicationLauren Perry2015-05-153-133/+24
| | | | | | | | | Create an InitiatorCommand class that accepts message_type and status_text parameters to be used by the distbuild-list-jobs, distbuild-status and distbuild-cancel commands to send request messages to the distbuild network Change-Id: Ib686dcd7c370d802b612e9aaa1e3df76f0275fae
* Add .gitreview fileRichard Ipsum2015-05-151-0/+5
| | | | Change-Id: I6cb572266f4d0be7e08c46b2adef3a701f582353
* Fix crash in handling of recursive submodulesSam Thursfield2015-05-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It turns out I completely broke Morph's handling of recursive submodules in commit 0855c357e74e6dd7a. We didn't notice because nothing in the reference systems actually uses recursive submodules. However, building the baserock/tlsa/mason2 branch of definitions.git triggered the bug due to the delta:python-packages/gitpython chunk, which contains delta:python-packages/gitdb which in turn contains delta:python-packages/smmap. The error is this: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/build_plugin.py", line 291, in build self._build(source_pool, filename, component_names=component_names) File "/src/morph/morphlib/plugins/build_plugin.py", line 316, in _build bc.build_in_order(component) File "/src/morph/morphlib/buildcommand.py", line 296, in build_in_order self.cache_or_build_source(s, build_env) File "/src/morph/morphlib/buildcommand.py", line 316, in cache_or_build_source self.build_source(source, build_env) File "/src/morph/morphlib/buildcommand.py", line 336, in build_source self.fetch_sources(source) File "/src/morph/morphlib/buildcommand.py", line 400, in fetch_sources self.lrc.ensure_submodules(source.repo, source.sha1) File "/src/morph/morphlib/localrepocache.py", line 308, in ensure_submodules if (submod.url, submod.commit) not in done: AttributeError: 'tuple' object has no attribute 'url' This commit fixes the breakage and the baserock/tlsa/mason2 branch of definitions.git now builds again. Change-Id: Id24ac40f4670a60655e84953bcfd84d8f77e1da9
* Fix prototype of create_source_pool()Sam Thursfield2015-05-1413-23/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
* Protocol check fixRichard Ipsum2015-05-131-2/+2
| | | | | | | This patch fixes an error where we can end up calling int(None) when we try to send an error response for a malformed message. Change-Id: Id3ee3298cfb6a5cb32e35fdc5916dab1e4c87a03
* Explain how to cancel a distbuildAdam Coldrick2015-05-123-16/+33
| | | | | | | | | | | | | | | | Cancelling a distbuild with ctrl+c no longer cancels the build itself. This commit adds some output explaining what should be done to cancel the build as well as the local process. This commit also fixes a bug where the BuildStarted event would be sent each time a chunk finished building, since it was being sent in _queue_worker_builds. This is fixed by adding a new function to be called when the build graph annotation is complete which sends BuildStarted and then calls _queue_worker_builds, which no longer sends the BuildStarted event. Change-Id: I26ddea2c9080887f449e87004411ddffe4e583b7
* distbuild: Set job status to failed when sending exec-cancelAdam Coldrick2015-05-121-0/+8
| | | | | | | | | Currently jobs may continue running after exec-cancel is sent if exec-response takes a while to be sent back. This commit makes the job's state be set to 'failed' when exec-cancel is sent, so that the wait for exec-response doesn't matter. Change-Id: I858d9efcba38c81a912cf57aee2bdd8c02cb466b
* Revert "distbuild: Track worker jobs using artifact basename only"Adam Coldrick2015-05-121-29/+48
| | | | | | | | | | This reverts commit 75ef3e9585091b463b60d2981b3b7283a2ea8eab. It turns out that the JobQueue may need to handle more than one build of the same artifact at once, as one may be in the process of being cancelled when another build of the same artifact is requested. So they do need an ID separate from the artifact ID. Change-Id: Ifa0c06987795a4aebdadbd9927de27919377b0a2
* Remove mention of MorphologyFactory in the unit testsAdam Coldrick2015-05-121-2/+0
| | | | Change-Id: I27f5319721aa3e996c186f92a3c2296d6df4bedb
* Clean up artifact serialisationAdam Coldrick2015-05-126-85/+99
| | | | | | | We no longer serialise whole artifacts, so it doesn't make sense for things to still refer to serialise-artifact and similar. Change-Id: Id4d563a07041bbce77f13ac71dc3f7de39df5e23
* Move duplicate fix_chunk_build_mode function to a common locationAdam Coldrick2015-05-123-52/+33
| | | | Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
* Enable native-bootstrap to continue build after recovered from fault.Kejia Hu (Terry)2015-05-121-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | The previous script creates new directory for the chunk it is going to build without checking whether the directory exists or not. It will fail back if the directory it attempted to creat exists. So if build failed, you always need to remove all .inst directories and let the native-bootstrap script build from the beginning. This patch improves this, and you can run the native bootstrap script direct after resumed without loss previous progress. A condition was added to determine whether previous native bootstrap script was terminated when it was building current chunk. As .build directory for certain chunk only exists during building phase of itself, it was created when started build, and cleaned up after building finished. If .inst for certain chunk exists, the .build directory doesn't, the building of the chunk should be succeed in previous build. The second go of native-bootstrap will skip all successful chunks and start where it left off. Change-Id: I91ae213ecc8c98808efdfd969624291e70f7e0fe
* Remove % from debug statementRichard Ipsum2015-05-121-1/+1
| | | | Change-Id: I674c39149aad82c07c85d2db3207280b91dfa292
* Add a common func for handling build terminationRichard Ipsum2015-05-121-20/+11
| | | | Change-Id: I95fbfcb2ed6a8ffdd946d36eacc030b4ae1b9b21
* Add GraphProgress messagesRichard Ipsum2015-05-125-10/+91
| | | | | | | Adds distinct message types to give us more flexibility over message handling now that we have multiple initiator types with different requirements. Change-Id: Ib2af8736b83d66ef20a8e37591ca68c9441b6497
* distbuild: Fix protocol version checking for distbuild commandsLauren Perry2015-05-111-0/+14
| | | | | | | | This fixes an issue with distbuild-status and distbuild-cancel crashing due to their appropriate Initiator classes not handling 'build-failed' messages Change-Id: Ia35c8e14a30e3a9bdea1e44f7726181db75dfbe5
* yarns: Add yarn for morph diffRichard Maw2015-05-111-0/+22
| | | | Change-Id: If3f6abdaab6518e77da911bfe1952c8ffe4bda34
* yarns: Add the ability to tag chunks and commit updates to definitionsRichard Maw2015-05-112-0/+30
| | | | Change-Id: Ia644ddfaa5138f0ad459099cf26f51b545a9f9ca
* morph: Add morph diff subcommandRichard Maw2015-05-112-0/+148
| | | | Change-Id: If74c97ccd81aa4d92ef247d2be59282f9552d4a1
* morphlib: Add utility module for parsing argv into lists of systemsRichard Maw2015-05-113-0/+143
| | | | | | | | | | | | | | The `morph anchor`, `morph build-morphology` and a potential `morph diff` command would all benefit from having a unified way to parse the argv for the systems it must operate on, especially in the case of the potential `morph diff`, which needs to be able to handle being given two sets of systems. `morph anchor` may make use of it now by passing the list of systems to the Source resolver, but `morph build-morphology` would have to iterate over the systems and graph each independently. Change-Id: I91ab4764ffca3aa16f144f89f68f37cc21b9f643
* distbuild: Builds currently break due to job being set twiceLauren Perry2015-05-111-1/+0
| | | | | | | Remove extra job set line as self._current_job no longer exists in worker_build_scheduler.py Change-Id: I8849742587f11f83ebba64f48eaf97fac83e6589
* SourceResolver: Allow the resolution of multiple systemsRichard Maw2015-05-112-5/+10
| | | | | | | | | | | | | | | | The existing Source resolution code handles resolution of multiple systems sufficiently. It is not appropriate to then take this source pool and attempt to create a build graph from it though, as the logical structure of the input of what we want to build, and the logical structure of what we will produce are conflated in the Source object. If we do not intend to create a build graph from the Source Pool we generate, then it is an appropriate data structure that may be used to analyse changes in the input to a build. Change-Id: If8e4a726f16f8aca000e59ecbbeb7d926cc08391
* LRC: Make get_updated_repo handle multiple refsRichard Maw2015-05-111-16/+27
| | | | | | | | | | Passing a single ref is still accepted, but if you have multiple refs you need to check from the same repository, it is more appropriate to do it in one call to get_updated_repo, as otherwise there will be unnecessary output about it not needing to be updated in multiple places. Change-Id: I194d7c0e3e84c4917518ba37672b508505c71b8e
* MorphologyLoader: Set filename attribute at parse timeRichard Maw2015-05-111-2/+3
| | | | Change-Id: I0e0b8d352eb4ef1ab6c50e0ba0162263d9bac09d
* morph anchor: Handle updating refsRichard Maw2015-05-111-5/+6
| | | | | | | | | | Previously it would not attempt to make commits it needed locally available if the commit was available on the remote repo cache. Now it will do the update if the commit is not available locally, and will obey --no-git-update. Change-Id: I80f1e351ce334641e2ef733fa4c9a6ab967f9b67
* morphlib.util: add word_join_listRichard Maw2015-05-111-0/+10
| | | | | | | This is useful for representing lists of items in status or exception messages. Change-Id: I530eecdc311ac77fca9922dab063f550ea810c06
* yarns: Fix incorrect chunk name in test suiteAdam Coldrick2015-05-111-1/+1
| | | | Change-Id: I60e808d7f42890dac2e1470a994e1a31a92401e7
* Fix mistake in sysroot.writeSam Thursfield2015-05-081-1/+1
| | | | | | The * should not be in quotes. Change-Id: Ieebdc7532ba1bff5ba9742f72440ed00b0c0de2a
* Raise an error if a stratum build-depends on itselfAdam Coldrick2015-05-082-0/+25
| | | | | | | | | | | | If a stratum build-depends on itself, the build graph calculation gets stuck in an infinite loop as it adds the same stratum to the queue of morphologies to inspect over and over again. This commit causes MorphologyLoader.validate_stratum to raise an error if a stratum contains itself in it's build-depends, as depending on itself makes no sense and will cause the above problem. Change-Id: I76df5b7d63d010ae3b17f72bfa39b273e74279dd
View Lorry Controller Status