summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
* Merge remote-tracking branch 'origin/baserock/ps/log-chunk-build-times'Richard Maw2014-10-091-0/+8
|\ | | | | | | | | Reviewed-by: Pedro Alvarez Reviewed-by: Richard Maw
| * Log elapsed time for each chunk buildbaserock/ps/log-chunk-build-timesPaul Sherwood2014-09-271-0/+7
| |
* | Update artifact resolver and ckc testsRichard Ipsum2014-10-092-9/+9
| | | | | | | | | | | | The original resolve_artifacts method is essentially unchanged, it's just private now. This means that aside from this name change the tests remain the same.
* | Tidy up artifact resolvingRichard Ipsum2014-10-092-34/+24
| | | | | | | | | | | | | | | | This patch started off as an attempt to address the comment in find_root_artifacts, though this patch doesn't impose an ordering on the list of artifacts as the comment suggested. The artifact resolver now returns a list of root artifacts which may make it easier to add multi-system builds to morph.
* | Merge branch 'sam/handle-missing-version-label'Sam Thursfield2014-10-082-2/+7
|\ \ | | | | | | | | | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk>
| * | deploy: Make ssh-rsync upgrade extension handle unset VERSION_LABELSam Thursfield2014-10-082-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It now gives an error message. Previously it would fail with a backtrace like this: 2014-10-08 09:51:37 [systems/genivi-baseline-system-armv7lhf-jetson.morph][self]Removing temporary mounts Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/exts/ssh-rsync.write", line 54, in process_args self.upgrade_remote_system(location, temp_root) File "/src/morph/morphlib/exts/ssh-rsync.write", line 107, in upgrade_remote_system location, ['btrfs', 'subvolume', 'delete', orig_dir]) UnboundLocalError: local variable 'orig_dir' referenced before assignment
* | | Merge branch 'baserock/richardmaw/fix-distbuild-v3'Richard Maw2014-10-085-107/+38
|\ \ \ | |/ / |/| | | | | | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Richard Ipsum Reviewed-by: Pedro Alvarez
| * | Allow distbuilds to choose where to put logsRichard Maw2014-10-081-0/+5
| | |
| * | Fix issues with distbuild caused by moving to building per-sourceRichard Maw2014-10-082-3/+5
| | |
| * | Allow ephemeral ports for distbuild servicesRichard Maw2014-10-021-1/+1
| | |
| * | distbuild: allow daemons to bind to ephemeral portsRichard Maw2014-10-011-2/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | You can bind to an ephemeral port by passing 0 as the port number. To work out which port you actually got, you need to call getsockname(). To facilitate being able to spawn multiple copies of the daemons for testing environments, you can pass a -file option, which will make the daemon write which port it actually bound to. If this path is a fifo, reading from it in the spawner process will allow synchronisation of only spawning services that require that port to be ready after it is.
| * | Allow default split rules to be overridden in-codeRichard Maw2014-10-011-4/+4
| | | | | | | | | | | | | | | | | | This is needed for distbuild to deserialise based on the split rules on the node that did the graph calculation, rather than the node that does the building.
| * | Remove overlap detection logicRichard Maw2014-10-012-96/+0
| | | | | | | | | | | | | | | | | | | | | I've rarely needed to use it, and on those rare occasions, it would have been easy enough to calculate it. Let's get rid of this step, and save everyone some time in future.
| * | Remove run-in-artifact subcommandRichard Maw2014-10-011-1/+1
| | | | | | | | | | | | It's easy enough to deploy the image.
* | | Merge branch 'sam/gbo-read-access'Sam Thursfield2014-10-071-2/+13
|\ \ \ | | | | | | | | | | | | | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk>
| * | | Never require SSH access to the Trove where baserock: content is hostedSam Thursfield2014-10-071-2/+13
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously if the user included 'baserock' in their list of trove-ids, the keyed URL expansions in the repo-alias field would change such that reading from git.baserock.org required SSH access. This isn't good and in particular breaks Mason, which by default isn't set up with a Gitano account on the Trove that it reads from, and therefore doesn't have SSH access to that Trove. With this change, the trove-id field will not cause the default 'baserock:' and 'upstream:' prefixes to be overridden, so setting 'trove-id = baserock' in morph.conf will not Morph's behaviour to change.
* | | xfer-hole: Fix bug in copy_slice_from_file.Lars Wirzenius2014-10-021-2/+8
| | | | | | | | | | | | | | | os.read is limited to an int in size. copy_slice_from_file was trying to os.read more than that causing an OverflowError.
* | | ssh-rsync: gett UUID of the disk before writing fstabPedro Alvarez2014-10-011-2/+5
| |/ |/| | | | | | | | | | | | | | | With this patch, the fstab of the system to be deployed as an upgrade will be conifgured using the UUID of the disk. Now when doing an upgrade is not needed to specify the ROOT_DEVICE.
* | Fix behaviour of `morph --version`Pedro Alvarez2014-09-301-2/+4
|/ | | | | | | | | | | | | | | | Currently, if morph is installed in the system, `morph --version` prints the sha1 of the version installed. $ morph --version e8adedb8f3f27d9212caf277b8e8f7c6792a20c2 If you run morph from git, the output will be something similar to the following. $ morph --version baserock-14.26-124-g7b73af4 This patch changes the behaviour of the latter to match the former.
* Safely interpolate in environment variable reportingRichard Maw2014-09-241-2/+2
| | | | | | | | | The msg parameter to status is a format string. If we pass a string directly to it, then we have to be careful to escape any formatting characters. However, we can just do the interpolation directly in the status call instead, which is less code.
* Add no cover to morphology hash functionRichard Ipsum2014-09-231-1/+1
|
* Merge branch 'baserock/richardmaw-os/tidy-build-logic-v7'Richard Maw2014-09-2227-938/+686
|\ | | | | | | | | | | Reviewed-by: Lars Wirzenius (+2 to misc fixups) Reviewed-by: Sam Thursfield (+1 to per-source building) Reviewed-by: Paul Sherwood (+1 to per-source building)
| * Remove Artifact compatibility methodsRichard Maw2014-09-193-40/+7
| |
| * Remove get_dependency_prefix_setRichard Maw2014-09-191-14/+0
| | | | | | | | | | | | | | | | This was used by artifacts to tell what they should put in their path, based on what prefixes were used earlier. This implementation didn't handle recursive deps, and it was trivial to open-code it at the call-site, so it is no longer useful.
| * Fix show-dependencies plugin to use source deps.Richard Maw2014-09-191-2/+2
| |
| * FIXUP: Fix unit tests after changing to per-source buildsRichard Maw2014-09-195-90/+28
| |
| * Build per-source rather than per-artifactRichard Maw2014-09-196-297/+314
| |
| * FIXUP: Add compatibility methods to Artifact for fields that moved to SourceRichard Maw2014-09-192-2/+36
| | | | | | | | | | This means we can avoid having to rewrite everything immediately after the fields moved.
| * Move dependencies and cache keys to SourcesRichard Maw2014-09-192-22/+23
| |
| * FIXUP: Fix up unit tests from change allowing multiple sourcesRichard Maw2014-09-199-276/+114
| | | | | | | | | | This is logically part of the previous patch, but has been split out to ease reviewing.
| * Create multiple sources per stratum morphologyRichard Maw2014-09-196-66/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Building per-artifact results in undesirable behaviour, as multiple artifacts are produced for every chunk build. It therefore makes more sense to build per-source. This implies that actually, the model of one source per morphology is wrong and we should move the dependencies into the source. Unlike chunks however, where every chunk artifact has the same dependencies, stratum artifacts can have different dependencies. So before we can move the dependencies into the Source, we need to have as many Sources as Stratum Artifacts.
| * Make morphologies hashableRichard Maw2014-09-191-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | I need to be able to deduplicate a list of morphologies. Putting it in a set is the easiest way, but it needs to be hashable. It's not included in dicts by default, since they're stored by reference, and you can change them while they're in the dict, so the hash value can change. I don't need to deduplicate morphologies by their contents, just by reference though, so using `id` as the hash function is sufficient.
| * Turn Morphology.needs_artifact_metadata_cached into a propertyRichard Maw2014-09-193-40/+4
| | | | | | | | | | This means we can remove some complication from the MorphologyFactory class.
| * Remove Morphology.builds_artifactsRichard Maw2014-09-195-44/+0
| | | | | | | | | | This was used before the Artifact splitting code landed to determine which artifacts should be produced.
| * Tidy morphlib.artifact.Artifact.metadata_basenameRichard Maw2014-09-191-4/+1
| |
| * Rename BuildCommand.get_sources to fetch_sourcesRichard Maw2014-09-192-3/+3
| | | | | | | | | | There's other methods called get_sources in other modules, and fetch_sources explains more about what it does in the context.
| * Move metadata_version value to cachekeycomputerRichard Maw2014-09-192-5/+1
| | | | | | | | | | | | | | | | | | | | | | | | metadata_version is for when the format of the /baserock metadata files changes. This means it would make sense for it to either live with the code for generating the metadata, or the cache key code so it lives with the rest of the compatibility values. Since the code for generating the metadata isn't in a nice module anywhere, I've put it in the cachekeycomputer module.
| * Eliminate BuilderBase.new_artifactRichard Maw2014-09-192-14/+3
| | | | | | | | | | | | | | | | | | | | It was an odd thing to have, when Artifact objects are part of your input. Its purpose appears to have been to allow the build step to produce an artifact called $morphology_name-rootfs, but since the split rules decide that the artifact is called that anyway, the new_artifact step is redundant.
| * Validate build-depends in MorphologyLoaderSam Thursfield2014-09-192-5/+56
| |
| * Don't validate morphologies during ArtifactResolverRichard Maw2014-09-172-49/+9
| |
| * Add __repr__ method to Source and SplitRulesRichard Maw2014-09-173-0/+23
| | | | | | | | | | This helps debugging issues with rule matching, since SplitRules can be print-statemented
| * Make remoteartifactcache tests define sensible artifactsRichard Maw2014-09-171-3/+3
| | | | | | | | | | | | | | | | It rather peculiarly defines artifacts that have different cache keys, but the same source. This flies in the face of how real artifacts get cache keys, and our ability to move the cache key to being per-source.
| * Be more lenient in Artifact dependency unit testsRichard Maw2014-09-171-4/+6
| | | | | | | | | | We don't care if add_dependency causes there to be multiple dependents, just that our artifacts are properly included.
| * Stop creating a separate -kernel artifact for systemsRichard Maw2014-09-171-20/+0
| | | | | | | | | | | | | | This pre-dates deployment, and if we need the kernel, we can always copy it out of the rootfs. It also uses new_artifact, which is a method I want to remove.
* | Prevent cliapp from logging env. variables with 'PASSWORD' in their nameSam Thursfield2014-09-193-0/+19
| | | | | | | | | | The upstream cliapp project is not interested in this functionality right now.
* | Don't log environment variables with 'PASSWORD' in their name.Sam Thursfield2014-09-193-21/+23
|/ | | | | | | | | | | | | | | | | | | This involved rewriting the util.log_dict_diff() function. It has been renamed to log_environment_changes() to better reflect its purpose. It no longer logs both the old and new values in the event of an environment variable changing. It now just logs the new value. This makes the code simpler and seems like it should not be a big problem. Some projects recommend passing credentials through the environment. OpenStack does this, for example, see: <http://docs.openstack.org/user-guide/content/cli_openrc.html> It's unlikely that users would be happy about applications saving these passwords in log files all over their system. I do not recommend ever storing valuable passwords in the environment.
* Fix tarfile.chown for versions of python smaller than 2.7.3baserock/franred/bugfix/python-tarfile-chownFrancisco Redondo Marchena2014-09-121-9/+38
| | | | | | | | This workaround fix http://bugs.python.org/issue12841 The code added in this patch is from tarfile.py from python 2.7.3. Also only use the workaround for tarfile.makefile when the ptyhon is smaller than 2.7.4
* Fix `morph branch-from-image` to work for deployed imagesbaserock/richardmaw/bugfix/branch-from-deployed-imageRichard Maw2014-09-111-1/+2
| | | | | | It broke when we added /baserock/deployment.meta. We didn't notice this because our test suite was looking at the artifact produced by morph build, and listed on the terminal.
* Fix up openstack.write extensionbaserock/pedroalvarez/openstack-exts-fixupsPedro Alvarez2014-09-091-1/+0
| | | | | | The openstack.write extension was calling a nonexistent method 'check_location'. This metod was moved to openstack.check in the commit ba7d1d1ed3bad002ce36e5d4adf4e3794625091a.
* Fix openstack write/check exts to pass the testsPedro Alvarez2014-09-092-2/+2
|