summaryrefslogtreecommitdiff
path: root/morphlib/util.py
Commit message (Collapse)AuthorAgeFilesLines
* More robust creation of cache dirsPedro Alvarez2016-01-221-3/+18
| | | | | | | | | | | | | | | | | | When starting various distbuilds at the same time we were having randomly errors like: 2016-01-06 12:14:03 Starting distributed build 2016-01-06 12:14:03 Connecting to mason-x86-64:7878 2016-01-06 12:14:03 Requesting build of git://192.168.222.58/baserock/baserock/definitions 28b92192c00a36395acd6a960959d3b4468f9894 systems/openstack-system-x86_64.morph ERROR: /srv/distbuild/artifacts: File exists ERROR: /srv/distbuild/artifacts: File exists ERROR: /srv/distbuild/artifacts: File exists ERROR: /srv/distbuild/artifacts: File exists This has been happening in Mason since we remove every folder used by Morph after Mason finishes. Change-Id: I175de7282302d9c1e2fb1b0872f7eb72c742f28e
* Don't show progress bars if stderr is being redirected to a fileSam Thursfield2015-12-211-7/+8
| | | | | | | | | | | | | | | | | | Currently the logs of our continuous builder (https://mason-x86-64.baserock.org) are filled with lots of this sort of crap: stage1-binutils-misc[ ] 2.4/73.0 MB stage1-binutils-misc[ ] 2.4/73.0 MB stage1-binutils-misc[ ] 2.4/73.0 MB stage1-binutils-misc[ ] 2.4/73.0 MB stage1-binutils-misc[# ] 2.4/73.0 MB stage1-binutils-misc[# ] 2.5/73.0 MB stage1-binutils-misc[# ] 2.5/73.0 MB ... Change-Id: I0b7a4c8421c3ecdd141fc8323d3001ae4fb44d9b
* Prepare the removal of workspace and system branches codeTiago Gomes2015-11-251-5/+1
| | | | Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
* Only run as many parallel jobs as there are available CPU coresSam Thursfield2015-11-091-4/+6
| | | | | | | YBD has been doing this for a while and seems like it is faster as a result. Change-Id: I8f95a53195cdbc2c75c06a8abe9eb089a84b1c1b
* Add support for Baserock definitions version 7Sam Thursfield2015-10-141-0/+7
| | | | | | | | | | | | | | | | | | | | | | This adds a new 'Defaults' class to represent definitions defaults The Python 'jsonschema' module is used to validate the contents of the Defaults file. This module is already included in Baserock 'build' and 'devel' reference systems by way of the 'openstack-common' stratum. This commit embeds a copy of the JSON-Schema schema for the DEFAULTS file. I think the canonical location of this schema should be in the reference definitions.git, for now. In future, the schemas should maybe have their own repos. Either way, Morph should embed a copy for the time being so that we are sure the schema matches how Morph expects to parse the file. Morph's automated tests are all updated to use definitions version 7. I removed most of the tests for built-in build systems, because the built-ins themselves are no longer part of Morph. Only the mechanism for defining them needs to be tested now. Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
* Display progress bar when fetching to local cacheRichard Ipsum2015-09-231-0/+44
| | | | | | | | | Looks like, 2015-07-05 16:08:10 [Build 1/304] [stage1-binutils] Fetching to local cache: artifact stage1-binutils-misc stage1-binutils-misc[##################### ] 51.9/73.0 MB Change-Id: Ib10f1cfaa0c1df80ae605ecfeb5b706c8d46c4a4
* Rework debugging information for failed buildsTiago Gomes2015-09-011-15/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Improve error messages and remove redundant information * Stop displaying the containerised settings, as they are not very helpful to debug a build failure * Display the error code * Change system integration commands logging to consistent with build commands logging. * Provide a log file in the staging directory for each chunk build failure or system integration failure. * Provide a chroot script in the staging directory for each chunk build failure or system integration build failure. This script allows the user to enter an environment similiar to the one where the build failed. * Fix a bug where the system integration commands stdout was being omitted when these commands failed. * Fix a bug where the error_message_for_containerised_commandline() was being called with an empty error message, as the stderr was being directed to stdout. Chunk build failure output before the this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands build failed # configure # # echo "configure-commands stdout" configure-commands stdout # # echo "configure-commands stderr" >&2 configure-commands stderr # build # # echo "build commands stdout" build commands stdout # # echo "build commands stderr" >&2 build commands stderr # # echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo some more sdout some more stderr /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin sh: foo: not found ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c 'echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo ': Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball', '/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True, 'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths': ['/src/tmp/staging/tmpglc5Sz/test.build', '/src/tmp/staging/tmpglc5Sz/test.inst', '/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc', '/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz', 'cwd': '/test.build'} Error output: $> Chunk build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands ### CONFIGURE-COMMANDS ### + echo configure-commands stdout configure-commands stdout + echo configure-commands stderr configure-commands stderr ### BUILD-COMMANDS ### + echo build commands stdout build commands stdout + echo build commands stderr build commands stderr + echo some more sdout some more sdout + echo some more stderr some more stderr + echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin + foo sh: foo: not found ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi t_code=127) $> System integration build failure output before this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while running system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while building system ERROR: Command failed: baserock/system-integration/00-test-test-misc-0002: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'} Error output: + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found $> System integration build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the system integration commands + echo stdout stdout + echo stderr stderr + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration commands failed (exit_code=127) $> Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
* morphlib.util: Add temp_dir context managerRichard Maw2015-06-031-0/+16
| | | | | | | We commonly create a temporary directory and clean it up after we're done. This can be encapsulated into a context manager. Change-Id: Ie50de5c3954141a3e7fd40e4627fb743287ef61f
* Do not call str() on None types in sanitize_envRichard Ipsum2015-05-291-1/+1
| | | | | | | | | | | | | | | | | | | | | | | For a cluster morph, - morph: systems/devel-system-x86_64-generic.morph deploy: self: type: ssh-rsync location: root@127.0.0.1 ATTACK_KITTENS: The yaml parser will assign the value None to the element with key 'ATTACK_KITTENS', sanitize_environment blindly calls str() on each value in the dictionary, the result is that our extensions receive an 'ATTACK_KITTENS' environment variable with the string value 'None', this is not useful and confuses anyone trying to do basic validation in their extension. This patch makes it so that any None types are replaced with the empty string, which seems a more sensible value for this case. Change-Id: I734d33e9304f0b7d4d590e464e350d2e5b4720d3
* Move duplicate fix_chunk_build_mode function to a common locationAdam Coldrick2015-05-121-0/+28
| | | | Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
* morphlib.util: add word_join_listRichard Maw2015-05-111-0/+10
| | | | | | | This is useful for representing lists of items in status or exception messages. Change-Id: I530eecdc311ac77fca9922dab063f550ea810c06
* Add armv5l to list of valid architecturesJavier Jardón2015-04-281-0/+1
| | | | | | | | Also add support to allow building compatible architectures on armv7 machines, as per the rationale in the comment in _validate_architecture(). Change-Id: Ie0fe4002523f1f92f576bac8b654d4ea6fad5cf2
* Change build environment to support MIPS32/64 BE and LESimon Hoinkis2015-04-101-0/+15
| | | | Change-Id: I9344b9b80a6ec008715559390b63c9003f34bf90
* Let morph recognise big-endian kernel on ARMv8Edward Cragg2015-03-251-1/+1
| | | | | | | Let morph recognise the architecture string reported by a big-endian ARMv8 kernel. Change-Id: I64323e6c2a4675a8cc34f185b10fa04754ebe8c8
* Add write_from_dict to utilRichard Ipsum2015-03-181-0/+32
| | | | A function to read lines from a dictionary and append them to a file
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Add armv8l64 and armv8b64 to list of valid architecturesJosh Malkinson2015-01-211-3/+3
|
* Support armv8l, armv8b, aarch64, and aarch64bEmmet Hikory2014-12-191-0/+4
| | | | | | Also add support to allow building compatible architectures on armv8 machines, as per the rationale in the comment in _validate_architecture().
* Merge branch 'sam/improve-command-failure-errors'Sam Thursfield2014-12-181-1/+16
|\ | | | | | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk> Reviewed-By: Mike Smith <mike.smith@codethink.co.uk>
| * Remove duplicate 'import' statementSam Thursfield2014-12-091-1/+0
| |
| * Give less scary error messages when a containerised command failsSam Thursfield2014-12-091-0/+16
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This affects errors encountered at build time and at system-integration time. New errors look like this: ERROR: Command failed: baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/var/tmp/staging/tmp1YQ2yN/minimal-system-x86_64-generic.inst'} Error output: + install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644 install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted Previously the error message would have been this: Command failed: unshare --mount -- sh -ec. mount --make-rprivate / root="$1" shift while true; do case "$1" in --) shift break ;; *) mount_point="$1" mount_type="$2" mount_source="$3" shift 3 path="$root/$mount_point" mount -t "$mount_type" "$mount_source" "$path" ;; esac done exec "$@" - /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ dev/shm tmpfs none tmp tmpfs none -- linux-user-chroot --chdir . --mount-proc proc /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000 + install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644 install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
* Move unsharing and containerising logic to utilRichard Maw2014-10-301-0/+114
| | | | | This way the build commands, system integration commands and deployment extension commands can all share the logic.
* Never require SSH access to the Trove where baserock: content is hostedSam Thursfield2014-10-071-2/+13
| | | | | | | | | | | | | | Previously if the user included 'baserock' in their list of trove-ids, the keyed URL expansions in the repo-alias field would change such that reading from git.baserock.org required SSH access. This isn't good and in particular breaks Mason, which by default isn't set up with a Gitano account on the Trove that it reads from, and therefore doesn't have SSH access to that Trove. With this change, the trove-id field will not cause the default 'baserock:' and 'upstream:' prefixes to be overridden, so setting 'trove-id = baserock' in morph.conf will not Morph's behaviour to change.
* Safely interpolate in environment variable reportingRichard Maw2014-09-241-2/+2
| | | | | | | | | The msg parameter to status is a format string. If we pass a string directly to it, then we have to be careful to escape any formatting characters. However, we can just do the interpolation directly in the status call instead, which is less code.
* Prevent cliapp from logging env. variables with 'PASSWORD' in their nameSam Thursfield2014-09-191-0/+11
| | | | | The upstream cliapp project is not interested in this functionality right now.
* Don't log environment variables with 'PASSWORD' in their name.Sam Thursfield2014-09-191-18/+21
| | | | | | | | | | | | | | | | | | | This involved rewriting the util.log_dict_diff() function. It has been renamed to log_environment_changes() to better reflect its purpose. It no longer logs both the old and new values in the event of an environment variable changing. It now just logs the new value. This makes the code simpler and seems like it should not be a big problem. Some projects recommend passing credentials through the environment. OpenStack does this, for example, see: <http://docs.openstack.org/user-guide/content/cli_openrc.html> It's unlikely that users would be happy about applications saving these passwords in log files all over their system. I do not recommend ever storing valuable passwords in the environment.
* Transfer sparse files faster for kvm, vbox deploymentLars Wirzenius2014-09-051-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The KVM and VirtualBox deployments use sparse files for raw disk images. This means they can store a large disk (say, tens or hundreds of gigabytes) without using more disk space than is required for the actual content (e.g., a gigabyte or so for the files in the root filesystem). The kernel and filesystem make the unwritten parts of the disk image look as if they are filled with zero bytes. This is good. However, during deployment those sparse files get transferred as if there really are a lot of zeroes. Those zeroes take a lot of time to transfer. rsync, for example, does not handle large holes efficiently. This change introduces a couple of helper tools (morphlib/xfer-hole and morphlib/recv-hole), which transfer the holes more efficiently. The xfer-hole program reads a file and outputs records like these: DATA 123 binary data (exaclyt 123 bytes and no newline at the end) HOLE 3245 xfer-hole can do this efficiently, without having to read through all the zeroes in the holes, using the SEEK_DATA and SEEK_HOLE arguments to lseek. Using this, the holes take only take a few bytes each, making it possible to transfer a disk image faster. In my benchmarks, transferring a 100G byte disk image took about 100 seconds for KVM, and 220 seconds for VirtualBox (which needs to more work at the receiver to convert the raw disk to a VDI). Both benchmarks were from a VM on my laptop to the laptop itself. The interesting bit here is that the receiver (recv-hole) is simple enough that it can be implemented in a bit of shell script, and the text of the shell script can be run on the remote end by giving it to ssh as a command line argument. This means there is no need to install any special tools on the receiver, which makes using this improvement much simpler.
* Use exact filenames to refer to morphology filesRichard Maw2014-07-101-7/+16
| | | | | | | | | | | | | | | | | | | Rather than repeatedly stripping and appending an optional .morph extension morphology names, instead always use the file path of the morphology relative to the definitions repository. This is an inversion of the previous logic, which would strip the .morph extension and use the "name" internally. The exception to this rule of always using the filename, is that `morph edit CHUNK` uses the name of the morphology as-defined in the stratum. This is based off Adam Coldrick's inital patch, but this version will allow the old style of providing the "name" by converting it into a path if it does not have either a / or a . in it. An unfortunate consequence of this change is that the show-dependencies command's output changed, so the test needed updating.
* Add armv7lhf detectionRichard Ipsum2014-01-211-1/+22
|
* LocalArtifactCache now takes a an FS objectDaniel Firth2013-12-201-3/+5
|
* Modify morph to strip .morph extensions from parametersBen Brown2013-12-161-0/+9
|
* sysbranchdir: Move load_all_morphologies helper hereRichard Maw2013-11-291-1/+0
| | | | | | | | | | | | | | | This was previously a private method of the branch and merge plugin, but it's useful to other plugins, so has been moved to the SystemBranchDirectory class, where everything else can get to it. It has an unpleasant amount of coupling to other classes, but in a *good* object oriented design it would either be a tiny module on its own, or not exist and leave all its users to re-implement the same logic multiple ways, so we've opted for a less clean, but more useful design. It is left un-covered by the unit tests, since it requires a great deal of instrumentation to test, at which point it may be best to leave it to integration tests.
* Make morph able to use ppc64 architecture (POWER PC 64 bits)Pedro Alvarez2013-11-271-0/+1
|
* util: Add helper for splitting up huge iterablesRichard Maw2013-11-221-0/+10
| | | | | | | | This is used to split a large iterable into more manageable chunks. This is important when a consumer can only handle so much input at once e.g. A program can take a large, but finite number of arguments, xargs exists to handle this for shell programs.
* Add --artifact-cache-server and --git-resolve-cache-serverLars Wirzenius2013-10-291-2/+18
|
* Tidy up debug output of morph buildDaniel Firth2013-10-281-6/+5
|
* Fix app.status call in log_dict_diffLars Wirzenius2013-10-241-2/+4
| | | | | | | | | | | | | | | | This bug was triggered by the fix to CachedRepo.resolve_ref and without this fix, the resolve_ref fix will break the test suite. The bug is that log_dict_diff calls the status method with an msg keyword argument that may contain percentage characters. status interprets the value of msg as a format string, and the percentage characters trigger formatting to happen. The fix for that is to not interpolate the value of key and dictA[key] and dictB[key] into msg before calling status, but letting status do that. Thus the msg values are changed to reference %(key)s instead and passing in a value for key as a separate argument. Ditto for dictA[key] and dictB[key].
* Reduced debug output for environment propertiesDan Firth2013-10-141-1/+2
|
* Added chunkname prefix to some elements of the debug log.Dan Firth2013-10-111-4/+4
|
* util: add find_leaves() supplimenting find_leaf()Richard Maw2013-09-171-13/+22
| | | | | | | | | | | | | | | The workspace needs to be able to list all its system branches, and the system branches need to be able to list all their git repositories. This is broadly the same thing, just with a different directory to look out for, so provide that utility in morphlib.util. find_leaf() is rewritten to use find_leaves(), this is less efficient since it waits until every leaf is found. I felt it was better to reduce the code than maintain a slightly more optimal algorithm. The find_leaf() algorithm could become more optimal if it could lazily check for at least one result in a generator.
* Utility function to convert all values in a dictionary to stringsTiago Gomes2013-08-161-1/+5
| | | | | | | | | This will be useful in the next commit, where we want to construct an environment to run a command from a dictionary. The cliapp runcmd method expects that the values in this dictionary are strings, so we need to convert them before.
* Add "morph print-architecture" subcommandLars Wirzenius2013-08-151-0/+22
|
* deploy: refactor environment argument parsingRichard Maw2013-08-131-0/+32
| | | | It is now a tested helper function in morphlib.util
* Add morphlib.util.find_root and find_leaf functionsLars Wirzenius2013-08-061-6/+53
| | | | | | | These will be used to find workspace and system branch root directories. Also accidentally removed some whitespace from ends of lines. Too lazy to split that into a separate commit.
* Merge remote-tracking branch ↵Daniel Silverstone2013-06-191-0/+4
|\ | | | | | | | | | | 'origin/baserock/richarddale/check-repo-alias-option' Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
| * Check that that the repo-alias option matches the expected RERichard Dale2013-06-171-0/+4
| |
* | util: suggest how to deal with insufficient spaceRichard Maw2013-06-071-1/+7
| |
* | util: split out space unify logicRichard Maw2013-06-071-6/+19
|/
* S7904: method for checking disk space is availableRichard Maw2013-06-051-0/+21
|
* S7904: method for testing paths are on the same fsRichard Maw2013-06-051-0/+10
|
View Lorry Controller Status