summaryrefslogtreecommitdiff
path: root/morphlib/stagingarea.py
Commit message (Collapse)AuthorAgeFilesLines
* Get rid of the CachedRepo class (almost)Sam Thursfield2016-03-031-1/+1
| | | | | | | | | | | | | | | | | | For a long time the CachedRepo class has basically been a wrapper around the GitDir class, but with a few extra methods that don't really even belong there. It is now a tiny class in the localrepocache module which just keeps track of a few extra attributes. All other functionality is provided by the gitdir module. This commit also removes the `git clone` approach for copying repos out of the cache. The alternative approach implemented by git.copy_repository() was slightly faster when I tested, so for now we should use that everywhere. Longer term we should find out why this is quicker than `git clone`, and fix Git itself to be fast. Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
* Remove old cruftTiago Gomes2015-11-201-1/+0
| | | | | | | Static analysis showed this code is not being used, and we can always bring it back if necessary through git. Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
* Replace cliapp.shell_quote() with pipes.quote()Tiago Gomes2015-09-281-1/+2
| | | | | | | | | Replace cliapp.shell_quote() with pipes.quote() to produce the chroot scripts. Despite pipes.quote() being deprecated, it produces much less escaping than cliapp.shell_quote(). This method should be replaced with shlex.quote() when migrating to Python 3. Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
* Simplify StagingArea classTiago Gomes2015-09-221-58/+26
| | | | | | | | | | | | Pass the Source to the staging area constructor so that we don't need to pass it as a parameter when we call some StagingArea methods. Also move the creation of the build and destdir directories to the constructor so that we can get rid of the chroot_open() and chroot_close() methods. Also, provide API to retrieve the relative locations for buildir and destdir. Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
* Rework debugging information for failed buildsTiago Gomes2015-09-011-7/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Improve error messages and remove redundant information * Stop displaying the containerised settings, as they are not very helpful to debug a build failure * Display the error code * Change system integration commands logging to consistent with build commands logging. * Provide a log file in the staging directory for each chunk build failure or system integration failure. * Provide a chroot script in the staging directory for each chunk build failure or system integration build failure. This script allows the user to enter an environment similiar to the one where the build failed. * Fix a bug where the system integration commands stdout was being omitted when these commands failed. * Fix a bug where the error_message_for_containerised_commandline() was being called with an empty error message, as the stderr was being directed to stdout. Chunk build failure output before the this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands build failed # configure # # echo "configure-commands stdout" configure-commands stdout # # echo "configure-commands stderr" >&2 configure-commands stderr # build # # echo "build commands stdout" build commands stdout # # echo "build commands stderr" >&2 build commands stderr # # echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo some more sdout some more stderr /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin sh: foo: not found ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c 'echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo ': Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball', '/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True, 'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths': ['/src/tmp/staging/tmpglc5Sz/test.build', '/src/tmp/staging/tmpglc5Sz/test.inst', '/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc', '/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz', 'cwd': '/test.build'} Error output: $> Chunk build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands ### CONFIGURE-COMMANDS ### + echo configure-commands stdout configure-commands stdout + echo configure-commands stderr configure-commands stderr ### BUILD-COMMANDS ### + echo build commands stdout build commands stdout + echo build commands stderr build commands stderr + echo some more sdout some more sdout + echo some more stderr some more stderr + echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin + foo sh: foo: not found ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi t_code=127) $> System integration build failure output before this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while running system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while building system ERROR: Command failed: baserock/system-integration/00-test-test-misc-0002: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'} Error output: + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found $> System integration build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the system integration commands + echo stdout stdout + echo stderr stderr + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration commands failed (exit_code=127) $> Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
* Stop moving staging areas of failed buildsTiago Gomes2015-09-011-20/+22
| | | | | | | | | | | | | | | | | | | | | Stop moving staging areas of failed builds from the 'staging' directory to the 'failed' directory. Moving staging areas make it very difficult to debug build failures on the build essential chunks, as the paths set on the configure scripts and some environment variables (e.g. STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area. This change will also make it easier to create scripts that chroot n environment similiar to the one where the build failure occurred. To make it still possible to safely do a build an run `morph gc` in parallel, we use flock(2) to control access to the staging area directory. Also, move the `test_supports_non_isolated_mode` test into a different class, as it requires a different SetUp() routine (the staging area is contructed with different parameters). Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
* morphlib: Use new temp_dir helper context managerRichard Maw2015-06-051-5/+2
| | | | Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-1/+1
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-3/+2
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Merge branch 'baserock/richardmaw/bugfix/stagingarea-mounts-inside-destdir-v2'Sam Thursfield2014-12-191-3/+10
|\ | | | | | | Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * Don't mount /dev/shm in bootstrap mode.Sam Thursfield2014-12-191-3/+4
| | | | | | | | | | | | There's no chroot used in bootstrap mode, so the host's /dev/shm can be used, and nothing in the world will be looking to /src/tmp/staging/tmpXXX/dev/shm for anything.
| * stagingarea: Mount things inside dirnamebaserock/richardmaw/bugfix/stagingarea-mounts-inside-destdir-v2Richard Maw2014-12-121-1/+6
| |
* | Give a useful path to failed staging areasSam Thursfield2014-12-091-4/+7
| | | | | | | | | | | | Morph tells the user that an error occurred in the staging area, then moves the staging area somewhere else. Giving the old path rather than the new path is pretty annoying.
* | Give less scary error messages when a containerised command failsSam Thursfield2014-12-091-15/+26
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This affects errors encountered at build time and at system-integration time. New errors look like this: ERROR: Command failed: baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/var/tmp/staging/tmp1YQ2yN/minimal-system-x86_64-generic.inst'} Error output: + install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644 install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted Previously the error message would have been this: Command failed: unshare --mount -- sh -ec. mount --make-rprivate / root="$1" shift while true; do case "$1" in --) shift break ;; *) mount_point="$1" mount_type="$2" mount_source="$3" shift 3 path="$root/$mount_point" mount -t "$mount_type" "$mount_source" "$path" ;; esac done exec "$@" - /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ dev/shm tmpfs none tmp tmpfs none -- linux-user-chroot --chdir . --mount-proc proc /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000 + install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644 install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
* stagingarea: Remove vestigial pre-command mounting logicRichard Maw2014-10-301-22/+2
|
* Make build commands use containerised_cmdlineRichard Maw2014-10-301-31/+14
|
* stagingarea: Mount proc and ccache inside the namespaceRichard Maw2014-10-241-9/+15
| | | | | | | | | | | This works towards allowing multiple concurrent builds in the same system, which has the same problem as deployments. This is the easy bit, since linux-user-chroot has support for bind mounts and /proc mounts. We also need to get rid of the /dev/shm mount to be able to build in parallel though.
* Create multiple sources per stratum morphologyRichard Maw2014-09-191-2/+2
| | | | | | | | | | | | | | | | | Building per-artifact results in undesirable behaviour, as multiple artifacts are produced for every chunk build. It therefore makes more sense to build per-source. This implies that actually, the model of one source per morphology is wrong and we should move the dependencies into the source. Unlike chunks however, where every chunk artifact has the same dependencies, stratum artifacts can have different dependencies. So before we can move the dependencies into the Source, we need to have as many Sources as Stratum Artifacts.
* StagingArea: Ensure staging are paths are bytestringsbaserock/richardmaw/bugfix/distbuild-eglibcRichard Maw2014-07-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | If the destdir path returned when creating a staging area is a unicode string, then when attempting to `os.walk(destdir)`, it will encounter unicode errors if there are file paths in the destdir that are not representable as unicode strings. For various as-yet unknown reasons, when building stage-2 eglibc it produces file paths that are not unicode compatible. There was previously a patch to fix this issue with regards to creating the metadata files, but it did not fix all the issues, because the build at the time was local rather than distributed. This is failing during a distributed build because morphologies are serialised into json, and during deserialisation their string values are left as unicode. Rather than doing the byte-string conversion during deserialisation, I have chosen to do it when the contents of the morphology are used, because it's only at the point where it's used to create a file path, that it matters whether it's unicode or not.
* Add logfile kwarg to staging area runcmdRichard Ipsum2014-05-141-1/+8
| | | | | | | | | | | | We use tee to write the output to a file as well as to stdout. Using Popen it should be straight forward to send the output to a pipe and then read that pipe and write to wherever. At the moment morph uses cliapp's runcmd rather than Popen. cliapp's runcmd is a blocking call, so we're not able to read from the pipe until the command has completed, which prevents real time logging to a number of files. One solution to this problem might be to spawn a thread which opens a pipe to the command being executed, the thread then reads from the pipe and writes to our collection of logfiles.
* Use morphlib.fsutils.mount when possiblePedro Alvarez2014-03-181-1/+1
|
* Reduce spam from setting up staging areaLars Wirzenius2014-03-101-1/+4
| | | | | | | | | | | Make the message "Installing chunk..." be chatty, i.e., only displayed when the user turns verbosity higher. Most of the time the chunk is already unpacked in the cache, so installing it takes a fraction of a second. Add a new message, at default verbosity, when a chunk needs to be unpacked. This can take a while, so a message is appropriate, so the user knows what is happening.
* Tidy up debug output of morph buildDaniel Firth2013-10-281-4/+1
|
* Added chunkname prefix to some elements of the debug log.Dan Firth2013-10-111-3/+3
|
* Remove redundant dict.getRichard Maw2013-08-231-1/+0
|
* Merge branch 'baserock/richardmaw/S8563/bootstrap-rootfs-protection'Richard Maw2013-08-231-31/+36
|\ | | | | | | Reviewed-by: Lars Wirzenius
| * Make the tempdir writable in bootstrap mode.Richard Maw2013-08-161-3/+7
| | | | | | | | | | Somehow this was working on x86 even though it had no ability to write to tempdirs, but on ARM it wasn't working.
| * stagingarea: use linux-user-chroot in bootstrapsRichard Maw2013-08-151-28/+30
| | | | | | | | | | This changes the semantics of the use_chroot flag to instead mean chrooting to /
| * stagingarea: use invert_paths to bind-mount roRichard Maw2013-08-151-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This improves upon the logic by allowing subdirectories to be marked as writable. This is not needed in its state here, but it will be built upon. It also does not attempt to make symlinks read-only, since the symlink resolution is done before chrooting, so there will be dangling links, which cause linux-user-chroot to fail during the bootstrap. This also uses the --chdir option of linux-user-chroot instead of running a shell script to cd and run the command.
* | Disable network during buildsJonathan Maw2013-08-131-0/+1
|/
* stagingarea: move to failed on construction failRichard Maw2013-08-061-0/+14
| | | | | | | | | This uses the same logic as when a build fails, so it's been consolidated into `StagingArea.abort()`. You could argue that if a build fails before any commands are run, then there's nothing interesting to see, but it will be useful if the hardlink/tarball extract algorithm fails in some corner case.
* stagingarea: Construct hardlink cache atomicallyRichard Maw2013-08-061-6/+14
| | | | | | | | | | | | Paul had file system problems which led to a partially constructed chunk hardlink cache, which caused later builds to fail, since they got a partially extracted chunk. This patch fixes the case where the failed extract caused unreproducible builds, but it's possible to corrupt the hardlink cache in other ways. Read-only btrfs subvolumes would fix this, but either tie us further to btrfs, or complicates the codebase with fallback logic.
* Change the structure of the temporary directory used by morphTiago Gomes2013-06-051-3/+3
| | | | | | Now, inside the temporary directory we will have the following subdirectories: chunks, staging, failed and deployments. The failed directory will contain the staging areas of failed builds.
* Added exception to give smaller error messageRic Holland2013-03-201-1/+7
|
* Fix long lineSam Thursfield2013-03-151-1/+2
|
* Only bind mount necessary directories in chrootSam Thursfield2013-03-131-7/+5
| | | | | If for example /bin is a symlink, we shouldn't and can't mount the host system's /bin over the top.
* Add 'prefix' property for chunks within strataSam Thursfield2013-03-131-2/+2
| | | | | | | | | Morph no longer supports setting the prefix using the --prefix argument / setting. This was only used in tests and during bootstrap. If a chunk build-depends on a chunk within a stratum which has a custom prefix, that prefix is appended to the PATH in the build environment.
* Add 'build-mode' field for chunks in a stratumSam Thursfield2013-03-131-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | Allowed values: staging: build with a staging chroot (default) test: build with the host's tools bootstrap: build with the host's tools, and do not include this chunk in the final stratum artifact In the past, 'normal mode' has been used to describe building a chunk with the host's tools. We don't want that mode to ever be used, because it is a huge hole in reproducability, but we need to keep it around to avoid making Morph's cmdtest suite depend on Baserock. Hopefully naming it 'test' should discourage potential abusers. It is unfortunate that the build tests now take a separate code path compared to real-world usage of Morph. However, this is necessary to avoid a circular dependency between Morph's test suite and the build-essential stratum in Baserock. We do whole-build testing of Baserock, too, so the 'staging' code path is still tested outside of Morph. However, testing a staging area requires populating it with at minimum a working shell, and this is a bit too complex to go in Morph's test suite.
* Refactor build processSam Thursfield2013-03-131-16/+36
| | | | | | | | | | | | | | | | | | Reorganise the build_artifact() and build_artifacts() functions to allow more complex work when setting up chunk builds in build_artifact(). The staging area now holds the BuildEnvironment object (the environment variables that should be set during build). This makes sense because all build commands should be run inside the staging area and therefore through the StagingArea object. The BuildEnvironment object is now considered immutable after it is created. The environment is used in cache key computation when computing what artifacts are required; if it changes after that point we risk either computing different artifact keys for the same artifact or missing data in the cache key that should be included in the hash. Better to force changes into a separate 'extra_env' variable.
* Remove support for bootstrap mode in staging areaSam Thursfield2013-03-131-12/+8
| | | | | This involved having a staging area with split personalities and was generally a bit ugly.
* Fix existence checks to not follow symlinksLars Wirzenius2013-03-011-4/+4
| | | | | | | | | | | | When we hardlink the staging area tree, we were checking if the target file already exists with os.path.exists. Unfortunately, this follows a symlink, and in this case, we don't want to: if the symlink target exists, and is a symlink, we want to remove the symlink only, and we don't care about its target. The target file may exists because of overlapping files in chunks. Reported-By: Paul Sherwood
* Perform *-commands in linux-user-chrootJonathan Maw2013-02-281-2/+17
|
* Hardlink files into the staging-areaJonathan Maw2013-02-281-2/+151
| | | | | | Instead of repeatedly unpacking tarballs into a succession of staging- areas, it will unpack each tarball only once, then hardlink all the files into the staging-area instead.
* Revert "Merge branch 'liw/hardlink-staging-area-pre-rebase'"Jannis Pohlmann2012-11-301-67/+4
| | | | | | | | | | | | | | This reverts commit c859d70d86423a52bc7053abf64e9ca21f62a487, reversing changes made to 3085a672d1e8b5177be33f0463385de72a0ef5bf. Unfortunately, hardlinking and linux-user-chroot both break builds in various ways. Hardlinking breaks the bootstrap process by creating symlinks like /usr/libexec that can then not be overwritten with real files by install scripts from morphologies like gcc. linux-user-chroot caused problems by breaking privileged operations such as chgrp and CAP_SETFCAP. As a consequence, chunks like util-linux and libcap can no longer be built.
* Change method comments to docstringsLars Wirzenius2012-11-291-3/+6
| | | | This also makes coverage.py be silent on Debian squeeze.
* Reformat code, plus adapt for non-staging-area buildsLars Wirzenius2012-11-291-15/+20
|
* Refactor and fix to make it pass (more of) the test suiteLars Wirzenius2012-11-291-8/+12
|
* Hardlinked chroot done with linux-user-chrootJoe Burmeister2012-11-291-3/+54
| | | | | | | | | | | | | | The patch gives two things. Improves morph build time by reusing decompressed files of chunks/stage-fillers with hardlinks from the chroot. Rather than decompressing each time into each chroot. Original: real 5h 17m 47s Hardlink: real 2h 52m 27s It uses linux-user-chroot to create the chroot and make all but the basics readonly.
* python scripts: pep8ize codebaseRichard Maw2012-08-011-24/+23
| | | | | | | | | This was done with the aid of the pep8 script, available by running `easy_install pep8`. It may be worth making this part of ./check, but that will require putting pep8 into the development tools stratum. This should be easy, given pep8 has no external dependencies.
* morph: remove dead code and replace Execute with app.runcmdRichard Maw2012-05-301-3/+3
|
View Lorry Controller Status