summaryrefslogtreecommitdiff
path: root/morphlib/builder.py
Commit message (Collapse)AuthorAgeFilesLines
* Use app.status() to log ldconfig messagesTiago Gomes2015-09-281-5/+7
| | | | | | | Using app.status() produces nicer logs as the messages will have the status prefix prepended. Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
* Replace cliapp.shell_quote() with pipes.quote()Tiago Gomes2015-09-281-1/+2
| | | | | | | | | Replace cliapp.shell_quote() with pipes.quote() to produce the chroot scripts. Despite pipes.quote() being deprecated, it produces much less escaping than cliapp.shell_quote(). This method should be replaced with shlex.quote() when migrating to Python 3. Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
* Remove no-longer-the-case FIXMETiago Gomes2015-09-221-3/+0
| | | | Change-Id: Iedefd1745e85e1f30d45ee8d26b315760a542a0b
* Simplify StagingArea classTiago Gomes2015-09-221-11/+9
| | | | | | | | | | | | Pass the Source to the staging area constructor so that we don't need to pass it as a parameter when we call some StagingArea methods. Also move the creation of the build and destdir directories to the constructor so that we can get rid of the chroot_open() and chroot_close() methods. Also, provide API to retrieve the relative locations for buildir and destdir. Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
* Rework debugging information for failed buildsTiago Gomes2015-09-011-45/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Improve error messages and remove redundant information * Stop displaying the containerised settings, as they are not very helpful to debug a build failure * Display the error code * Change system integration commands logging to consistent with build commands logging. * Provide a log file in the staging directory for each chunk build failure or system integration failure. * Provide a chroot script in the staging directory for each chunk build failure or system integration build failure. This script allows the user to enter an environment similiar to the one where the build failed. * Fix a bug where the system integration commands stdout was being omitted when these commands failed. * Fix a bug where the error_message_for_containerised_commandline() was being called with an empty error message, as the stderr was being directed to stdout. Chunk build failure output before the this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands build failed # configure # # echo "configure-commands stdout" configure-commands stdout # # echo "configure-commands stderr" >&2 configure-commands stderr # build # # echo "build commands stdout" build commands stdout # # echo "build commands stderr" >&2 build commands stderr # # echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo some more sdout some more stderr /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin sh: foo: not found ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c 'echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo ': Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball', '/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True, 'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths': ['/src/tmp/staging/tmpglc5Sz/test.build', '/src/tmp/staging/tmpglc5Sz/test.inst', '/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc', '/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz', 'cwd': '/test.build'} Error output: $> Chunk build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands ### CONFIGURE-COMMANDS ### + echo configure-commands stdout configure-commands stdout + echo configure-commands stderr configure-commands stderr ### BUILD-COMMANDS ### + echo build commands stdout build commands stdout + echo build commands stderr build commands stderr + echo some more sdout some more sdout + echo some more stderr some more stderr + echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin + foo sh: foo: not found ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi t_code=127) $> System integration build failure output before this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while running system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while building system ERROR: Command failed: baserock/system-integration/00-test-test-misc-0002: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'} Error output: + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found $> System integration build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the system integration commands + echo stdout stdout + echo stderr stderr + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration commands failed (exit_code=127) $> Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
* Stop moving staging areas of failed buildsTiago Gomes2015-09-011-2/+0
| | | | | | | | | | | | | | | | | | | | | Stop moving staging areas of failed builds from the 'staging' directory to the 'failed' directory. Moving staging areas make it very difficult to debug build failures on the build essential chunks, as the paths set on the configure scripts and some environment variables (e.g. STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area. This change will also make it easier to create scripts that chroot n environment similiar to the one where the build failure occurred. To make it still possible to safely do a build an run `morph gc` in parallel, we use flock(2) to control access to the staging area directory. Also, move the `test_supports_non_isolated_mode` test into a different class, as it requires a different SetUp() routine (the staging area is contructed with different parameters). Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
* Use the shell X-ray modeTiago Gomes2015-08-121-5/+2
| | | | | | | | | | | Instead of logging every command for ourselves, use the shell X-ray mode. As side effect, each command argument will now be printed out in its evaluated form, which aids debugging. Also, improve the visual separation between the different build steps, and display those steps with finer granularity. Change-Id: I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8
* Fix crash if artifact doesn't contain a /baserock directorySam Thursfield2015-08-111-4/+8
| | | | | | | This is probably a sign that something is broken, but it's better to cope with it rather than crashing. Change-Id: I4e9c40d0b7db031ea271143938db61c4ad70a888
* Warn if system-integration commands are defined for an unknown artifactSam Thursfield2015-08-111-0/+7
| | | | | | | | | | | | The system-integration field in a chunk .morph file defines which of the chunk artifacts the integration script should live. If you list an artifact that doesn't exist, the integration script will not go anywhere and the commands won't run. Ideally system-integration commands will be become a bit more simple to use, but hopefully this warning helps in the meantime. Change-Id: I10a068e79eb46f98f0bd308a5caf3c445dda22ed
* Set chunk commands' stdin to /dev/nullRichard Ipsum2015-08-071-7/+9
| | | | | | | Chunk commands will currently block if they read from stdin. To fix this we set stdin to /dev/null so that the first read returns EOF. Change-Id: I143396f4c443c098b2880c5925e6f0907cb4fb17
* Treat True and False as actual commands, rather than booleans.Jim MacArthur2015-06-161-0/+3
| | | | | | | | | | | It's quite easy to forget that "true" and "false" are special words in YAML, and write "false" to halt the build process while debugging. This was treated as a Boolean instead of a string, so causes an unhelpful error in morph. Since there is no use for booleans in morphologies, they should be treated as strings. Change-Id: I7c872f9696611920febec5f375b599eee89d040e
* cachekey/builder: Use and cache strip commandsRichard Maw2015-06-121-5/+10
| | | | Change-Id: I19a6c31979aa36ff3c03f41e16e2d25ef407533b
* morphlib/builder.py: Add comment, os-release is now generated as an extensionJavier Jardón2015-04-281-0/+2
| | | | | | | | Do not remove this code for now as its possible that new versions of morph build old sytems without the install-essential-files extension Change-Id: I5e49d0bc6baee73bedf16469d922822e76286bc1
* Give more helpful error when stratum artifacts in cache are corruptedSam Thursfield2015-04-141-10/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | I had a stratum artifact in my artifact cache which for some reason was 0 bytes long. When building a system that included this stratum, `morph build` gave me the following output 2015-04-13 13:48:57 ERROR Traceback (most recent call last): File "/src/morph/morphlib/builder.py", line 539, in build_and_cache self.unpack_strata(fs_root) File "/src/morph/morphlib/builder.py", line 600, in unpack_strata chunks = [ArtifactCacheReference(c) for c in json.load(f)] File "/usr/lib/python2.7/json/__init__.py", line 290, in load **kw) File "/usr/lib/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded With this patch, I get a better error: ERROR: Corruption detected: No JSON object could be decoded while loading /src/build/cache/artifacts/8b4422c58ecb2a085b142fbba74b760f501f65d4b2885bf707994973230e0c58.stratum.build-essential-minimal Change-Id: I0ad359901c5da75bd26d5a1a8108ef4e6f1d7708
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-3/+3
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Merge "Use the modern way of the GPL copyright header: URL instead real address"Javier Jardón2015-03-161-2/+1
|\
| * Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | | | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* | Use python3 compatible notation for octal constantsJavier Jardón2015-03-131-2/+2
|/ | | | Change-Id: I771c3de9cecda7a503f4d36ae5d9fabc040892e4
* Remove two unused functionsSam Thursfield2015-02-111-17/+0
|
* Remove some ununsed importsSam Thursfield2015-02-111-4/+0
|
* Fix line > 79 characters that was breaking ./checkSam Thursfield2015-01-191-1/+2
|
* Update CopyrightRichard Ipsum2015-01-191-1/+1
|
* Correct resolution of error_message_for_containerised_commandline method.Craig Griffiths2015-01-161-1/+1
|
* Rename builder2 module to builderSam Thursfield2014-12-191-0/+720
|
* Get rid of the old internal morph APIsLars Wirzenius2012-05-021-699/+0
|
* Make copy_repository get a repo instead of treeishLars Wirzenius2012-04-121-1/+1
| | | | This helps refactoring morph to get rid of treeishes.
* builder: ensure working directory matches commitRichard Maw2012-04-111-0/+1
| | | | | | | | When making a copy of the repository to build from, builder copies the .git directory then checks out the ref it wants. However sometimes this doesn't add the files to the working directory, possibly because the files it is missing weren't changed in the difference between what the .git thought HEAD was and what is now checked out.
* Avoid setting the mtime of broken symlinks.Jannis Pohlmann2012-04-051-1/+3
|
* Fall back to 'manual' build system if none is specified.Jannis Pohlmann2012-04-021-4/+1
|
* Automatically deduce type of chunk if one doesn't existLars Wirzenius2012-03-301-1/+2
|
* Make use of the new build system class in ChunkBuilderLars Wirzenius2012-03-301-36/+2
|
* Refactor: remove unnecessary methodLars Wirzenius2012-03-271-7/+5
| | | | | | | I found the write_cache_metadata and prepare_binary_metadata methods be confusingly similar, when skimming code. Since write_cache_metadata was very short and only used from one place, inlined it, which removes the confusion without, I think, adding complexity.
* Add --target-cflags option; set arch/fp in baserock-bootstrap on ARM.Jannis Pohlmann2012-03-261-1/+1
| | | | | | | | The --target-cflags option is really just a temporary solution. At some point we will want to have a proper target architecture/device description that morph uses to define compiler flags etc. For now, I hope this will do.
* Fix a few typos and so on.Jannis Pohlmann2012-03-231-1/+1
|
* fix overly long linesRichard Maw2012-03-231-2/+4
|
* morphlib: move filesystem stuff out of builderRichard Maw2012-03-231-39/+8
| | | | | The system images will later need to be read, so useful commands want to be shared
* make get_cache_id not require a blob to be passedRichard Maw2012-03-231-15/+13
| | | | It only needs a morph, not a blob
* Squashme: Temporary changes to make boot with btrfs work.Jannis Pohlmann2012-03-221-26/+32
| | | | | This is achieved by copying vmlinuz, System.map and extlinux.conf to the default subvolume.
* Squashme: Switch to generating images using btrfs.Super user2012-03-221-9/+16
|
* Build system images using btrfs and factory/factory-run subvolumes.Jannis Pohlmann2012-03-221-16/+31
|
* Merge remote branch 'origin/master' into rm/morph-pass1Richard Maw2012-03-161-4/+4
|\
| * tar: add -h option to extractionRichard Maw2012-03-161-1/+1
| | | | | | | | | | | | This should stop tar from replacing symbolic links with directories and instead follow the symbolic links, extracting the files into the linked directories
| * SystemBuilder: add newlines to fstabRichard Maw2012-03-161-3/+3
| | | | | | | | f.write does not add them
* | Merge branch 'master' into rm/morph-pass1-mergeRichard Maw2012-03-161-20/+44
|\ \ | |/
| * Fix morph cache key to use morphology filename rather than declared nameLars Wirzenius2012-03-151-1/+1
| | | | | | | | | | | | | | Previously, we could get a situation where chunks named the same way in two morphologies could interfere with each other: they could generate the same cache key, resulting in their chunks overwriting each other. Now this won't happen anymore.
| * Fix system image building with morphLars Wirzenius2012-03-151-19/+43
| | | | | | | | The mkimage.sh script should now be unnecessary again.
* | build-system: autotools: use PREFIXRichard Maw2012-03-141-1/+1
| |
* | morphlib.builder.Builder: PREFIX changes cache-keyRichard Maw2012-03-131-1/+1
| | | | | | | | | | Add PREFIX to the whitelist of environment variables that can affect the cache keys.
* | morphlib.builder.Builder: fix sharing envRichard Maw2012-03-131-5/+8
| | | | | | | | | | | | | | | | | | | | Previous change would share the same environment between all Builders this may not have broken anything yet, but this change in behaviour was unintentional. The relevant cache keys are not altered by Builders and they are set at the time the Builder is created, so the build string can be cached rather than generated every time a cache key is needed
* | morphlib.builder.Builder: key depends on some envRichard Maw2012-03-131-2/+5
|/ | | | | | env is created in Builder's constructor from app.clean_env cache keys now contain parts of the environment that may cause chunks to be built differently
View Lorry Controller Status