summaryrefslogtreecommitdiff
path: root/morphlib/builder.py
Commit message (Collapse)AuthorAgeFilesLines
* builder.py: Run commands using '-e' flag in 'sh'Pedro Alvarez2016-08-021-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Some failing commands in a yaml block were being ignored. For example the following commands... pre-configure-commands: - | cp nonexistent foo echo foo ... where giving the following result without making the build fail: ### PRE-CONFIGURE-COMMANDS ### + cp nonexistent foo cp: can't stat 'nonexistent': No such file or directory + echo foo foo This commit also adjusts the test-shell.c to allow the use of '-e' flag. No other changes because the shell was already failing after the first failure found. Change-Id: Ie78b25a99f1b5d6d0d26be74ddc377025dff69fd
* Make clear how long a command took to failPedro Alvarez2016-06-031-0/+2
| | | | | | | | | | | | | | This is done by adding an status update when a command fails. An example of the new output is: 2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Creating staging area 2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Starting actual build: stage2-libstdc++ fc4868e 2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Extracting /src/gcc-tarball into /src/tmp/staging/tmpE5b84y/stage2-libstdc++.build 2016-06-03 10:35:38 [Build 5/133] [stage2-libstdc++] Running configure-commands 2016-06-03 10:35:55 [Build 5/133] [stage2-libstdc++] Running build-commands 2016-06-03 10:36:14 [Build 5/133] [stage2-libstdc++] Error when running build-commands Change-Id: Iac94794311e649d13c534f453d3473ac8003e8f4
* Add support for definitions version 8Pedro Alvarez2016-03-291-13/+29
| | | | | | | | | This code is a rework from changes done by: - Tiago Gomes <tiago.gomes@codethink.co.uk> https://storyboard.baserock.org/#!/story/86 Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
* Unify local and remote repo cache modulesSam Thursfield2016-03-161-2/+2
| | | | | | | | | | | | | | | | | | | | | There's not really any reason you'd want to use the RemoteRepoCache class except as a workaround for the slow speed of some LocalRepoCache operations, so I can't see this ruining anyone's day. The main reason for doing this is so we can simply the sourceresolver code. One reason that the sourceresolver class is so hopelessly complicated is that it right now has to use two incompatible interfaces for Git repo caches. I've taken the opportunity to detangle the RepoCache class from the App class. Now all of the configuration for the RepoCache class is passed into the constructor explicitly. This makes the class usable from outside Morph: resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[]) repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver) Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
* Get rid of the CachedRepo class (almost)Sam Thursfield2016-03-031-3/+3
| | | | | | | | | | | | | | | | | | For a long time the CachedRepo class has basically been a wrapper around the GitDir class, but with a few extra methods that don't really even belong there. It is now a tiny class in the localrepocache module which just keeps track of a few extra attributes. All other functionality is provided by the gitdir module. This commit also removes the `git clone` approach for copying repos out of the cache. The alternative approach implemented by git.copy_repository() was slightly faster when I tested, so for now we should use that everywhere. Longer term we should find out why this is quicker than `git clone`, and fix Git itself to be fast. Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
* Hide get_repo() and cache_repo() functions, always use get_updated_repo()Sam Thursfield2016-03-031-1/+1
| | | | | | | | | | This allows us to simplify a couple of places. I cannot think of a single situation where'd you want to get the cached copy of a repo, but not update it. Think about it -- the repo might be *years* behind the upstream remote. Change-Id: I60340c7fb33e7bfe871ad30c0a9322a7202548e2
* Fix unused 'bs' var in builderRichard Ipsum2016-02-151-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There's no longer any need to look up the buildsystem in builder, since the morphloader now sets commands directly in the morphology. Removing this unused variable also fixes a bug where morph can crash if the chunk being built is being built with a buildsystem that is not hardcoded in morph, error below: 2016-02-15 15:29:46 Starting build of devel-system-x86_64-generic 2016-02-15 15:29:46 [Build 70/407] [cython.morph] Building chunk cython.morph 2016-02-15 15:29:46 [Build 70/407] [cython.morph] Creating staging area 2016-02-15 15:29:47 [Build 70/407] [cython.morph] Starting actual build: cython.morph 4dd8e76 2016-02-15 15:29:47 [Build 70/407] [cython.morph] Extracting upstream:cython into /src/tmp/staging/tmpLI9TUj/cython.morph.build Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 299, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/build_plugin.py", line 208, in build self._build(source_pool, filename, component_names=component_names) File "/src/morph/morphlib/plugins/build_plugin.py", line 237, in _build bc.build_in_order(component) File "/src/morph/morphlib/buildcommand.py", line 297, in build_in_order self.cache_or_build_source(s, build_env) File "/src/morph/morphlib/buildcommand.py", line 317, in cache_or_build_source self.build_source(source, build_env) File "/src/morph/morphlib/buildcommand.py", line 374, in build_source self.build_and_cache(staging_area, source, setup_mounts) File "/src/morph/morphlib/buildcommand.py", line 545, in build_and_cache return builder.build_and_cache(source) File "/src/morph/morphlib/builder.py", line 748, in build_and_cache built_artifacts = o.build_and_cache() File "/src/morph/morphlib/builder.py", line 288, in build_and_cache self.run_commands(temppath, stdout) File "/src/morph/morphlib/builder.py", line 314, in run_commands bs = morphlib.buildsystem.lookup_build_system(m['build-system']) File "/src/morph/morphlib/buildsystem.py", line 300, in lookup_build_system raise KeyError('Unknown build system: %s' % name) KeyError: 'Unknown build system: python3-distutils' Change-Id: Iddf651b408c612c7c80296012cf1c97ab99e8ccc
* Use app.status() to log ldconfig messagesTiago Gomes2015-09-281-5/+7
| | | | | | | Using app.status() produces nicer logs as the messages will have the status prefix prepended. Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
* Replace cliapp.shell_quote() with pipes.quote()Tiago Gomes2015-09-281-1/+2
| | | | | | | | | Replace cliapp.shell_quote() with pipes.quote() to produce the chroot scripts. Despite pipes.quote() being deprecated, it produces much less escaping than cliapp.shell_quote(). This method should be replaced with shlex.quote() when migrating to Python 3. Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
* Remove no-longer-the-case FIXMETiago Gomes2015-09-221-3/+0
| | | | Change-Id: Iedefd1745e85e1f30d45ee8d26b315760a542a0b
* Simplify StagingArea classTiago Gomes2015-09-221-11/+9
| | | | | | | | | | | | Pass the Source to the staging area constructor so that we don't need to pass it as a parameter when we call some StagingArea methods. Also move the creation of the build and destdir directories to the constructor so that we can get rid of the chroot_open() and chroot_close() methods. Also, provide API to retrieve the relative locations for buildir and destdir. Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
* Rework debugging information for failed buildsTiago Gomes2015-09-011-45/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Improve error messages and remove redundant information * Stop displaying the containerised settings, as they are not very helpful to debug a build failure * Display the error code * Change system integration commands logging to consistent with build commands logging. * Provide a log file in the staging directory for each chunk build failure or system integration failure. * Provide a chroot script in the staging directory for each chunk build failure or system integration build failure. This script allows the user to enter an environment similiar to the one where the build failed. * Fix a bug where the system integration commands stdout was being omitted when these commands failed. * Fix a bug where the error_message_for_containerised_commandline() was being called with an empty error message, as the stderr was being directed to stdout. Chunk build failure output before the this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands build failed # configure # # echo "configure-commands stdout" configure-commands stdout # # echo "configure-commands stderr" >&2 configure-commands stderr # build # # echo "build commands stdout" build commands stdout # # echo "build commands stderr" >&2 build commands stderr # # echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo some more sdout some more stderr /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin sh: foo: not found ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c 'echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo ': Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball', '/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True, 'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths': ['/src/tmp/staging/tmpglc5Sz/test.build', '/src/tmp/staging/tmpglc5Sz/test.inst', '/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc', '/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz', 'cwd': '/test.build'} Error output: $> Chunk build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands ### CONFIGURE-COMMANDS ### + echo configure-commands stdout configure-commands stdout + echo configure-commands stderr configure-commands stderr ### BUILD-COMMANDS ### + echo build commands stdout build commands stdout + echo build commands stderr build commands stderr + echo some more sdout some more sdout + echo some more stderr some more stderr + echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin + foo sh: foo: not found ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi t_code=127) $> System integration build failure output before this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while running system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while building system ERROR: Command failed: baserock/system-integration/00-test-test-misc-0002: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'} Error output: + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found $> System integration build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the system integration commands + echo stdout stdout + echo stderr stderr + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration commands failed (exit_code=127) $> Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
* Stop moving staging areas of failed buildsTiago Gomes2015-09-011-2/+0
| | | | | | | | | | | | | | | | | | | | | Stop moving staging areas of failed builds from the 'staging' directory to the 'failed' directory. Moving staging areas make it very difficult to debug build failures on the build essential chunks, as the paths set on the configure scripts and some environment variables (e.g. STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area. This change will also make it easier to create scripts that chroot n environment similiar to the one where the build failure occurred. To make it still possible to safely do a build an run `morph gc` in parallel, we use flock(2) to control access to the staging area directory. Also, move the `test_supports_non_isolated_mode` test into a different class, as it requires a different SetUp() routine (the staging area is contructed with different parameters). Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
* Use the shell X-ray modeTiago Gomes2015-08-121-5/+2
| | | | | | | | | | | Instead of logging every command for ourselves, use the shell X-ray mode. As side effect, each command argument will now be printed out in its evaluated form, which aids debugging. Also, improve the visual separation between the different build steps, and display those steps with finer granularity. Change-Id: I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8
* Fix crash if artifact doesn't contain a /baserock directorySam Thursfield2015-08-111-4/+8
| | | | | | | This is probably a sign that something is broken, but it's better to cope with it rather than crashing. Change-Id: I4e9c40d0b7db031ea271143938db61c4ad70a888
* Warn if system-integration commands are defined for an unknown artifactSam Thursfield2015-08-111-0/+7
| | | | | | | | | | | | The system-integration field in a chunk .morph file defines which of the chunk artifacts the integration script should live. If you list an artifact that doesn't exist, the integration script will not go anywhere and the commands won't run. Ideally system-integration commands will be become a bit more simple to use, but hopefully this warning helps in the meantime. Change-Id: I10a068e79eb46f98f0bd308a5caf3c445dda22ed
* Set chunk commands' stdin to /dev/nullRichard Ipsum2015-08-071-7/+9
| | | | | | | Chunk commands will currently block if they read from stdin. To fix this we set stdin to /dev/null so that the first read returns EOF. Change-Id: I143396f4c443c098b2880c5925e6f0907cb4fb17
* Treat True and False as actual commands, rather than booleans.Jim MacArthur2015-06-161-0/+3
| | | | | | | | | | | It's quite easy to forget that "true" and "false" are special words in YAML, and write "false" to halt the build process while debugging. This was treated as a Boolean instead of a string, so causes an unhelpful error in morph. Since there is no use for booleans in morphologies, they should be treated as strings. Change-Id: I7c872f9696611920febec5f375b599eee89d040e
* cachekey/builder: Use and cache strip commandsRichard Maw2015-06-121-5/+10
| | | | Change-Id: I19a6c31979aa36ff3c03f41e16e2d25ef407533b
* morphlib/builder.py: Add comment, os-release is now generated as an extensionJavier Jardón2015-04-281-0/+2
| | | | | | | | Do not remove this code for now as its possible that new versions of morph build old sytems without the install-essential-files extension Change-Id: I5e49d0bc6baee73bedf16469d922822e76286bc1
* Give more helpful error when stratum artifacts in cache are corruptedSam Thursfield2015-04-141-10/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | I had a stratum artifact in my artifact cache which for some reason was 0 bytes long. When building a system that included this stratum, `morph build` gave me the following output 2015-04-13 13:48:57 ERROR Traceback (most recent call last): File "/src/morph/morphlib/builder.py", line 539, in build_and_cache self.unpack_strata(fs_root) File "/src/morph/morphlib/builder.py", line 600, in unpack_strata chunks = [ArtifactCacheReference(c) for c in json.load(f)] File "/usr/lib/python2.7/json/__init__.py", line 290, in load **kw) File "/usr/lib/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded With this patch, I get a better error: ERROR: Corruption detected: No JSON object could be decoded while loading /src/build/cache/artifacts/8b4422c58ecb2a085b142fbba74b760f501f65d4b2885bf707994973230e0c58.stratum.build-essential-minimal Change-Id: I0ad359901c5da75bd26d5a1a8108ef4e6f1d7708
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-3/+3
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Merge "Use the modern way of the GPL copyright header: URL instead real address"Javier Jardón2015-03-161-2/+1
|\
| * Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | | | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* | Use python3 compatible notation for octal constantsJavier Jardón2015-03-131-2/+2
|/ | | | Change-Id: I771c3de9cecda7a503f4d36ae5d9fabc040892e4
* Remove two unused functionsSam Thursfield2015-02-111-17/+0
|
* Remove some ununsed importsSam Thursfield2015-02-111-4/+0
|
* Fix line > 79 characters that was breaking ./checkSam Thursfield2015-01-191-1/+2
|
* Update CopyrightRichard Ipsum2015-01-191-1/+1
|
* Correct resolution of error_message_for_containerised_commandline method.Craig Griffiths2015-01-161-1/+1
|
* Rename builder2 module to builderSam Thursfield2014-12-191-0/+720
|
* Get rid of the old internal morph APIsLars Wirzenius2012-05-021-699/+0
|
* Make copy_repository get a repo instead of treeishLars Wirzenius2012-04-121-1/+1
| | | | This helps refactoring morph to get rid of treeishes.
* builder: ensure working directory matches commitRichard Maw2012-04-111-0/+1
| | | | | | | | When making a copy of the repository to build from, builder copies the .git directory then checks out the ref it wants. However sometimes this doesn't add the files to the working directory, possibly because the files it is missing weren't changed in the difference between what the .git thought HEAD was and what is now checked out.
* Avoid setting the mtime of broken symlinks.Jannis Pohlmann2012-04-051-1/+3
|
* Fall back to 'manual' build system if none is specified.Jannis Pohlmann2012-04-021-4/+1
|
* Automatically deduce type of chunk if one doesn't existLars Wirzenius2012-03-301-1/+2
|
* Make use of the new build system class in ChunkBuilderLars Wirzenius2012-03-301-36/+2
|
* Refactor: remove unnecessary methodLars Wirzenius2012-03-271-7/+5
| | | | | | | I found the write_cache_metadata and prepare_binary_metadata methods be confusingly similar, when skimming code. Since write_cache_metadata was very short and only used from one place, inlined it, which removes the confusion without, I think, adding complexity.
* Add --target-cflags option; set arch/fp in baserock-bootstrap on ARM.Jannis Pohlmann2012-03-261-1/+1
| | | | | | | | The --target-cflags option is really just a temporary solution. At some point we will want to have a proper target architecture/device description that morph uses to define compiler flags etc. For now, I hope this will do.
* Fix a few typos and so on.Jannis Pohlmann2012-03-231-1/+1
|
* fix overly long linesRichard Maw2012-03-231-2/+4
|
* morphlib: move filesystem stuff out of builderRichard Maw2012-03-231-39/+8
| | | | | The system images will later need to be read, so useful commands want to be shared
* make get_cache_id not require a blob to be passedRichard Maw2012-03-231-15/+13
| | | | It only needs a morph, not a blob
* Squashme: Temporary changes to make boot with btrfs work.Jannis Pohlmann2012-03-221-26/+32
| | | | | This is achieved by copying vmlinuz, System.map and extlinux.conf to the default subvolume.
* Squashme: Switch to generating images using btrfs.Super user2012-03-221-9/+16
|
* Build system images using btrfs and factory/factory-run subvolumes.Jannis Pohlmann2012-03-221-16/+31
|
* Merge remote branch 'origin/master' into rm/morph-pass1Richard Maw2012-03-161-4/+4
|\
| * tar: add -h option to extractionRichard Maw2012-03-161-1/+1
| | | | | | | | | | | | This should stop tar from replacing symbolic links with directories and instead follow the symbolic links, extracting the files into the linked directories
| * SystemBuilder: add newlines to fstabRichard Maw2012-03-161-3/+3
| | | | | | | | f.write does not add them
View Lorry Controller Status