summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
* deploy: Fix an unlikely crash at deploy-timeSam Thursfield2016-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For some reason I had a deployment .morph file that specified: deploy-defaults: This resulted in the deploy_defaults variable having value None, which lead to a crash further down: 2016-02-19 12:03:52 Deciding on task order Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 299, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 574, in upgrade self.deploy(args) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 370, in deploy env_vars, deployments) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 403, in deploy_cluster parent_location='') File "/src/morph/morphlib/plugins/deploy_plugin.py", line 462, in deploy_system env_vars, deployment_filter, parent_location) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 483, in deploy_system_with_source_pool system_id, env_vars, deploy_defaults, deploy_params) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 59, in configuration_for_system deploy_params.items() + AttributeError: 'NoneType' object has no attribute 'items' This change makes the code work as expected. Change-Id: I667f4142667be31797cac7c8994d35a404119cca
* sourceresolver: Use real name for sources without chunk filePedro Alvarez2016-02-161-11/+12
| | | | | | | | | | | | | | | | | | The filename was being used instead. I noticed this bug after seeing cache artifacts named like: de31c7bcac0240315bc4e0994b34f3757d37d35e4179cdbe0a8b2e60e26449f8.chunk.xorg-proto-x11proto.morph-bins de7b9f6f7a10b8dc04b52a08e74987df7da310b59dc25c6cf083b0381726e15f.chunk.networkx.morph-bins def910ed756be94c626d59d73f0770a445d62278a2eb46e031af42563691c6ed.chunk.pv.morph-bins e003da7f8c697286fa4d550299920160a6e76df449b7e434b270cd760c0c9304.chunk.cython.morph-bins e193576471c957387c9684c385a8c03db1e70618890210f6272c362ccc3e259b.chunk.xorg-proto-xcmiscproto.morph-bins e3b0b0dbb7b3de4bb8c72cd064408956296e290fc4b56acbbee241e2d78bf533.chunk.greenlet.morph-bins e3f09aa095e5839aa9bf7774720951c8b03d78300b056f7bbdeacdf4c5d5944d.chunk.libvorbis.morph-bins e437810247200713da17937181cc914e080ac7d6030451ccfaa86ddccf7a395b.chunk.python-requests.morph-bins e4b7de417c5691794f4ad6a6481cacc16242af072db39bcadede6ca2bf48241e.chunk.flac.morph-bins e4f1e6dbaeed869b445f9b45b98033b6e6a2952674b76b8465e40a67e675333c.chunk.Sub-Name.morph-bins Change-Id: I8a8697a12cf7e4c7891c93e4f774819fe2f8ea86
* Fix unused 'bs' var in builderRichard Ipsum2016-02-151-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There's no longer any need to look up the buildsystem in builder, since the morphloader now sets commands directly in the morphology. Removing this unused variable also fixes a bug where morph can crash if the chunk being built is being built with a buildsystem that is not hardcoded in morph, error below: 2016-02-15 15:29:46 Starting build of devel-system-x86_64-generic 2016-02-15 15:29:46 [Build 70/407] [cython.morph] Building chunk cython.morph 2016-02-15 15:29:46 [Build 70/407] [cython.morph] Creating staging area 2016-02-15 15:29:47 [Build 70/407] [cython.morph] Starting actual build: cython.morph 4dd8e76 2016-02-15 15:29:47 [Build 70/407] [cython.morph] Extracting upstream:cython into /src/tmp/staging/tmpLI9TUj/cython.morph.build Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 299, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/build_plugin.py", line 208, in build self._build(source_pool, filename, component_names=component_names) File "/src/morph/morphlib/plugins/build_plugin.py", line 237, in _build bc.build_in_order(component) File "/src/morph/morphlib/buildcommand.py", line 297, in build_in_order self.cache_or_build_source(s, build_env) File "/src/morph/morphlib/buildcommand.py", line 317, in cache_or_build_source self.build_source(source, build_env) File "/src/morph/morphlib/buildcommand.py", line 374, in build_source self.build_and_cache(staging_area, source, setup_mounts) File "/src/morph/morphlib/buildcommand.py", line 545, in build_and_cache return builder.build_and_cache(source) File "/src/morph/morphlib/builder.py", line 748, in build_and_cache built_artifacts = o.build_and_cache() File "/src/morph/morphlib/builder.py", line 288, in build_and_cache self.run_commands(temppath, stdout) File "/src/morph/morphlib/builder.py", line 314, in run_commands bs = morphlib.buildsystem.lookup_build_system(m['build-system']) File "/src/morph/morphlib/buildsystem.py", line 300, in lookup_build_system raise KeyError('Unknown build system: %s' % name) KeyError: 'Unknown build system: python3-distutils' Change-Id: Iddf651b408c612c7c80296012cf1c97ab99e8ccc
* Fix breakage when new build systems are added to DEFAULTSSam Thursfield2016-02-111-4/+8
| | | | | | | | | | | | | It turns out that one part of Morph was still using the predefined build systems that were built into the 'morphlib.buildsystem' Python module, instead of using the ones from the DEFAULTS file. This meant that Morph wasn't actually implementing Baserock definitions version 7 properly. https://storyboard.baserock.org/#!/story/72 Change-Id: I440a8f7455874a1b8491b48b0b122792aa8189ae
* More robust creation of tempdirsPedro Alvarez2016-02-051-2/+9
| | | | | | | | | | | | | | | | | | | This commit addresses the same issue that we tried to solve in fa431c3ce38204f5237337fd2045ad6269413258 but this time with temp dirs: + echo INFO: Mason building: master at fe64c00f17a4922c7499f9bc48f672cca293fc65 INFO: Mason building: master at fe64c00f17a4922c7499f9bc48f672cca293fc65 + scripts/release-build --no-default-configs --trove-host 192.168.222.58 --artifact-cache-server http://192.168.222.14:8080/ --controllers x86_64:mason-x86-64 clusters/ci.morph ERROR: /srv/distbuild/tmp/chunks: File exists ERROR: /srv/distbuild/tmp/chunks: File exists This has been happening in Mason since we remove every folder used by Morph after Mason finishes. Change-Id: I5a5bfd3c01b6fd0da4b913c5bb8eac77a3233d1e
* More robust creation of cache dirsPedro Alvarez2016-01-221-3/+18
| | | | | | | | | | | | | | | | | | When starting various distbuilds at the same time we were having randomly errors like: 2016-01-06 12:14:03 Starting distributed build 2016-01-06 12:14:03 Connecting to mason-x86-64:7878 2016-01-06 12:14:03 Requesting build of git://192.168.222.58/baserock/baserock/definitions 28b92192c00a36395acd6a960959d3b4468f9894 systems/openstack-system-x86_64.morph ERROR: /srv/distbuild/artifacts: File exists ERROR: /srv/distbuild/artifacts: File exists ERROR: /srv/distbuild/artifacts: File exists ERROR: /srv/distbuild/artifacts: File exists This has been happening in Mason since we remove every folder used by Morph after Mason finishes. Change-Id: I175de7282302d9c1e2fb1b0872f7eb72c742f28e
* Don't show progress bars if stderr is being redirected to a fileSam Thursfield2015-12-211-7/+8
| | | | | | | | | | | | | | | | | | Currently the logs of our continuous builder (https://mason-x86-64.baserock.org) are filled with lots of this sort of crap: stage1-binutils-misc[ ] 2.4/73.0 MB stage1-binutils-misc[ ] 2.4/73.0 MB stage1-binutils-misc[ ] 2.4/73.0 MB stage1-binutils-misc[ ] 2.4/73.0 MB stage1-binutils-misc[# ] 2.4/73.0 MB stage1-binutils-misc[# ] 2.5/73.0 MB stage1-binutils-misc[# ] 2.5/73.0 MB ... Change-Id: I0b7a4c8421c3ecdd141fc8323d3001ae4fb44d9b
* morphloader: remove aliasTiago Gomes2015-12-211-6/+5
| | | | | | | The field 'alias' is not defined in the schema for definitions version 7. Change-Id: Id595562258ebe4c3f228cf3f04456e7f69ce63ae
* morphloader: remove unset functionsTiago Gomes2015-12-212-150/+0
| | | | Change-Id: I2a9177245e8cd4eac54ef8a2079eaed2e98e88e7
* morphloader: remove arch normalizationTiago Gomes2015-12-212-16/+0
| | | | | | | It is a non-documented secret sauce and having to type an extra letter for the ARMv7 little-endian architecture does not justify it. Change-Id: I733d1b5a5a5a53ed4dd9815d16b946f9b0a980ec
* morphloader: remove obsolete fieldsTiago Gomes2015-12-212-160/+2
| | | | Change-Id: If7bf7af93eb404be48f752ed64e8b038a3073668
* Cleanup buildbranch.py moduleTiago Gomes2015-12-112-178/+139
| | | | | | | | | | | | | | | | Now that we don't support system branches, we don't need to iterate over a list of repos to create temporary build branches, commit their local changes and push those branches. We only need to do this for the definitions repository itself. A bug is also fixed where the local build branch was not being deleted due a missing call to _register_cleanup() when commiting the changes. This commit also renames some functions to more clear names, and moves the logic on pushed_build_branch() to the definitions_repo module, the only place where it is used. Change-Id: Id86240d0c189245bed36bc46355be13d46498dbc
* Remove (dist)?build-morphology commandsTiago Gomes2015-12-021-79/+0
| | | | Change-Id: Ifb2bc2e2084806fe70b6db96828a390d3082288f
* Fix displaying the help for extensionsTiago Gomes2015-11-261-0/+3
| | | | | | Also, add integration tests to prevent regressions. Change-Id: I80421112382168a71ca69e4bafe3c3a174e7f9ef
* Remove SystemBranchDirectory and Workspace classesTiago Gomes2015-11-254-744/+0
| | | | Change-Id: I7766d3eac28cf34aedb7f3edbc0ff3dda26ac79a
* Prepare the removal of workspace and system branches codeTiago Gomes2015-11-2510-215/+53
| | | | Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
* Cease modifying the morphologiesTiago Gomes2015-11-253-122/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | With `morph edit` removed, there is no need to load all the morphologies, check whether a chunk was `morph edited`, change the current ref to the build ref and write back the morphologies to a temporary branch. That is a lot of work as code profiling demonstrated. With this patch applied, morph execution finishes around 10 seconds sooner on my machine. This is not a big achievement when a full build is performed where the wall clock time is dictated by the actual build commands, but it will provide a much quicker feedback when the build artifacts were already cached, or the semantic validation of morphologies fail. We add the option `--untracked-files` to _get_status() in the GitIndex class so that uncommitted morphologies on a uncommitted directory are considered. Previously this was being done by calling the following call in inject_build_refs(): self._root_index.add_files_from_index_info( self._hash_morphologies(self._root, morphs.morphologies)) This commit also removes some now unused code. Change-Id: I14215db5c06ab06045ce901131e4e341271a039d
* Remove SystemMetadataDir classTiago Gomes2015-11-243-162/+0
| | | | | | This class was only used on the now removed branch-from-image command. Change-Id: I75a0b4618b16a24bd9f3b3ea7b3a6228db723715
* Remove branch-and-merge pluginTiago Gomes2015-11-241-482/+0
| | | | | | RIP. Change-Id: I6aac995415c5d67c60687367697173be52cd2bde
* Remove old cruftTiago Gomes2015-11-2013-120/+0
| | | | | | | Static analysis showed this code is not being used, and we can always bring it back if necessary through git. Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
* Fix unit testsTiago Gomes2015-11-141-10/+4
| | | | | | | Tests were broken since https://gerrit.baserock.org/#/c/1418/ was merged. Change-Id: I4e5a139d84946514634214938990cca3c86f4512
* Only run as many parallel jobs as there are available CPU coresSam Thursfield2015-11-091-4/+6
| | | | | | | YBD has been doing this for a while and seems like it is faster as a result. Change-Id: I8f95a53195cdbc2c75c06a8abe9eb089a84b1c1b
* Don't require chunks in a stratum to appear before their dependenciesSam Thursfield2015-11-052-26/+68
| | | | | | | | | | | | | | | | | | | | Currently Morph enforces that chunk A must be defined before anything that build-depends on it. YBD doesn't enforce that. Definitions format at <http://wiki.baserock.org/definitions/current/> doesn't mention ordering currently. I propose that we make Morph be permissive about ordering, like YBD is, and update the spec to mandate no restrictions on ordering. Since behaviour was previously undefined, making Morph be more permissive about this shouldn't require a new version number of the definitions format. I still think we need to make sure stratum .morph files are ordered logically, but that is in the realm of 'code style', it shouldn't be being enforced by a build tool. Change-Id: I425f2e5b9dfb62e4a26ed11f5c50e3978a0dd1a6
* Improve reliability of the staging area unit testsTiago Gomes2015-11-031-5/+5
| | | | | | | | | | | Accordingly to [1], the listdir function (called by os.walk) returns a list of filepaths on an arbitrary order. Sort the list of file paths to produce always the same list. [1]: https://docs.python.org/2/library/os.html Change-Id: I4bb9842b1722f27a8becb9c50391cda089bb0a33
* Improve get-repo error messageRichard Ipsum2015-10-221-1/+5
| | | | | | | | | | | | | | When attempting to get a repo with no commit at the ref we want, we get a 'No such file or directory' error, which is confusing, this is because we attempt to remove the directory we plan to clone into, but if there's no commit at the ref we want then we don't get as far as running git clone, so the directory doesn't exist, and we fail to remove it. This commit improves the error message by catching the InvalidRefError separately. Change-Id: Ibca20dd995d858713e56ab834daa97a8297cafc8
* Error on duplicate chunksRichard Ipsum2015-10-151-0/+32
| | | | | | | | | | | | | | This commit makes it an error for a system to contain duplicate chunk sources, example error message below, ERROR: Multiple `syslinux' chunks detected: upstream:syslinux|d715b39c0801ecea5e52f9029cea7c76320f93cf|strata/bsp-x86_both-tools/syslinux.morph|syslinux upstream:syslinux|2aab8555987b547b617cbb887e61083fece01541|strata/bsp-x86_64-generic/syslinux.morph|syslinux Multiple `nasm' chunks detected: upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_64-generic/nasm.morph|nasm upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_both-tools/nasm.morph|nasm Change-Id: I1d1539a46ce6eb098d3a559295ab9a08d6d2865c
* Add support for Baserock definitions version 7Sam Thursfield2015-10-1417-105/+416
| | | | | | | | | | | | | | | | | | | | | | This adds a new 'Defaults' class to represent definitions defaults The Python 'jsonschema' module is used to validate the contents of the Defaults file. This module is already included in Baserock 'build' and 'devel' reference systems by way of the 'openstack-common' stratum. This commit embeds a copy of the JSON-Schema schema for the DEFAULTS file. I think the canonical location of this schema should be in the reference definitions.git, for now. In future, the schemas should maybe have their own repos. Either way, Morph should embed a copy for the time being so that we are sure the schema matches how Morph expects to parse the file. Morph's automated tests are all updated to use definitions version 7. I removed most of the tests for built-in build systems, because the built-ins themselves are no longer part of Morph. Only the mechanism for defining them needs to be tested now. Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
* Ensure Git clones in a chunk build directory have correct ownershipSam Thursfield2015-10-071-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When preparing the build directory for a chunk, Morph clones the Git repo being built (and any submodules) into the staging area. Instead of using `git clone --no-hardlinks`, the morphlib.git.copy_repository() function uses 'cp -a' plus some faffing to achieve the same thing a bit faster. The 'cp -a' command will preserve the ownership and permissions of the cached copy. These may not match up with which user is doing the build. I found that files in my Git cache were owned by UID 1002. This caused chunks that needed to access the .git directory at build time to fail with strange errors. Worse, it would trigger a bug in Git[1] that leads to a fork bomb, which would cause Linux to freeze up completely. This occured even though I was building as `root`, because of the way `linux-user-chroot` drops certain privileges: presumably, setting SECBIT_NOROOT leads to CAP_DAC_OVERRIDE being unset. To avoid this bug, the code now ensures the copied .git repos are owned by the user and group who ran `morph`. Another way to fix this would be to change the morphlib.gits.copy_repository() function to use `git clone --no-hardlinks`. This is what YBD does. I found that there is a slight speed benefit to using the current code ... these are results of cloning repos from the cache with the 2 methods: With 'cp -a' plus a chown: upstream:bison: 4.54 seconds average (10 results) upstream:ybd: 0.13 seconds average (10 results) upstream:linux: 40.51 seconds average (10 results) With 'git clone --hardlinks': upstream:bison: 6.23 seconds average (10 results) upstream:ybd: 0.11 seconds average (10 results) upstream:linux: 43.36 seconds average (10 results) Test code is: <https://gist.github.com/ssssam/833e0ef8d04fb1fb6ff3>. Ideally we would fix `git clone --no-hardlinks` to be faster, but we may as well keep the existing code for the time being. [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750687 Change-Id: Ieea87322ea7b7f62975b9480f877755665656217
* Change how MorphologyLoader instances are createdSam Thursfield2015-09-2810-44/+57
| | | | | | | | | | | | | | Loading .morph files is becoming a bit more complicated, as we need to deal with the VERSION file, and possibly soon with a DEFAULTS file as well. The logic of loading and parsing .morph files is done either in the sourceresolver module, or the morphloader module. This change means that all users of the latter module can use the get hold of a MorphologyLoader instance with VERSION already parsed. If DEFAULTS is added then it is also simple to parse DEFAULTS. Change-Id: Ib33756e9dbd078e38f12dd7f776c89584b178959
* Use app.status() to log ldconfig messagesTiago Gomes2015-09-282-6/+8
| | | | | | | Using app.status() produces nicer logs as the messages will have the status prefix prepended. Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
* Replace cliapp.shell_quote() with pipes.quote()Tiago Gomes2015-09-282-2/+4
| | | | | | | | | Replace cliapp.shell_quote() with pipes.quote() to produce the chroot scripts. Despite pipes.quote() being deprecated, it produces much less escaping than cliapp.shell_quote(). This method should be replaced with shlex.quote() when migrating to Python 3. Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
* Remove edit commandRichard Ipsum2015-09-251-94/+0
| | | | Change-Id: I276a00bb9117aaa2d4fcc7e187b846eedae65591
* buildsystem.py: some autotools projects use a script called bootstrap[.sh]Javier Jardón2015-09-251-0/+2
| | | | | | ... instead the more common autogen[.sh] Change-Id: I3d797581708ffaefd1a1d4880253cfc2a3f5aa85
* Display progress bar when fetching to local cacheRichard Ipsum2015-09-232-3/+75
| | | | | | | | | Looks like, 2015-07-05 16:08:10 [Build 1/304] [stage1-binutils] Fetching to local cache: artifact stage1-binutils-misc stage1-binutils-misc[##################### ] 51.9/73.0 MB Change-Id: Ib10f1cfaa0c1df80ae605ecfeb5b706c8d46c4a4
* Remove no-longer-the-case FIXMETiago Gomes2015-09-221-3/+0
| | | | Change-Id: Iedefd1745e85e1f30d45ee8d26b315760a542a0b
* Simplify StagingArea classTiago Gomes2015-09-225-98/+54
| | | | | | | | | | | | Pass the Source to the staging area constructor so that we don't need to pass it as a parameter when we call some StagingArea methods. Also move the creation of the build and destdir directories to the constructor so that we can get rid of the chroot_open() and chroot_close() methods. Also, provide API to retrieve the relative locations for buildir and destdir. Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
* Remove support for Baserock definitions format versions 3, 4 and 5Sam Thursfield2015-09-189-812/+84
| | | | Change-Id: Iad95af65bd5c528d2e72f5b2ffa80a01152f50ff
* Remove foreach commandRichard Ipsum2015-09-171-59/+0
| | | | | | | | We're beginning a transition away from workspaces, foreach in particular is possibly one of the least used commands, and is only mentioned in tutorials explicitly marked as 'old'. Change-Id: I2c95ce689bb5550bf50feb6b55be6c58671c4c4a
* Make `check --full` pass againTiago Gomes2015-09-141-1/+1
| | | | Change-Id: I9e5b1a70946a87eb29009df2def3bd98ecc7ad2a
* Add support for Module::Build build systemRichard Ipsum2015-09-071-1/+42
| | | | | | | | ExtUtils::MakeMaker is preferred, Module::Build was meant to replace it but essentially wasn't good enough, some projects still use Module::Build though. Change-Id: I124ee7b33f32167302e9bcb5299f6422f4fc346e
* Make ExtUtilsMakeMakerBuildSystem be more standardRichard Ipsum2015-09-071-10/+19
| | | | | | Install to default locations rather than overriding. Change-Id: Ieccd06afdf623eee0d09b8d29d40d147dc8c5e7c
* Rename CPANBuildSystem to ExtUtilsMakeMakerBuildSystemRichard Ipsum2015-09-071-2/+2
| | | | | | There are several perl build systems. Change-Id: I17b5d61b7bffbdbcb9944e9e41449bec7a69c527
* Rework debugging information for failed buildsTiago Gomes2015-09-013-67/+79
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Improve error messages and remove redundant information * Stop displaying the containerised settings, as they are not very helpful to debug a build failure * Display the error code * Change system integration commands logging to consistent with build commands logging. * Provide a log file in the staging directory for each chunk build failure or system integration failure. * Provide a chroot script in the staging directory for each chunk build failure or system integration build failure. This script allows the user to enter an environment similiar to the one where the build failed. * Fix a bug where the system integration commands stdout was being omitted when these commands failed. * Fix a bug where the error_message_for_containerised_commandline() was being called with an empty error message, as the stderr was being directed to stdout. Chunk build failure output before the this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands build failed # configure # # echo "configure-commands stdout" configure-commands stdout # # echo "configure-commands stderr" >&2 configure-commands stderr # build # # echo "build commands stdout" build commands stdout # # echo "build commands stderr" >&2 build commands stderr # # echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo some more sdout some more stderr /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin sh: foo: not found ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c 'echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo ': Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball', '/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True, 'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths': ['/src/tmp/staging/tmpglc5Sz/test.build', '/src/tmp/staging/tmpglc5Sz/test.inst', '/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc', '/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz', 'cwd': '/test.build'} Error output: $> Chunk build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands ### CONFIGURE-COMMANDS ### + echo configure-commands stdout configure-commands stdout + echo configure-commands stderr configure-commands stderr ### BUILD-COMMANDS ### + echo build commands stdout build commands stdout + echo build commands stderr build commands stderr + echo some more sdout some more sdout + echo some more stderr some more stderr + echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin + foo sh: foo: not found ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi t_code=127) $> System integration build failure output before this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while running system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while building system ERROR: Command failed: baserock/system-integration/00-test-test-misc-0002: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'} Error output: + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found $> System integration build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the system integration commands + echo stdout stdout + echo stderr stderr + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration commands failed (exit_code=127) $> Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
* Stop moving staging areas of failed buildsTiago Gomes2015-09-016-32/+64
| | | | | | | | | | | | | | | | | | | | | Stop moving staging areas of failed builds from the 'staging' directory to the 'failed' directory. Moving staging areas make it very difficult to debug build failures on the build essential chunks, as the paths set on the configure scripts and some environment variables (e.g. STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area. This change will also make it easier to create scripts that chroot n environment similiar to the one where the build failure occurred. To make it still possible to safely do a build an run `morph gc` in parallel, we use flock(2) to control access to the staging area directory. Also, move the `test_supports_non_isolated_mode` test into a different class, as it requires a different SetUp() routine (the staging area is contructed with different parameters). Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
* Expose clone_into in GitDirectory classRichard Ipsum2015-08-241-0/+3
| | | | Change-Id: I2e1ae43da6e7dd0d7ddcb51d3cd9bc1794bb80a5
* Move definitions VERSION parsing code into its own definitions_version moduleSam Thursfield2015-08-185-49/+108
| | | | | | | I need to use this outside of the 'sourceresolver' module. Also, the 'sourceresolver' module is too big. Change-Id: I523bcc9555193b7369768441b72f1059e6adde5c
* Log a message if a file isn't found in definitionsSam Thursfield2015-08-181-0/+1
| | | | | | Just for future debuggers. Change-Id: Ie915a9f47fbf3ca56d6e75b816eaa5a4fb36bc88
* Use the shell X-ray modeTiago Gomes2015-08-121-5/+2
| | | | | | | | | | | Instead of logging every command for ourselves, use the shell X-ray mode. As side effect, each command argument will now be printed out in its evaluated form, which aids debugging. Also, improve the visual separation between the different build steps, and display those steps with finer granularity. Change-Id: I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8
* Fix crash if artifact doesn't contain a /baserock directorySam Thursfield2015-08-111-4/+8
| | | | | | | This is probably a sign that something is broken, but it's better to cope with it rather than crashing. Change-Id: I4e9c40d0b7db031ea271143938db61c4ad70a888
* Warn if system-integration commands are defined for an unknown artifactSam Thursfield2015-08-111-0/+7
| | | | | | | | | | | | The system-integration field in a chunk .morph file defines which of the chunk artifacts the integration script should live. If you list an artifact that doesn't exist, the integration script will not go anywhere and the commands won't run. Ideally system-integration commands will be become a bit more simple to use, but hopefully this warning helps in the meantime. Change-Id: I10a068e79eb46f98f0bd308a5caf3c445dda22ed