summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* WIP Add support for multiple sources per chunkbaserock/tiagogomes/multiple-sourcesTiago Gomes2015-11-2019-71/+401
| | | | | | | | | TODO: - Add API to the cache server to retrieve a submodule commit from a given path. - Fix cross-bootstrap command. Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
* Improve status messagesTiago Gomes2015-11-202-5/+6
| | | | Change-Id: I3ecd315df73b74177dd42c424be0250def65a2b4
* morphloader: invert repo/name logicTiago Gomes2015-11-202-10/+10
| | | | | | | Getting the `repo` from the given `name` will most likely result on a not valid git URL. Change-Id: Iff6cb5abe5a32419a290eca28df4ca2d1f0182aa
* morphloader: add missing validation to stratum chunksTiago Gomes2015-11-202-37/+66
| | | | Change-Id: I7268b732ab0e37a198cd0484f491334ff456de2b
* morphloader: remove arch normalizationTiago Gomes2015-11-202-17/+1
| | | | | | Remove this secret sauce that appears to not be documented anywhere. Change-Id: I733d1b5a5a5a53ed4dd9815d16b946f9b0a980ec
* morphloader: remove duplicated APITiago Gomes2015-11-203-37/+23
| | | | Change-Id: Idbbd5a08e8b16d8e01bb9539274092978b64f6f0
* morphloader: remove obsolete fieldsTiago Gomes2015-11-202-160/+2
| | | | | | The older definitions version supported by Morph (6) does not uses them. Change-Id: If7bf7af93eb404be48f752ed64e8b038a3073668
* Fix unit testsTiago Gomes2015-11-141-10/+4
| | | | | | | Tests were broken since https://gerrit.baserock.org/#/c/1418/ was merged. Change-Id: I4e5a139d84946514634214938990cca3c86f4512
* Only run as many parallel jobs as there are available CPU coresSam Thursfield2015-11-091-4/+6
| | | | | | | YBD has been doing this for a while and seems like it is faster as a result. Change-Id: I8f95a53195cdbc2c75c06a8abe9eb089a84b1c1b
* Don't require chunks in a stratum to appear before their dependenciesSam Thursfield2015-11-052-26/+68
| | | | | | | | | | | | | | | | | | | | Currently Morph enforces that chunk A must be defined before anything that build-depends on it. YBD doesn't enforce that. Definitions format at <http://wiki.baserock.org/definitions/current/> doesn't mention ordering currently. I propose that we make Morph be permissive about ordering, like YBD is, and update the spec to mandate no restrictions on ordering. Since behaviour was previously undefined, making Morph be more permissive about this shouldn't require a new version number of the definitions format. I still think we need to make sure stratum .morph files are ordered logically, but that is in the realm of 'code style', it shouldn't be being enforced by a build tool. Change-Id: I425f2e5b9dfb62e4a26ed11f5c50e3978a0dd1a6
* Improve reliability of the staging area unit testsTiago Gomes2015-11-031-5/+5
| | | | | | | | | | | Accordingly to [1], the listdir function (called by os.walk) returns a list of filepaths on an arbitrary order. Sort the list of file paths to produce always the same list. [1]: https://docs.python.org/2/library/os.html Change-Id: I4bb9842b1722f27a8becb9c50391cda089bb0a33
* Improve get-repo error messageRichard Ipsum2015-10-221-1/+5
| | | | | | | | | | | | | | When attempting to get a repo with no commit at the ref we want, we get a 'No such file or directory' error, which is confusing, this is because we attempt to remove the directory we plan to clone into, but if there's no commit at the ref we want then we don't get as far as running git clone, so the directory doesn't exist, and we fail to remove it. This commit improves the error message by catching the InvalidRefError separately. Change-Id: Ibca20dd995d858713e56ab834daa97a8297cafc8
* Error on duplicate chunksRichard Ipsum2015-10-151-0/+32
| | | | | | | | | | | | | | This commit makes it an error for a system to contain duplicate chunk sources, example error message below, ERROR: Multiple `syslinux' chunks detected: upstream:syslinux|d715b39c0801ecea5e52f9029cea7c76320f93cf|strata/bsp-x86_both-tools/syslinux.morph|syslinux upstream:syslinux|2aab8555987b547b617cbb887e61083fece01541|strata/bsp-x86_64-generic/syslinux.morph|syslinux Multiple `nasm' chunks detected: upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_64-generic/nasm.morph|nasm upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_both-tools/nasm.morph|nasm Change-Id: I1d1539a46ce6eb098d3a559295ab9a08d6d2865c
* Add support for Baserock definitions version 7Sam Thursfield2015-10-1432-142/+518
| | | | | | | | | | | | | | | | | | | | | | This adds a new 'Defaults' class to represent definitions defaults The Python 'jsonschema' module is used to validate the contents of the Defaults file. This module is already included in Baserock 'build' and 'devel' reference systems by way of the 'openstack-common' stratum. This commit embeds a copy of the JSON-Schema schema for the DEFAULTS file. I think the canonical location of this schema should be in the reference definitions.git, for now. In future, the schemas should maybe have their own repos. Either way, Morph should embed a copy for the time being so that we are sure the schema matches how Morph expects to parse the file. Morph's automated tests are all updated to use definitions version 7. I removed most of the tests for built-in build systems, because the built-ins themselves are no longer part of Morph. Only the mechanism for defining them needs to be tested now. Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
* distbuild: When a build finishes, say which worker it was built onSam Thursfield2015-10-075-6/+14
| | | | Change-Id: I493fced8cf2664283923f6f41097ca991d3fc3de
* Remove some unused Yarn IMPLEMENTS linesSam Thursfield2015-10-071-55/+0
| | | | | | | | Some tests were removed in commit f4d3cbaf1d6e5c3bdbb1b2e2 but it seems there are still code remnants that we don't need any more. There may be more -- I haven't done a thorough job of this. Change-Id: I0d7151402365f9e6f23a93a1d30df0198de9ef02
* Ensure Git clones in a chunk build directory have correct ownershipSam Thursfield2015-10-071-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When preparing the build directory for a chunk, Morph clones the Git repo being built (and any submodules) into the staging area. Instead of using `git clone --no-hardlinks`, the morphlib.git.copy_repository() function uses 'cp -a' plus some faffing to achieve the same thing a bit faster. The 'cp -a' command will preserve the ownership and permissions of the cached copy. These may not match up with which user is doing the build. I found that files in my Git cache were owned by UID 1002. This caused chunks that needed to access the .git directory at build time to fail with strange errors. Worse, it would trigger a bug in Git[1] that leads to a fork bomb, which would cause Linux to freeze up completely. This occured even though I was building as `root`, because of the way `linux-user-chroot` drops certain privileges: presumably, setting SECBIT_NOROOT leads to CAP_DAC_OVERRIDE being unset. To avoid this bug, the code now ensures the copied .git repos are owned by the user and group who ran `morph`. Another way to fix this would be to change the morphlib.gits.copy_repository() function to use `git clone --no-hardlinks`. This is what YBD does. I found that there is a slight speed benefit to using the current code ... these are results of cloning repos from the cache with the 2 methods: With 'cp -a' plus a chown: upstream:bison: 4.54 seconds average (10 results) upstream:ybd: 0.13 seconds average (10 results) upstream:linux: 40.51 seconds average (10 results) With 'git clone --hardlinks': upstream:bison: 6.23 seconds average (10 results) upstream:ybd: 0.11 seconds average (10 results) upstream:linux: 43.36 seconds average (10 results) Test code is: <https://gist.github.com/ssssam/833e0ef8d04fb1fb6ff3>. Ideally we would fix `git clone --no-hardlinks` to be faster, but we may as well keep the existing code for the time being. [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750687 Change-Id: Ieea87322ea7b7f62975b9480f877755665656217
* Change how MorphologyLoader instances are createdSam Thursfield2015-09-2815-45/+73
| | | | | | | | | | | | | | Loading .morph files is becoming a bit more complicated, as we need to deal with the VERSION file, and possibly soon with a DEFAULTS file as well. The logic of loading and parsing .morph files is done either in the sourceresolver module, or the morphloader module. This change means that all users of the latter module can use the get hold of a MorphologyLoader instance with VERSION already parsed. If DEFAULTS is added then it is also simple to parse DEFAULTS. Change-Id: Ib33756e9dbd078e38f12dd7f776c89584b178959
* Use app.status() to log ldconfig messagesTiago Gomes2015-09-282-6/+8
| | | | | | | Using app.status() produces nicer logs as the messages will have the status prefix prepended. Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
* Replace cliapp.shell_quote() with pipes.quote()Tiago Gomes2015-09-282-2/+4
| | | | | | | | | Replace cliapp.shell_quote() with pipes.quote() to produce the chroot scripts. Despite pipes.quote() being deprecated, it produces much less escaping than cliapp.shell_quote(). This method should be replaced with shlex.quote() when migrating to Python 3. Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
* Remove edit commandRichard Ipsum2015-09-2516-515/+0
| | | | Change-Id: I276a00bb9117aaa2d4fcc7e187b846eedae65591
* buildsystem.py: some autotools projects use a script called bootstrap[.sh]Javier Jardón2015-09-251-0/+2
| | | | | | ... instead the more common autogen[.sh] Change-Id: I3d797581708ffaefd1a1d4880253cfc2a3f5aa85
* Remove some tests which dependended on `morph foreach`Sam Thursfield2015-09-234-120/+0
| | | | | | | | | | Commit a4d8098b229592db40565747b0444b518bf8a6eb removed the `morph foreach` command, which broke some `morph edit` tests. I don't think we are ready to remove `morph edit` yet (documentation still mentions it) but we can remove these tests, which require `morph foreach` to work. Change-Id: Id372760f69f31a46d1d872ee36ec6fdfb324ce34
* Display progress bar when fetching to local cacheRichard Ipsum2015-09-232-3/+75
| | | | | | | | | Looks like, 2015-07-05 16:08:10 [Build 1/304] [stage1-binutils] Fetching to local cache: artifact stage1-binutils-misc stage1-binutils-misc[##################### ] 51.9/73.0 MB Change-Id: Ib10f1cfaa0c1df80ae605ecfeb5b706c8d46c4a4
* Remove no-longer-the-case FIXMETiago Gomes2015-09-221-3/+0
| | | | Change-Id: Iedefd1745e85e1f30d45ee8d26b315760a542a0b
* Simplify StagingArea classTiago Gomes2015-09-225-98/+54
| | | | | | | | | | | | Pass the Source to the staging area constructor so that we don't need to pass it as a parameter when we call some StagingArea methods. Also move the creation of the build and destdir directories to the constructor so that we can get rid of the chroot_open() and chroot_close() methods. Also, provide API to retrieve the relative locations for buildir and destdir. Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
* Remove support for Baserock definitions format versions 3, 4 and 5Sam Thursfield2015-09-1836-2990/+1808
| | | | Change-Id: Iad95af65bd5c528d2e72f5b2ffa80a01152f50ff
* Remove foreach commandRichard Ipsum2015-09-177-122/+0
| | | | | | | | We're beginning a transition away from workspaces, foreach in particular is possibly one of the least used commands, and is only mentioned in tutorials explicitly marked as 'old'. Change-Id: I2c95ce689bb5550bf50feb6b55be6c58671c4c4a
* Make anchor yarn use defsrepo instead of workspaceRichard Ipsum2015-09-172-19/+36
| | | | Change-Id: Icec7e948c60decefe5a8bab6141bc89603c265b7
* Remove yarns for functionality we no longer wantRichard Ipsum2015-09-171-471/+0
| | | | Change-Id: I6c45763fb4d9f304695c1894625ced36bda36f89
* Force test suite to observe morph.conf tempdirMike Williams2015-09-174-3/+31
| | | | | | | | | This change, suggested in story #16, causes the testing suite to use the user-defined tempdir in /etc/morph.conf if it exists or continue using /tmp if not. This will prevent the test suite from filling drives that are not intended to be used for development work. Change-Id: I94edabe4ea1eca7a55ece7424ece7bbd4b68c513
* Make `check --full` pass againTiago Gomes2015-09-141-1/+1
| | | | Change-Id: I9e5b1a70946a87eb29009df2def3bd98ecc7ad2a
* Add support for Module::Build build systemRichard Ipsum2015-09-071-1/+42
| | | | | | | | ExtUtils::MakeMaker is preferred, Module::Build was meant to replace it but essentially wasn't good enough, some projects still use Module::Build though. Change-Id: I124ee7b33f32167302e9bcb5299f6422f4fc346e
* Make ExtUtilsMakeMakerBuildSystem be more standardRichard Ipsum2015-09-071-10/+19
| | | | | | Install to default locations rather than overriding. Change-Id: Ieccd06afdf623eee0d09b8d29d40d147dc8c5e7c
* Rename CPANBuildSystem to ExtUtilsMakeMakerBuildSystemRichard Ipsum2015-09-071-2/+2
| | | | | | There are several perl build systems. Change-Id: I17b5d61b7bffbdbcb9944e9e41449bec7a69c527
* Rework debugging information for failed buildsTiago Gomes2015-09-014-68/+80
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Improve error messages and remove redundant information * Stop displaying the containerised settings, as they are not very helpful to debug a build failure * Display the error code * Change system integration commands logging to consistent with build commands logging. * Provide a log file in the staging directory for each chunk build failure or system integration failure. * Provide a chroot script in the staging directory for each chunk build failure or system integration build failure. This script allows the user to enter an environment similiar to the one where the build failed. * Fix a bug where the system integration commands stdout was being omitted when these commands failed. * Fix a bug where the error_message_for_containerised_commandline() was being called with an empty error message, as the stderr was being directed to stdout. Chunk build failure output before the this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands build failed # configure # # echo "configure-commands stdout" configure-commands stdout # # echo "configure-commands stderr" >&2 configure-commands stderr # build # # echo "build commands stdout" build commands stdout # # echo "build commands stderr" >&2 build commands stderr # # echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo some more sdout some more stderr /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin sh: foo: not found ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c 'echo "some more sdout" echo "some more stderr" >&2 echo $PATH foo ': Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball', '/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True, 'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths': ['/src/tmp/staging/tmpglc5Sz/test.build', '/src/tmp/staging/tmpglc5Sz/test.inst', '/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc', '/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz', 'cwd': '/test.build'} Error output: $> Chunk build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands ### CONFIGURE-COMMANDS ### + echo configure-commands stdout configure-commands stdout + echo configure-commands stderr configure-commands stderr ### BUILD-COMMANDS ### + echo build commands stdout build commands stdout + echo build commands stderr build commands stderr + echo some more sdout some more sdout + echo some more stderr some more stderr + echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin + foo sh: foo: not found ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi t_code=127) $> System integration build failure output before this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while running system integration commands 2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while building system ERROR: Command failed: baserock/system-integration/00-test-test-misc-0002: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'} Error output: + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found $> System integration build failure output after this change and change I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8: 2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the system integration commands + echo stdout stdout + echo stderr stderr + foo baserock/system-integration/00-test-test-misc-0002: line 1: foo: not found ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration commands failed (exit_code=127) $> Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
* Stop moving staging areas of failed buildsTiago Gomes2015-09-016-32/+64
| | | | | | | | | | | | | | | | | | | | | Stop moving staging areas of failed builds from the 'staging' directory to the 'failed' directory. Moving staging areas make it very difficult to debug build failures on the build essential chunks, as the paths set on the configure scripts and some environment variables (e.g. STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area. This change will also make it easier to create scripts that chroot n environment similiar to the one where the build failure occurred. To make it still possible to safely do a build an run `morph gc` in parallel, we use flock(2) to control access to the staging area directory. Also, move the `test_supports_non_isolated_mode` test into a different class, as it requires a different SetUp() routine (the staging area is contructed with different parameters). Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
* Expose clone_into in GitDirectory classRichard Ipsum2015-08-241-0/+3
| | | | Change-Id: I2e1ae43da6e7dd0d7ddcb51d3cd9bc1794bb80a5
* Move definitions VERSION parsing code into its own definitions_version moduleSam Thursfield2015-08-185-49/+108
| | | | | | | I need to use this outside of the 'sourceresolver' module. Also, the 'sourceresolver' module is too big. Change-Id: I523bcc9555193b7369768441b72f1059e6adde5c
* Log a message if a file isn't found in definitionsSam Thursfield2015-08-181-0/+1
| | | | | | Just for future debuggers. Change-Id: Ie915a9f47fbf3ca56d6e75b816eaa5a4fb36bc88
* Use the shell X-ray modeTiago Gomes2015-08-125-19/+19
| | | | | | | | | | | Instead of logging every command for ourselves, use the shell X-ray mode. As side effect, each command argument will now be printed out in its evaluated form, which aids debugging. Also, improve the visual separation between the different build steps, and display those steps with finer granularity. Change-Id: I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8
* Fix crash if artifact doesn't contain a /baserock directorySam Thursfield2015-08-111-4/+8
| | | | | | | This is probably a sign that something is broken, but it's better to cope with it rather than crashing. Change-Id: I4e9c40d0b7db031ea271143938db61c4ad70a888
* Fix Yarn tests to not include .git directory in built artifactsSam Thursfield2015-08-111-1/+6
| | | | | | | | | When I spotted that artifacts contained a .git directory I was worried that it might indicate a Morph bug. It turned out to be due to the way the 'test shell' used in the Yarn tests works. This change should save anyone else from similar confusion. Change-Id: I97b57eff7aa947506d4062a76e9803398e46c2b6
* Warn if system-integration commands are defined for an unknown artifactSam Thursfield2015-08-111-0/+7
| | | | | | | | | | | | The system-integration field in a chunk .morph file defines which of the chunk artifacts the integration script should live. If you list an artifact that doesn't exist, the integration script will not go anywhere and the commands won't run. Ideally system-integration commands will be become a bit more simple to use, but hopefully this warning helps in the meantime. Change-Id: I10a068e79eb46f98f0bd308a5caf3c445dda22ed
* Set chunk commands' stdin to /dev/nullRichard Ipsum2015-08-071-7/+9
| | | | | | | Chunk commands will currently block if they read from stdin. To fix this we set stdin to /dev/null so that the first read returns EOF. Change-Id: I143396f4c443c098b2880c5925e6f0907cb4fb17
* Print warning messages nicelySam Thursfield2015-07-301-1/+1
| | | | | | They were all run onto the same line before with no linebreaks. Change-Id: Ibdff91a23221034a3d345542f1268f8863cf4515
* Add support for Baserock definitions version 6Sam Thursfield2015-07-295-100/+150
| | | | Change-Id: I891d1b13ed0581b293fe6b09b3cc73af8fd81d67
* Remove support for Baserock definitions format versions 0, 1 and 2Sam Thursfield2015-07-236-63/+13
| | | | | | | | | | | The previous release of the Baserock reference systems (baserock-15.25 tag) used Baserock definitions version 3, so this version of Morph can still build the last release of the Baserock reference systems. In the course of fixing up the cmdtest tests to use version 5 definitions, I removed a couple of bits of redundant code. Change-Id: If9d480aa966d2ea5939cf04351f3d488cccea67f
* Don't provide default strip-commands for 'manual' build systemSam Thursfield2015-07-231-1/+6
| | | | | | | | | | | | | | | | | | | | | This makes more sense to me conceptually: the idea of the 'manual' build system is you specify *everything* manually. However, I made this change to work around a problem in the Yarn test suite. If the definitions used for testing are set to version 5 or newer, `morph build` will try to run 'strip' and 'objcopy', which fails because they don't exist in the test definitions. We could solve this by setting 'strip-commands: []' in the test chunk .morph files, but this triggers an error further down where `morph edit` fails with 'Field strip-commands not allowed in morphology test-chunk.morph'. This bug is caused by the conditional way that strip-commands are currently implemented in Morph. When we drop support for versions of the definitions format older than V5, the bug will go away, and the `morph edit` command is deprecated in any case, so I think it makes sense to work around it instead of trying to fix it. Change-Id: Ib5124d72466a77cf3f28ed3e14cc4c231bdce4c4
* Extract downloaded tarball as current userRichard Ipsum2015-07-201-1/+1
| | | | | | | | | | | | | | | | | | tar will preserve ownership and permissions by default if running as the superuser, whereas git doesn't track ownership, the owner is set to the user running the clone. This is a reproducibility hole since the resulting repo may differ depending on whether the repo was obtained from a tarball or via a git clone, but morph considers the repos to be equivalent. Running tar with --no-same-owner ensures the files written by tar will be owned by the user extracting the archive. This only fixes half the problem, lorry may also need to be modified to ensure files in the tarball have the same mode as those in the git repo. Change-Id: I849d054bc4574f9c0bbcfd608914306e52b7a22e
View Lorry Controller Status