summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Fix: check that path to chunk morph existsbaserock-definitions-v2Richard Ipsum2015-04-111-5/+33
| | | | | | | | | | | | | | | | | | | | | This fixes a bug that allows stratum definitions to carry paths to non-existent chunk morphs. This fix changes the interpretation of definitions so necessarily introduces a new definitions version (version 2) Assuming a morph field defined in a stratum: "morph: strata/cats/xattr.morph" and assuming "strata/cats/xattr.morph" does not exist, this patch will not alter current behaviour for definitions versions 0 and 1, besides producing a warning message like this: "Warning! `strata/cats/xattr.morph' referenced in `strata/swift.morph' does not exist" for definitions version 2 and greater the following error will be raised: "ERROR: Couldn't find morphology: strata/cats/xattr.morph referenced in strata/swift.morph" Change-Id: I4def5e92741cce25168f1038136503022ab27ffd
* Make _check_version_file return versionRichard Ipsum2015-04-111-6/+16
| | | | | | Also add a distinct error message for an invalid version file Change-Id: I674e6af284b33705ae8235054906f56b8cd8a130
* Fix _env_for_arch()Paul Martin2015-04-101-1/+1
| | | | Change-Id: I1ffb63340d3facb608708d04a0a21c5a9e290c14
* Change build environment to support MIPS32/64 BE and LESimon Hoinkis2015-04-103-6/+36
| | | | Change-Id: I9344b9b80a6ec008715559390b63c9003f34bf90
* Remove description of definitions format from Morph's READMESam Thursfield2015-04-101-183/+6
| | | | | | | | | | I propose putting it here instead: <http://wiki.baserock.org/definitions/current> I've also updated a few anachronisms in the README, but it could do with further work. Change-Id: I803246c123d99990e941afa66f96ba9fd210c28e
* distbuild: Remove unneeded debugging statementSam Thursfield2015-04-091-6/+0
| | | | | | | A JsonMachine object can be set to log all messages that it sends, we don't need to handle it in the WorkerConnection class as well. Change-Id: Idfdc06953363a016708b5dda50c978eb93b1113c
* distbuild: Disable extra message debugging in worker log filesSam Thursfield2015-04-091-1/+0
| | | | | | | | | | | | | | | | | | | | | | | Worker log files are overly verbose with this enabled, each message is dumped 6 times: 2015-03-19 11:00:11 DEBUG JsonMachine: Received: '"{...}\\n"\n' 2015-03-19 11:00:11 DEBUG JsonMachine: line: '"{...}\\n"' 2015-03-19 11:00:11 DEBUG JsonRouter: got msg: {...} 2015-03-19 11:00:11 DEBUG JsonMachine: Sending message {...} 2015-03-19 11:00:11 DEBUG JsonMachine: As '"{...}\\n"' 2015-03-19 11:00:11 DEBUG JsonRouter: sent to client: {...} With this setting disabled, the message is only logged by the JsonRouter class, so appears only twice: 2015-03-19 11:00:11 DEBUG JsonRouter: got msg: {...} 2015-03-19 11:00:11 DEBUG JsonRouter: sent to client: {...} We've not seen any issues with message encoding/decoding recently so I think it's safe to disable this debugging output by default. Change-Id: I7d22ed29e81d6c594cb2c639abf3b40bfb27e3ad
* distbuild: Make 'Current jobs' log message more usefulSam Thursfield2015-04-091-2/+11
| | | | | | | | | | | | | | | | | | | It's good to know which jobs are in progress and which are queued, when reading morph-controller.log. Old output: 2015-04-09 10:40:58 DEBUG Current jobs: ['3f647933a1effbb128c857225ba77e9aa775d92314ef0acf3e58e084a7248c73.chunk.stage1-binutils-misc', 'd7279e4179a31d8a3a98c27d5b01ad1bb7387c7fab623fee1086ab68af2784bb.chunk.stage2-fhs-dirs-misc'] New output: 2015-04-09 10:40:58 DEBUG Current jobs: ['3f647933a1effbb128c857225ba77e9aa775d92314ef0acf3e58e084a7248c73.chunk.stage1-binutils-misc (given to worker1:3434)', 'd7279e4179a31d8a3a98c27d5b01ad1bb7387c7fab623fee1086ab68af2784bb.chunk.stage2-fhs-dirs-misc (given to worker2:3434)'] Change-Id: Ie89e6723b0da5f930813591a3166301fd3966804
* Make kvm deploy check that host has virt-installRichard Ipsum2015-04-071-0/+9
| | | | | | | This allows us to catch a case where virt-install hasn't been installed on the host we're deploying to much earlier in the deployment process. Change-Id: I413ad804a7e8bef4fc2d1231411e01d30d0cb9e8
* Make source resolver pass filenameRichard Ipsum2015-04-041-1/+1
| | | | | | | | | | | | | Passing the filename when loading a morphology lets us give a more useful error message: e.g. ERROR: Missing field name from morphology strata/build-essential.morph instead of ERROR: Missing field name from morphology string Change-Id: I2e5b33e8154567469aa56adbf942fb7b6e9dd1ac
* distbuild: Fix issues in build cancellationSam Thursfield2015-04-021-8/+13
| | | | | | | | | A cancel during the 'graphing' or 'annotating' stages would be ignored as the BuildController was listening for the InitiatorDisconnect message from the wrong event source. In 'building' state the actual build would be stopped, but the BuildController instance would stick around due to sending the message class instead of an instance of the message. Change-Id: I222a8aa39bf7fffab4d89e12997ffd18cd1b54fc
* Add a test for partial buildsAdam Coldrick2015-04-022-0/+17
| | | | Change-Id: Ibc95fe1d78c8d68dd20a4d2187f93f363369dfec
* Implement partial distbuildsAdam Coldrick2015-04-025-32/+120
| | | | | | | | | | | | | | In addition to partial builds we also want to be able to do partial distbuilds, and distbuild uses a different codepath. This commit updates the distbuild code to know what to do if a partial build is requested. It only builds up to the latest chunk/stratum that was requested, and displays where to find the artifacts for each of the chunks/strata requested upon completion of the build. The usage is the same as for local builds. Change-Id: I0537f74e2e65c7aefe5e71795f17999e2415fce5
* Add `morph certify` plugin to check for build reproducibility.Michael Drake2015-04-012-0/+141
| | | | | | | | | | | Currently does three checks: 1. Checks that all chunks in given system(s) have sha1 refs. 2. Checks that all sha1 refs are anchored. Unanchored refs can be removed on `git gc`. 3. Checks that all chunk repos are on the trove-host. Change-Id: Iaf105b1614a45616684e68a08f28b8529d4321fa
* Make warning messages more user friendly.Michael Drake2015-04-011-0/+7
| | | | | | | | Printing the file and line number of the warning's origin makes the warning itself harder to spot, and can make it appear as if something has actually gone wrong. Change-Id: Ie77357d9a061d31e234e6e434521d2d274409773
* Functions for checking if ref is in branches/tags of a cached repoMichael Drake2015-04-011-0/+20
| | | | Change-Id: I63f0d9da8b12930bd9539710429756a0821ce884
* Add functions for checking which tags/branches contain a refMichael Drake2015-04-012-0/+58
| | | | Change-Id: Ic6e613c21ed26c528ad7c75f41af01d7552729fd
* Improve build starting outputAdam Coldrick2015-04-011-1/+2
| | | | Change-Id: Ica2a270b254b01b9fb15956556efb74ee641e1b4
* Implement partial buildsAdam Coldrick2015-04-013-23/+94
| | | | | | | | | | | | | | | | | | This commit allows you to build only a single chunk or stratum and its dependencies, instead of a whole system. This will be useful for developers who want to quickly check the build commands of a chunk for example. You can give more than one chunk or stratum to the command, and the build will run up to the latest one, building the dependencies of both. An example of the usage: morph build systems/devel-system-x86_64-generic.morph \ strata/build-essential.morph strata/core/curl.morph This will only build up to curl in the core stratum. Change-Id: Id55b8335fc61e8fdf847f5e383cfedd13d07e6d2
* Fix: strip 'network=' from NIC_CONFIGRichard Ipsum2015-04-011-3/+11
| | | | | | Also ensure NIC_CONFIG begins with 'network=', 'bridge=' or is 'user' Change-Id: I3bcbd25eb2c9a05b7fa276697f97a1080cb0316e
* build: Clearer messages when creating temporary build branchesSam Thursfield2015-03-312-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | The temporary build branch (--local-changes=include) feature can be convenient, but it is also slow, and causes the resulting system to be 'unreproducible' (built from temporary commits). Before: $ morph build systems/devel-system-x86_64-generic.morph 2015-03-31 10:47:18 Starting build 53546647f92a4283bf96e33ea93e767e 2015-03-31 10:47:18 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master 2015-03-31 10:47:25 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph 2015-03-31 10:47:25 Deciding on task order ... After: $ morph build systems/devel-system-x86_64-generic.morph 2015-03-31 10:46:21 Looking for uncommitted changes (pass --local-changes=ignore to skip) 2015-03-31 10:46:21 Creating temporary branch in /src/ws/master/baserock/baserock/definitions named refs/heads/baserock/builds/f0b21fe240b244edb7e4142b6e201658/8df11f234ab24d22a9616ce911542332 2015-03-31 10:46:28 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph 2015-03-31 10:46:28 Deciding on task order ... Change-Id: I21a3e65c29adf0b07f743289c6b3f0f4dddc34be
* build: Print the correct name of the temporary build branchSam Thursfield2015-03-311-1/+1
| | | | | | | | | | | | | | | | | | | | Before: $ morph build systems/devel-system-x86_64-generic.morph --verbose 2015-03-31 09:35:36 new environment variable GIT_NO_REPLACE_OBJECTS = "1" 2015-03-31 09:35:36 Starting build 3fba9406f75242afbfdb413bf9658516 2015-03-31 09:35:36 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master 2015-03-31 09:25:08 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f6137060cd0> After: $ morph build systems/devel-system-x86_64-generic.morph --verbose 2015-03-31 09:35:36 new environment variable GIT_NO_REPLACE_OBJECTS = "1" 2015-03-31 09:35:36 Starting build 3fba9406f75242afbfdb413bf9658516 2015-03-31 09:35:36 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master 2015-03-31 09:35:37 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to refs/heads/baserock/builds/f0b21fe240b244edb7e4142b6e201658/8df11f234ab24d22a9616ce911542332 Change-Id: I5b61ae6f10b26458178ead551e7ec12c1926014e
* Reduce the number of 'new environment variable' logs on startupSam Thursfield2015-03-311-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reduces noise on the console when using `morph build --verbose`. Before (example from my current chroot): $ morph build systems/devel-system-x86_64-generic.morph --verbose 2015-03-31 09:25:08 new environment variable SCHROOT_UID = "0" 2015-03-31 09:25:08 new environment variable SCHROOT_CHROOT_NAME = "baserock-2015-03-24-ostree" 2015-03-31 09:25:08 new environment variable LOGNAME = "root" 2015-03-31 09:25:08 new environment variable XDG_SEAT = "seat0" 2015-03-31 09:25:08 new environment variable PATH = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" 2015-03-31 09:25:08 new environment variable XDG_VTNR = "1" 2015-03-31 09:25:08 new environment variable HOME = "/root" 2015-03-31 09:25:08 new environment variable SCHROOT_ALIAS_NAME = "baserock-2015-03-24-ostree" 2015-03-31 09:25:08 new environment variable TERM = "xterm" 2015-03-31 09:25:08 new environment variable SHELL = "/bin/bash" 2015-03-31 09:25:08 new environment variable SHLVL = "1" 2015-03-31 09:25:08 new environment variable SCHROOT_USER = "root" 2015-03-31 09:25:08 new environment variable EDITOR = "vim" 2015-03-31 09:25:08 new environment variable SCHROOT_GID = "0" 2015-03-31 09:25:08 new environment variable SCHROOT_GROUP = "root" 2015-03-31 09:25:08 new environment variable GIT_NO_REPLACE_OBJECTS = "1" 2015-03-31 09:25:08 new environment variable SCHROOT_COMMAND = "/usr/bin/env SSH_AUTH_SOCK=/tmp/tmp.eVw95CcpjS/auth-sock /bin/bash -l" 2015-03-31 09:25:08 new environment variable PYTHONPATH = "/src/morph/" 2015-03-31 09:25:08 new environment variable SSH_AUTH_SOCK = "/tmp/tmp.eVw95CcpjS/auth-sock" 2015-03-31 09:25:08 new environment variable SCHROOT_SESSION_ID = "baserock-2015-03-24-ostree-f3c36e20-c3db-4404-8808-e805ca0c1ac4" 2015-03-31 09:25:08 new environment variable MANPAGER = "less -R" 2015-03-31 09:25:08 new environment variable XDG_SESSION_ID = "2" 2015-03-31 09:25:08 new environment variable _ = "/src/morph/morph" 2015-03-31 09:25:08 new environment variable OLDPWD = "/src/morph" 2015-03-31 09:25:08 new environment variable PWD = "/src/ws/master/baserock/baserock/definitions" 2015-03-31 09:25:08 new environment variable USER = "root" 2015-03-31 09:25:08 Starting build 8f80a816dfe44f6c8352a4242b84bf48 2015-03-31 09:25:08 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master 2015-03-31 09:25:08 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f6137060cd0> 2015-03-31 09:25:15 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph ... After: $ morph build systems/devel-system-x86_64-generic.morph --verbose 2015-03-31 09:23:38 new environment variable GIT_NO_REPLACE_OBJECTS = "1" 2015-03-31 09:23:38 Starting build acfdf9a405384e1ea4e1b62cd1bce27f 2015-03-31 09:23:38 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master 2015-03-31 09:23:38 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f8fcbe86bd0> 2015-03-31 09:23:45 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph ... Change-Id: I20bbdf35b280ccd305bc7e4a70b5f3de7679a368
* Set PATH to fixed valueKejia Hu (Terry)2015-03-301-1/+1
| | | | | | | In the native build phase, building of cmake will fail if the order of pathes in $PATH is twisted. So we set it to fixed to avoid problem. Change-Id: I6b4d3ed3d5331c2db4f5f3324207fabb4310985c
* Add plugin to get details of given chunk name.Michael Drake2015-03-302-0/+80
| | | | | | | | | | | | | | | | | | | | | Usefully, this provides the resolved repo URL, which may be aliased in the actual defintions. This makes it easier to use a workflow that avoids `morph edit`. Resolved repo URLs can be git cloned. It also provides other info about the chunk. In the case of multiple strata containing the same chunk name, the tool lists them all, saying which stratum they come from. Example output: # morph get-chunk-details libpng In stratum graphics-common: Chunk: libpng Repo: git://git.baserock.org/delta/libpng Ref: 259fb7761d747655c607efcec7a12ff1f3c24561 Change-Id: I6b540e38d9521f3f473b51c9031e508ddcb458ee
* % escape all % charactersRichard Ipsum2015-03-271-1/+1
|
* Merge "Let morph recognise big-endian kernel on ARMv8"Sam Thursfield2015-03-251-1/+1
|\
| * Let morph recognise big-endian kernel on ARMv8Edward Cragg2015-03-251-1/+1
| | | | | | | | | | | | | | Let morph recognise the architecture string reported by a big-endian ARMv8 kernel. Change-Id: I64323e6c2a4675a8cc34f185b10fa04754ebe8c8
* | Merge "simple-network: only try to rename "10-dhcp.network" if it actually ↵Pedro Alvarez2015-03-251-6/+8
|\ \ | | | | | | | | | exist"
| * | simple-network: only try to rename "10-dhcp.network" if it actually existJavier Jardon2015-03-241-6/+8
| | | | | | | | | | | | Change-Id: I1521c0bdec4d7a6812f8988a2349e66b08161de8
* | | Merge "simple-network: Use function to ensure directory path will exist"Pedro Alvarez2015-03-251-0/+2
|\ \ \ | |/ /
| * | simple-network: Use function to ensure directory path will existJavier Jardon2015-03-241-0/+2
| | | | | | | | | | | | Change-Id: I44693d15aa5e92d5f09720065788adff34f8685c
* | | Merge "simple-network: Add functionn to create a path if it doesnt exist"Pedro Alvarez2015-03-251-0/+11
|\ \ \ | |/ /
| * | simple-network: Add functionn to create a path if it doesnt existJavier Jardon2015-03-241-0/+11
| | | | | | | | | | | | Change-Id: If011a5518fd30914c89b00099b9d05cff5cd3959
* | | Merge "simple-network: separate creation of directory and file path"Pedro Alvarez2015-03-251-4/+8
|\ \ \ | |/ / | | / | |/ |/|
| * simple-network: separate creation of directory and file pathJavier Jardon2015-03-241-4/+8
| | | | | | | | Change-Id: Ic715815bbad3ef1ee9ab457b62a194eaef45744c
* | Ignore files in the copyright check scriptAdam Coldrick2015-03-241-0/+4
|/ | | | | | | Add the ability to ignore files when checking copyright years, and start ignoring the COPYING file. Change-Id: I0cc11456f896e27458357ab0c5729223c71b406e
* Add COPYING with the GPLv2 licenseJavier Jardón2015-03-211-0/+339
| | | | Change-Id: I00bcedae271c20273020fa3a60bb21decde85eed
* Merge "deploy: Tighten SSH connectivity check"Javier Jardón2015-03-191-1/+5
|\
| * deploy: Tighten SSH connectivity checkSam Thursfield2015-03-191-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I accidentally tried to deploy a Baserock upgrade to a Fedora cloud machine. Every SSH command that Morph ran got the following output: Please login as the user "fedora" rather than the user "root". The existing implementation of check_ssh_connectivity() didn't raise an exception, leading to confusing errors further down. The new implementation produces this error: ERROR: Unexpected output from remote machine: Please login as the user "fedora" rather than the user "root". Change-Id: Ida5a82b25d759167aa842194b0d833d0565b4acf
* | Merge "morphlib/sourceresolver.py: parse VERSION file in a function"Javier Jardón2015-03-181-6/+22
|\ \
| * | morphlib/sourceresolver.py: parse VERSION file in a functionJavier Jardón2015-03-171-6/+22
| | | | | | | | | | | | Change-Id: I6e714d1994632875a5a15f840fe8ab3a66dddc77
* | | Merge "morphlib/sourceresolver.py: Fail if morph doesnt support the version ↵Javier Jardón2015-03-181-8/+12
|\ \ \ | |/ / | | | | | | of definitions format"
| * | morphlib/sourceresolver.py: Fail if morph doesnt support the version of ↵baserock-definitions-v1Javier Jardón2015-03-171-8/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | definitions format This patch will add the following restriction: if VERSION exist and its a YAML file and its a dict and has the key 'version' and the type stored in the 'version' key is an int and that int is not in the supported format, then fail. So, if someone is using 'version: 4' in VERSION, morph will fail (as opposed to current morph, that will not fail in the check but will likely fail when it tries to compile) Change-Id: I555f7e6018b9bdf18c80039df92a253acbd51c8c
* | | Merge branch 'baserock/richardipsum/add-hosts-conf-ext-v3'Richard Ipsum2015-03-183-18/+87
|\ \ \ | | | | | | | | | | | | | | | | | | | | Reviewed by: Richard Maw <richard.maw@gmail.com> (+2) Pedro Alvarez <pedro.alvarez@codethink.co.uk> (v2) (+1)
| * | | Add hosts.configureRichard Ipsum2015-03-181-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | This adds a new config extension to allow deployments to write to /etc/hosts by adding HOSTS_x: </etc/hosts line> to a cluster morph in a similar manner to the fstab.configure extension.
| * | | Make fstab.configure use write_from_dictRichard Ipsum2015-03-181-18/+7
| | | |
| * | | Add write_from_dict to utilRichard Ipsum2015-03-181-0/+32
|/ / / | | | | | | | | | A function to read lines from a dictionary and append them to a file
* | | scripts/test-shell.c: Add missing license headerJavier Jardón2015-03-171-0/+15
| | | | | | | | | | | | Change-Id: I46ad9c15064de752e839611d2ce3ee016ec7cc46
* | | Fix line lengths in morphlib/exts/simple-network.configureAdam Coldrick2015-03-171-5/+9
|/ / | | | | | | | | | | | | | | Some lines were more than 79 characters long. This was causing ./check --full to fail on master of morph. This commit fixes the lines in question. Change-Id: I80969d2d89d3922e021b716c250796188d7a7f4c