summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Ensure that none of the _orig_* fields are in the cache keybaserock/adamcoldrick/no-orig-fields-in-cache-keybaserock/adamcoldrick/no-fake-morphAdam Coldrick2014-08-011-1/+2
| | | | | | | | morph2.Morphology adds a number of fields beginning with _orig_, which contain default values. There is no reason for these values to be in the cache key, and this fixes a bug where distbuild created systems with a different cache key than local builds did when the configuration extensions list was empty.
* Merge branch 'baserock/richardmaw/bugfix/fix-list-artifacts'Adam Coldrick2014-07-291-1/+1
|\ | | | | | | | | Reviewed-by: Daniel Silverstone Reviewed-by: Adam Coldrick
| * Fix list-artifacts post chunks-in-definitionsbaserock/richardmaw/bugfix/fix-list-artifactsRichard Maw2014-07-281-1/+1
| | | | | | | | | | | | | | The chunks in definitions change changed the api of create_source_pool, and because list artifacts is not covered by any tests, and the create_source_pool method was confused with a function of the same name that did not need changing, it failed to be fixed.
* | Merge branch 'baserock/richardmaw/bugfix/symlink-include'Adam Coldrick2014-07-291-1/+1
|\ \ | |/ |/| | | Reviewed-by: Daniel Silverstone
| * Fix not including symlinks in chunks if they exist on the hostbaserock/richardmaw/bugfix/symlink-includeRichard Maw2014-07-291-1/+1
|/ | | | | | | | | | | | | | | | | With os.walk, if the target of the link doesn't exist, or it is a link to a file, it ends up in the basenames list. If it is a link to a directory, it goes in the subdirs list. There's a bug in the subdirsymlinks check, in that it checks if the wrong file is a symlink, so it never returns them. This was missed, since we did not have the cross bootstrap in CI. This is not eligible for our yarn tests, since to trigger this would require changes to the host system's rootfs, so it's a system-level test. To test this properly requires putting the cross bootstrap in CI.
* Merge branch 'sam/remote-git-cache-fixes'Sam Thursfield2014-07-231-1/+1
|\ | | | | | | Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * Fix an incorrect log message when using remote Git cacheSam Thursfield2014-07-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | It was referring to the remote git cache as 'remote artifact cache'. These are usually the same server (a Trove, defaulting to git.baserock.org) but can be configured independently, so they should not be considered the same. Indentally, the configuration / commandline options in question are 'git-resolve-cache-server' and 'artifact-cache-server'. Both default to 'cache-server' if not set, which in turn defaults to http://<trove-host>:8080/.
* | Merge branch 'baserock/richardmaw-os/chunks-in-defs-v3'Richard Maw2014-07-2217-306/+133
|\ \ | |/ |/| | | | | Reviewed-by: Daniel Silverstone Reviewed-by: Sam Thursfield
| * yarns: Make test-chunk use files from its source repoRichard Maw2014-07-221-3/+16
| | | | | | | | | | | | It is entirely possible that we could accidentally give chunks that use morphologies from the definitions repository, the definitions repository to build from, rather than the source repository.
| * yarns: Test building and deployment of a system with a chunk in definitionsAdam Coldrick2014-07-223-23/+26
| |
| * Rewrite traverse_morphs to be able to load chunk morphologies from definitionsRichard Maw2014-07-222-33/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | It will now load the morphology from the definitions repository if the "morph" field is present in the chunk spec. Rather than adapting the loop to fit yet-more changing circumstances, it has been partially rewritten, so there is one loop for loading systems and strata from definitions.git, another for chunks from definitions.git, and a third for chunks in the source repository. This is tidier than attempting to fit the logic in the main loop, as it makes it easier to remove afterwards when we no longer need to load chunk morphologies from the source repository.
| * Don't set 'morph' field by default, and don't expect to find itAdam Coldrick2014-07-224-6/+6
| |
| * unittests: Test that the description field is ignored for chunksAdam Coldrick2014-07-221-3/+6
| |
| * Use chunk morpholgy contents in cache keysRichard Maw2014-07-221-9/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously the contents of the morphology would be included by virtue of the fact that it came from the source repository, so would be included in the "tree" field. Now that chunk morphologies can come from the definitions repository, it is not always included in the "tree" field, so the logical contents of the morphology need to be included in the cache key computation. Build commands are included after looking them up in the build-system, so that in future, we don't need to change the chunk morphology compatibility version when we change how build-systems work. Since we may be moving the morphologies about in the definitions repository, it would suck if we had to do a full rebuild after we move things, so I dropped the filename from the cache key. This also tweaks the system and stratum cache keys to include the contents directly, rather than hashed in the "morphology-sha1" field.
| * Remove update-gits commandRichard Maw2014-07-226-222/+0
| | | | | | | | | | | | | | | | | | This hasn't been used in a long time to my knowledge, its API completely misses the point, and its implementation relies on old APIs that need to change. Until we discover we need it, and work out what it should look like, I think the best thing to do is get rid of it.
| * Remove use of "morph" field in chunk specs in yarnsRichard Maw2014-07-223-7/+6
| | | | | | | | | | | | We don't use this in definitions.git, and we're going to change its semantics, so the test suite would break until we introduced the new semantics, unless we remove its use of the old semantics first.
| * Tidy deploy_plugin a littleRichard Maw2014-07-221-5/+5
| |
| * Use run_morph in yarnsRichard Maw2014-07-221-1/+1
|/ | | | | | This is required to ensure the right version of morph is used. I have a .bashrc that causes `morph` to be "$HOME/morph/morph", so it fails to find morph, because HOME is set to a directory inside DATADIR.
* Revert distbuild parts of "Make our use of json binary path safe"baserock/richardmaw/bugfix/stop-decoding-distbuild-commsRichard Maw2014-07-183-6/+6
| | | | | | | | | | | | | | | | | | | | | | | The "unicode fix" worked for the subset of cases relevant, and only broke distbuild because its tests have not been integrated with ./check, so the fact that it broke for any string ending with a \ escaped notice, if you will excuse the pun. During json.load, the encode option is for specifying the character encoding of the file or string that is being loaded. During json.dump, the encode option is for the encoding of `str` keys and values. The fact that it worked for the set of cases we cared about is a small mystery, probably caused by the strings we happened to give it being valid unicode-escape encoded `str`ings. A full fix would require either converting all these cases to a different format, such as YAML, which will handle input data not being valid Unicode, or pre-processing the data that is passed to `json.dump` to convert all `str` instances to an appropriately escaped `unicode`, and converting back on `json.load`, but this is a quick fix to get the distbuild code working again.
* Fix JSON file handling to be binary filename safeLars Wirzenius2014-07-1510-30/+47
|\ | | | | | | | | Reviewed-by: Lars Wirzenius Reviewed-by: Pedro Alvarez
| * Allow non-unicode paths to be hardlinked into staging areasbaserock/richardmaw/bugfix/unicode-safe-pathsRichard Maw2014-07-112-8/+17
| | | | | | | | | | | | | | | | | | | | Parts of the morphology go into the name of the staging area, so it helps to convert them into a str, so later attempts to join it with another string don't result in a unicode string. pyfilesystem insists that file paths must be unicode. It is incorrect, but we passed something unicode compatible in in the first place, so we can get away with converting it back to a bytestring.
| * Make our use of json binary path safebaserock/richardmaw/bugfix/unicode-safe-jsonRichard Maw2014-07-118-22/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | json only accepts unicode. Various APIs such as file paths and environment variables allow binary data, so we need to support this properly. This patch changes every[1] use of json.load or json.dump to escape non-unicode data strings. This appears exactly as it used to if the input was valid unicode, if it isn't it will insert \xabcd escapes in the place of non-unicode data. When loading back in, if json.load is told to unescape it with `encoding='unicode-escape'` then it will convert it back correctly. This change was primarily to support file paths that weren't valid unicode, where this would choke and die. Now it works, but any tools that parsed the metadata need to unescape the paths. [1]: The interface to the remote repo cache uses json data, but I haven't changes its json.load calls to unescape the data, since the repo caches haven't been made to escape the data.
* | Fix bug in list-artifacts pluginRichard Ipsum2014-07-111-1/+1
| |
* | Log the address we attempt to bindRichard Ipsum2014-07-111-0/+1
|/ | | | | This will make it easier to determine what is wrong if the controller daemon is run with a bad controller host address.
* Merge remote-tracking branch ↵Richard Maw2014-07-1162-5495/+2626
|\ | | | | | | | | | | | | 'origin/baserock/richardmaw/S11284/morphologies-by-path-v4' Reviewed-by: Sam Thursfield Reviewed-by: Lars Wirzenius
| * yarns: Adapt to put morphologies in subdirsbaserock/richardmaw/S11284/morphologies-by-path-v4Richard Maw2014-07-108-195/+191
| | | | | | | | | | | | | | | | | | | | | | | | | | We want to move our morphologies in our definitions repository into subdirectories, so they're more organised. We'd prefer to only refer to morphologies by file path, rather than a name that loosely corresponds to the file path, but we need to support that for backwards compatibility until we can move all of our morphologies into the definitions repository. However, since we want to eventually remove this, and we want to ensure that file paths work, we change the yarn tests to use file paths.
| * Don't check if a file exists before trying to read itRichard Maw2014-07-101-28/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We used to check whether a file existed before trying to read it. We used to be able to get away with only looking at the top-level directory, which made using ls-files before trying to cat-file it better. Unfortunately, we need to look at the files in subdirectories now, so this no longer works. We could make it include files in subdirectories, but for repositories with many files, you would end up reading a file listing longer than the morphology, so even in the slow case of needing to read the entire morphology file, it would be faster to attempt to read the file first. So now we beg forgiveness rather than asking permission.
| * Use exact filenames to refer to morphology filesRichard Maw2014-07-1014-1757/+1768
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rather than repeatedly stripping and appending an optional .morph extension morphology names, instead always use the file path of the morphology relative to the definitions repository. This is an inversion of the previous logic, which would strip the .morph extension and use the "name" internally. The exception to this rule of always using the filename, is that `morph edit CHUNK` uses the name of the morphology as-defined in the stratum. This is based off Adam Coldrick's inital patch, but this version will allow the old style of providing the "name" by converting it into a path if it does not have either a / or a . in it. An unfortunate consequence of this change is that the show-dependencies command's output changed, so the test needed updating.
| * Don't attempt to load symlinks as morphologiesRichard Maw2014-07-073-1/+49
| | | | | | | | | | | | | | | | | | | | | | This was noticed because our definitions.git had a dangling symlink, so it failed to construct the temporary build branch. We shouldn't process symlinks as morphologies either, since either we make symlinked morphologies a first-class behaviour, and validate that the link points inside the repository, which is a lot of work for something we don't and probably won't need; or we can just assume that we deal with the morphology this is linked to correctly anyway.
| * Make MorphologyFinder use file pathsRichard Maw2014-07-074-28/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We want to use file paths to locate morphologies now, so the old model of get a list of names and hand it those back to get the contents doesn't really make sense any more. This abstraction initially came about as one idea I had for moving morphologies out of the source tree was to put them in something like git notes, where it's possible to look up information for one commit in another ref in the repository, at which point this abstraction would have been flexible enough to handle that as well as in the However, moving the chunk morphologies into the definitions repository has other benefits too, so it makes more sense to be honest about using filenames in the API. It remains as a single point where we can put the logic for knowing which files in a repository look like morphologies, but if we need to remove any further functionality, it should be replaced by a single function.
| * Add class hierarchies for Morphology load errorsRichard Maw2014-07-071-28/+36
| | | | | | | | | | There's a lot of them, it's too much of a pain to enumerate them all, so it's convenient to provide a hierachy and catch the base exceptions.
| * Exorcise some old and unused commandsRichard Maw2014-07-0736-3309/+536
| | | | | | | | | | | | | | | | | | | | | | | | | | `morph merge` only worked for a small subset of cases, and has been left to bit-rot, since we don't use it. `morph tag` is just a `git tag` when we have petrified definitions repository. We don't use it, nor do we need it, so it can go away rather than take up valuable development time fixing it when requirements change. `old-foo` have all been superceded by newer versions and are no-longer used.
| * Remove check on checkout/branch that there are systemsRichard Maw2014-07-072-31/+0
| | | | | | | | | | | | | | This was the wrong response to the problem of accidentally checking-out morph when trying to check out morphs. Now that it's called definitions, this is irrelevent, and aborting a checkout when this check fails is the wrong thing to do.
| * Remove code made vestigial from the `morph edit` rewriteRichard Maw2014-07-073-119/+1
| | | | | | | | | | | | This was originally used by `morph edit`, but since we removed the need to run `morph edit system stratum` and could shorten it to `morph edit chunk`, this function is no longer used.
* | Revert "Revert "Update NEWS for release 14.28.""baserock-14.28Michael Drake2014-07-111-0/+6
| | | | | | | | This reverts commit ad65450c145e3283f95a40397280fc8b938a04c0.
* | Revert "Update NEWS for release 14.28."Daniel Silverstone2014-07-091-6/+0
| | | | | | | | This reverts commit a6488c42e79177576d8c73927dac7c1e86311122.
* | Update NEWS for release 14.28.Michael Drake2014-07-091-0/+6
| |
* | Merge branch 'sam/splitting-fixes'Sam Thursfield2014-07-071-10/+14
|\ \ | | | | | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * | Rewrite SplitRules.partition() docstring to be clearer to meSam Thursfield2014-07-041-3/+4
| | |
| * | Fix issue when overriding default chunk splitting rulesSam Thursfield2014-07-041-7/+10
| |/ | | | | | | | | | | | | | | | | | | | | When overriding a default chunk splitting rule, the use 'break' would mean that some of the other default chunk split rules might be ignored. This isn't ever what you want. I also clarified the current behaviour in a comment. I think in future we should add a mechanism to extend the default rules, as well as the current mechanism which allows only replacing them. But that is a separate issue.
* | Merge branch 'sam/update-gits-fix'Sam Thursfield2014-07-073-13/+46
|\ \ | |/ |/| | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
| * Fix Morph failing to update some cached git reposSam Thursfield2014-07-073-13/+46
|/ | | | | | | | | | | | | | | | | | | | | | | I was getting the following error when running the 'do-release.py' script: ERROR:root:Command failed: morph --quiet --trove-host=git.baserock.org list-artifacts baserock:baserock/definitions sam/auto-release minimal-system-x86_32-generic ERROR: Ref d67a0e110187abd560a1de63fa172894a52839d5 is an invalid reference for repo git://git.baserock.org/delta/linux The commit that it wants did actually exist in git.baserock.org and the logs show that Morph hadn't done a `git remote update` to get the commit locally. This turned out to be because it'd had already looked up a different ref in linux.git. It hadn't needed to run 'git remote update', but it *had* added linux.git to a "don't need to update this repo again" list. Oops! I've moved the code in question to the cachedrepo module so that the repo object keeps that of whether it should be updated. The bug is now fixed. As a side effect this patch fixes the spurious 'Updating repo file:///...' messages, which were printed even though repos with file:/// URLs are never updated.
* Update NEWS filebaserock-14.26Richard Ipsum2014-06-271-0/+9
|
* Merge remote-tracking branch ↵Lars Wirzenius2014-06-262-31/+37
|\ | | | | | | | | | | | | 'origin/baserock/richardipsum/improve_getmorph_errmsg' Reviewed-by: Lars Wirzenius Reviewed-by: Emmet Hikory
| * Make morph factory tests expect MorphologyNotFoundErrorRichard Ipsum2014-06-251-6/+5
| |
| * Replace AutodetectError with MorphologyNotFoundErrorRichard Ipsum2014-06-251-24/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The error message: ERROR: Failed to determine the build system of repo file://foo/bar/baz at ref 59713cf997385a094091443fdcce9d5c17313f39: was looking for distbuild-system-x86-64.morph is confusing since our system morph has nothing to do with build systems, the fact that build system autodetection is executed when looking for distbuild-system-x86-64.morph is an implementation detail that shouldn't be exposed to the user. This patch replaces this error message with: ERROR: Couldn't find morphology: distbuild-system-x86-64.morph This is still not ideal, since there are cases where we may not be able find the morph because build system autodection has failed, but out of the two user typos/mistakes are probably more likely. Differentiating between user error and build system detection failure would require more substantial changes. This patch also splits the get_morphology function into two functions _get_morphology_text, that actually fetches the morph text from the cache or otherwise infers it from the build system, and get_morphology which constructs a Morphology object from the text.
| * Add chatty status messages to aid the userRichard Ipsum2014-06-251-5/+12
|/
* Improve var names in morphloaderRichard Ipsum2014-06-241-24/+22
|
* Fix duplicate names error messageRichard Ipsum2014-06-241-5/+6
| | | | | This error message didn't report the cluster morph at fault. We also fix a minor formatting issue.
* Merge branch 'sam/distbuild-busyloop-debug'Sam Thursfield2014-06-201-0/+4
|\ | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>