summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
* StagingArea: Ensure staging are paths are bytestringsbaserock/richardmaw/bugfix/distbuild-eglibcRichard Maw2014-07-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | If the destdir path returned when creating a staging area is a unicode string, then when attempting to `os.walk(destdir)`, it will encounter unicode errors if there are file paths in the destdir that are not representable as unicode strings. For various as-yet unknown reasons, when building stage-2 eglibc it produces file paths that are not unicode compatible. There was previously a patch to fix this issue with regards to creating the metadata files, but it did not fix all the issues, because the build at the time was local rather than distributed. This is failing during a distributed build because morphologies are serialised into json, and during deserialisation their string values are left as unicode. Rather than doing the byte-string conversion during deserialisation, I have chosen to do it when the contents of the morphology are used, because it's only at the point where it's used to create a file path, that it matters whether it's unicode or not.
* Fix JSON file handling to be binary filename safeLars Wirzenius2014-07-156-22/+39
|\ | | | | | | | | Reviewed-by: Lars Wirzenius Reviewed-by: Pedro Alvarez
| * Allow non-unicode paths to be hardlinked into staging areasbaserock/richardmaw/bugfix/unicode-safe-pathsRichard Maw2014-07-112-8/+17
| | | | | | | | | | | | | | | | | | | | Parts of the morphology go into the name of the staging area, so it helps to convert them into a str, so later attempts to join it with another string don't result in a unicode string. pyfilesystem insists that file paths must be unicode. It is incorrect, but we passed something unicode compatible in in the first place, so we can get away with converting it back to a bytestring.
| * Make our use of json binary path safebaserock/richardmaw/bugfix/unicode-safe-jsonRichard Maw2014-07-114-14/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | json only accepts unicode. Various APIs such as file paths and environment variables allow binary data, so we need to support this properly. This patch changes every[1] use of json.load or json.dump to escape non-unicode data strings. This appears exactly as it used to if the input was valid unicode, if it isn't it will insert \xabcd escapes in the place of non-unicode data. When loading back in, if json.load is told to unescape it with `encoding='unicode-escape'` then it will convert it back correctly. This change was primarily to support file paths that weren't valid unicode, where this would choke and die. Now it works, but any tools that parsed the metadata need to unescape the paths. [1]: The interface to the remote repo cache uses json data, but I haven't changes its json.load calls to unescape the data, since the repo caches haven't been made to escape the data.
* | Fix bug in list-artifacts pluginRichard Ipsum2014-07-111-1/+1
|/
* Merge remote-tracking branch ↵Richard Maw2014-07-1121-2745/+756
|\ | | | | | | | | | | | | 'origin/baserock/richardmaw/S11284/morphologies-by-path-v4' Reviewed-by: Sam Thursfield Reviewed-by: Lars Wirzenius
| * Don't check if a file exists before trying to read itRichard Maw2014-07-101-28/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We used to check whether a file existed before trying to read it. We used to be able to get away with only looking at the top-level directory, which made using ls-files before trying to cat-file it better. Unfortunately, we need to look at the files in subdirectories now, so this no longer works. We could make it include files in subdirectories, but for repositories with many files, you would end up reading a file listing longer than the morphology, so even in the slow case of needing to read the entire morphology file, it would be faster to attempt to read the file first. So now we beg forgiveness rather than asking permission.
| * Use exact filenames to refer to morphology filesRichard Maw2014-07-1013-78/+89
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rather than repeatedly stripping and appending an optional .morph extension morphology names, instead always use the file path of the morphology relative to the definitions repository. This is an inversion of the previous logic, which would strip the .morph extension and use the "name" internally. The exception to this rule of always using the filename, is that `morph edit CHUNK` uses the name of the morphology as-defined in the stratum. This is based off Adam Coldrick's inital patch, but this version will allow the old style of providing the "name" by converting it into a path if it does not have either a / or a . in it. An unfortunate consequence of this change is that the show-dependencies command's output changed, so the test needed updating.
| * Don't attempt to load symlinks as morphologiesRichard Maw2014-07-073-1/+49
| | | | | | | | | | | | | | | | | | | | | | This was noticed because our definitions.git had a dangling symlink, so it failed to construct the temporary build branch. We shouldn't process symlinks as morphologies either, since either we make symlinked morphologies a first-class behaviour, and validate that the link points inside the repository, which is a lot of work for something we don't and probably won't need; or we can just assume that we deal with the morphology this is linked to correctly anyway.
| * Make MorphologyFinder use file pathsRichard Maw2014-07-074-28/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We want to use file paths to locate morphologies now, so the old model of get a list of names and hand it those back to get the contents doesn't really make sense any more. This abstraction initially came about as one idea I had for moving morphologies out of the source tree was to put them in something like git notes, where it's possible to look up information for one commit in another ref in the repository, at which point this abstraction would have been flexible enough to handle that as well as in the However, moving the chunk morphologies into the definitions repository has other benefits too, so it makes more sense to be honest about using filenames in the API. It remains as a single point where we can put the logic for knowing which files in a repository look like morphologies, but if we need to remove any further functionality, it should be replaced by a single function.
| * Add class hierarchies for Morphology load errorsRichard Maw2014-07-071-28/+36
| | | | | | | | | | There's a lot of them, it's too much of a pain to enumerate them all, so it's convenient to provide a hierachy and catch the base exceptions.
| * Exorcise some old and unused commandsRichard Maw2014-07-072-2442/+536
| | | | | | | | | | | | | | | | | | | | | | | | | | `morph merge` only worked for a small subset of cases, and has been left to bit-rot, since we don't use it. `morph tag` is just a `git tag` when we have petrified definitions repository. We don't use it, nor do we need it, so it can go away rather than take up valuable development time fixing it when requirements change. `old-foo` have all been superceded by newer versions and are no-longer used.
| * Remove check on checkout/branch that there are systemsRichard Maw2014-07-071-22/+0
| | | | | | | | | | | | | | This was the wrong response to the problem of accidentally checking-out morph when trying to check out morphs. Now that it's called definitions, this is irrelevent, and aborting a checkout when this check fails is the wrong thing to do.
| * Remove code made vestigial from the `morph edit` rewriteRichard Maw2014-07-073-119/+1
| | | | | | | | | | | | This was originally used by `morph edit`, but since we removed the need to run `morph edit system stratum` and could shorten it to `morph edit chunk`, this function is no longer used.
* | Merge branch 'sam/splitting-fixes'Sam Thursfield2014-07-071-10/+14
|\ \ | | | | | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * | Rewrite SplitRules.partition() docstring to be clearer to meSam Thursfield2014-07-041-3/+4
| | |
| * | Fix issue when overriding default chunk splitting rulesSam Thursfield2014-07-041-7/+10
| |/ | | | | | | | | | | | | | | | | | | | | When overriding a default chunk splitting rule, the use 'break' would mean that some of the other default chunk split rules might be ignored. This isn't ever what you want. I also clarified the current behaviour in a comment. I think in future we should add a mechanism to extend the default rules, as well as the current mechanism which allows only replacing them. But that is a separate issue.
* | Fix Morph failing to update some cached git reposSam Thursfield2014-07-073-13/+46
|/ | | | | | | | | | | | | | | | | | | | | | | I was getting the following error when running the 'do-release.py' script: ERROR:root:Command failed: morph --quiet --trove-host=git.baserock.org list-artifacts baserock:baserock/definitions sam/auto-release minimal-system-x86_32-generic ERROR: Ref d67a0e110187abd560a1de63fa172894a52839d5 is an invalid reference for repo git://git.baserock.org/delta/linux The commit that it wants did actually exist in git.baserock.org and the logs show that Morph hadn't done a `git remote update` to get the commit locally. This turned out to be because it'd had already looked up a different ref in linux.git. It hadn't needed to run 'git remote update', but it *had* added linux.git to a "don't need to update this repo again" list. Oops! I've moved the code in question to the cachedrepo module so that the repo object keeps that of whether it should be updated. The bug is now fixed. As a side effect this patch fixes the spurious 'Updating repo file:///...' messages, which were printed even though repos with file:/// URLs are never updated.
* Make morph factory tests expect MorphologyNotFoundErrorRichard Ipsum2014-06-251-6/+5
|
* Replace AutodetectError with MorphologyNotFoundErrorRichard Ipsum2014-06-251-24/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The error message: ERROR: Failed to determine the build system of repo file://foo/bar/baz at ref 59713cf997385a094091443fdcce9d5c17313f39: was looking for distbuild-system-x86-64.morph is confusing since our system morph has nothing to do with build systems, the fact that build system autodetection is executed when looking for distbuild-system-x86-64.morph is an implementation detail that shouldn't be exposed to the user. This patch replaces this error message with: ERROR: Couldn't find morphology: distbuild-system-x86-64.morph This is still not ideal, since there are cases where we may not be able find the morph because build system autodection has failed, but out of the two user typos/mistakes are probably more likely. Differentiating between user error and build system detection failure would require more substantial changes. This patch also splits the get_morphology function into two functions _get_morphology_text, that actually fetches the morph text from the cache or otherwise infers it from the build system, and get_morphology which constructs a Morphology object from the text.
* Add chatty status messages to aid the userRichard Ipsum2014-06-251-5/+12
|
* Improve var names in morphloaderRichard Ipsum2014-06-241-24/+22
|
* Fix duplicate names error messageRichard Ipsum2014-06-241-5/+6
| | | | | This error message didn't report the cluster morph at fault. We also fix a minor formatting issue.
* Merge branch 'sam/only-update-git-when-needed-v2'Sam Thursfield2014-06-191-3/+11
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
| * Only updated cached git repos when necessarySam Thursfield2014-06-181-3/+11
| | | | | | | | | | | | | | | | | | | | When the given ref points to a specific commit, and it's already available in the locally cached copy of repo, there's no need to update the repo. If the ref points to a branch or tag, and the user didn't pass --no-git-update, the locally cached copy of the repo will still be updated. This should speed up many Morph commands.
* | Update the help text for morph deployAdam Coldrick2014-06-181-1/+5
| |
* | Allow the user to specify deployments in a clusterAdam Coldrick2014-06-181-8/+43
| | | | | | | | | | | | | | | | Instead of taking the name of a cluster morphology and zero or more parameters for overriding the cluster morphology, morph deploy should take the name of a cluster morphology and the names of zero or more system deployments that are defined in the cluster morphology. If no deployment names are given then all deployments are deployed.
* | Add unit test to test the validation of cluster morphsAdam Coldrick2014-06-181-0/+18
| |
* | Validate cluster morphologiesAdam Coldrick2014-06-181-1/+26
| | | | | | | | | | The names of deployments in cluster morphologies will need to be unique in order for the deployment of selected systems to work.
* | Fix: use end of options option with run-partsRichard Ipsum2014-06-181-1/+6
|/ | | | | | We want to supply the end of options option before the SYSTEM_INTEGRATION_PATH so that the system integration path doesn't get interpreted as an option if it happens to begin with a -
* Merge remote-tracking branch 'origin/sam/distbuild-gc-fix'Sam Thursfield2014-06-161-0/+3
|\ | | | | | | Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * Don't delete subdirectories of tempdir in gc pluginSam Thursfield2014-06-131-0/+3
| | | | | | | | | | | | There is code that assumes these exist in at least one place: StagingArea.abort(). That code should be fixed, but we should also stop deleting them every time we run 'morph gc'.
* | Check for presence of btrfs before trying to use itSam Thursfield2014-06-115-1/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If btrfs is not present in the kernel we end up with strange output like this: Error creating disk image2014-06-10 16:00:40 [devel-system-x86_64-generic][my-raw-disk-image][rawdisk.write]Failure to create disk image at /src/tmp/testdev.img ERROR: Command failed: mount -o loop /src/tmp/testdev.img /src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE mount: mounting /dev/loop0 on /src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE failed: Device or resource busy To avoid this confusing error, Morph should explicitly check first.
* | Make buildcommand use InitiatorConnectionMachineRichard Ipsum2014-06-111-2/+11
| | | | | | | | | | | | After a failed attempt to connect to the controller node the initiator will wait 30 seconds before attempting a reconnect, if this reconnect fails the initiator gives up.
* | Note future improvement for fetching artifacts from remote cacheSam Thursfield2014-06-101-0/+4
| |
* | Merge branch 'sam/remote-artifact-cache-failure'Sam Thursfield2014-06-103-50/+60
|\ \ | | | | | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * | Fix up sam/remote-artifact-cache-failure branch after reviewSam Thursfield2014-06-101-3/+10
| | |
| * | Ensure that transferring an artifact from the remote cache is atomicSam Thursfield2014-06-031-15/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Artifacts can have multiple parts; while this may not be an ideal design, changing the format of artifacts has implications for backwards compatibility. We should transfer all parts at once and delete them all if we encounter any errors, to reduce the change of getting the local artifact cache into an inconsistent state.
| * | Improve robustness when fetching artifacts from remote artifact cacheSam Thursfield2014-06-032-30/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously Morph would check if an artifact is present in the remote artifact cache, then fetch the necessary files. If an error occured during fetching, it would raise an error and abort. Instead, we should just try and fetch the files and if anything fails, move on to building locally. This avoids the situation where an error in the remote cache makes local building impossible, which we experienced recently.
| * | Tweak exception message of remote artifact cache GetErrorSam Thursfield2014-06-031-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before: ERROR: Failed to get metadata meta for the artifact file:///src/ws-baserock-hawk/baserock/ps/build-system/baserock/baserock/definitions|refs/heads/baserock/builds/778b1a370a1f43c497c1354a2a949de1/56c9ec89d09240fd80faa7d2226b7eda|core|core-devel from the artifact cache http://git.baserock.org:8080/ After: ERROR: Failed to get metadata meta for the artifact f896a081beacd4a99ded38d28b44fbf02970038fb53349265f85f8f3298ead9d.stratum.core-devel from the artifact cache http://git.baserock.org:8080/ When debugging artifact cache issues, the information that's most useful is the filename of the artfact.
| * | Make all remote artifact cache get errors inherit from GetErrorSam Thursfield2014-06-031-3/+3
| | | | | | | | | | | | This saves from having to catch three separate exceptions.
* | | Morph deploy should never update git reposPaul Sherwood2014-06-101-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | The deploy plugin is relying on existing code which respects the 'no-git-update' setting and updates gits if it is not set. Since deploy can only work for built systems anyway, we can force this True for deployments, to avoid wasting the user/caller's time.
* | | Merge branch 'baserock/richardmaw/S11226/yarns-v2'Richard Maw2014-06-091-2/+2
|\ \ \ | |_|/ |/| | | | | Reviewed-by: Sam Thursfield and Richard Ipsum
| * | initramfs.write: create parent directories of locationRichard Maw2014-06-061-2/+2
| | |
* | | Fix up before mergeSam Thursfield2014-06-091-36/+43
| | | | | | | | | | | | | | | | | | | | | - All tests now pass - The odd case of chunks with the same name but different repo URLs now correctly informs the user of the multiple checkouts that were done. - Tidyups
* | | Add messages to help user know what is happeningPaul Sherwood2014-06-081-0/+16
| | |
* | | Simplify morph edit syntax and logicPaul Sherwood2014-06-081-124/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since we removed ref: fields from the system morph files, some of the logic in morph edit is no longer needed. In particular, running morph edit <system> <stratum> is a no-op. This is because the version of <system> and <stratum> are now implicit from the context of what we are doing. In other words we're always working with the current version of <system> and <stratum> from the system branch we are in. Because of the complexity of morph's logic, we didn't notice this when dropping the ref: fields, and we missed the opportunity to simplify our logic for 'morph edit' This patch aims to provide the simplest possible implementation of morph edit: morph edit <chunk> It checks all strata for instances of <chunk>, and does what morph edit should do for the instances it finds. A later patch can add warnings to help users deal with situations where <chunk> is not found, or is found more than once. Also since this changes the syntax of morph, it breaks many of our tests. Later patches will address that.
* | | Make morph gc clear out deployments in tmpdirPaul Sherwood2014-06-071-1/+6
|/ / | | | | | | | | | | | | | | Generally deployment temp dirs are removed by morph during deployment, but in some cases deployment dirs may not be cleared up, for example if morph gets a SIGKILL or something unexpected happens that causes morph to terminate without having a chance to cleanup.
* | Merge branch 'baserock/richardmaw/S11226/writeext-fstab'Richard Maw2014-06-061-5/+7
|\ \ | | | | | | | | | | | | Reviewed-by: Pedro Alvarez Reviewed-by: Sam Thursfield
| * | Use UUID in fstab entries if providedRichard Maw2014-06-041-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes systems use the UUID of the disk in the fstab when there is no pre-existing fstab entry for /. This happens whether the system has an initramfs or not, since it should be harmless, and by this point we're in userland, so can know what the UUIDs are. Passing the UUID to `complete_fstab_for_btrfs_layout` is optional, and defaults to the old behaviour of using /dev/sda, since it is called directly by some write extensions for doing upgrades, and upgrading systems that use an initramfs will be part of a later patch.
View Lorry Controller Status