summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Check OpenStack credentials in openstack.checkbaserock/pedroalvarez/openstack-check-improvementsPedro Alvarez2014-09-052-13/+49
| | | | | | | | If the credentials are wrong, then morph will fail before attempting the OpenStack deployment. To achieve that openstack.check will attempt to run `glance image-list`.
* Merge branch 'baserock/richardmaw/less-ls-remote'Richard Maw2014-09-022-10/+23
|\ | | | | | | | | Reviewed-by: Lars Wirzenius Reviewed-by: Sam Thursfield
| * Use git's remotes to determine unpushed branchesbaserock/richardmaw/less-ls-remoteRichard Maw2014-09-022-10/+23
|/ | | | | | | | | | | | | | | | | | | | | | Rather than using `git ls-remote` every time to see if there are changes at the remote end, use a local cache. Git already solves this problem with its refs/remotes/$foo branch namespace, so we can just use that instead. In addition, the detection of which upstream branch to use has been improved; it now uses git's notion of which the upstream branch of your current branch is, which saves effort in the implementation, and allows the name of the local branch to differ from that of the remote branch. This now won't notice if the branch you currently have checked out had commits pushed from another source, but for some use-cases this is preferable, as the result equivalent to if you had built before the other push. It may make sense to further extend this logic to check that the local branch is not ahead of the remote branch, instead of requiring them to be equal.
* Merge branch 'sam/less-cache-key-logging'Sam Thursfield2014-09-021-15/+5
|\ | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * Remove some cache key related log messagesSam Thursfield2014-08-291-15/+5
| | | | | | | | | | | | | | | | | | There's no need to log every time we look something up in a dict. This just makes log files huge. The CacheKeyComputer.compute_key() function still logs the first time it calculates a cache key for an Artifact instance. This message now includes the hex string that is used to identify the artifact.
* | Merge branch 'deploy-improvements'Sam Thursfield2014-09-016-36/+198
|\ \ | | | | | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * | deploy: Make Python extensions log debug messages to MORPH_LOG_FD by defaultSam Thursfield2014-09-011-1/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously logging was disabled for Python deploy extensions. We get a lot of useful information for free in the log file by doing this: the environment will be written when the subprocess starts, and if it crashes the full backtrace will be written there too. Subcommand execution with cliapp.runcmd() will also be logged.
| * | deploy: Allow extensions to write to Morph log fileRichard Maw2014-09-012-26/+143
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is achieved by passing it the write end of a pipe, so that the extension has somewhere to write debug messages without clobbering either stdout or stderr. Previously deployment extensions could only display status messages on stdout, which is good for telling the user what is happening but is not useful when trying to do post-mortem debugging, when more information is usually required. This uses the asyncore asynchronous event framework, which is rather specific to sockets, but can be made to work with file descriptors, and has the advantage that it's part of the python standard library. It relies on polling file descriptors, but there's a trick with pipes to use them as a notification that a process has exited: 1. Ensure the write-end of the pipe is only open in the process you want to know when it exits 2. Make sure the pipe's FDs are set in non-blocking read/write mode 3. Call select or poll on the read-end of the file descriptor 4. If select/poll says you can read from that FD, but you get an EOF, then the process has closed it, and if you know it doesn't do that in normal operation, then the process has terminated. It took a few weird hacks to get the async_chat module to unregister its event handlers on EOF, but the result is an event loop that is asleep until the kernel tells it that it has to do something.
| * | Add `morph upgrade` command, deprecate `morph deploy --upgrade`Sam Thursfield2014-09-014-8/+30
| | | | | | | | | | | | | | | | | | The arguments to `morph deploy` can get quite long, any way we can make it shorter and clearer is useful. We can also avoid having the strange --no-upgrade flag in future.
| * | Clarify that multiple images can be deployed at once by `morph deploy`Sam Thursfield2014-09-011-1/+1
|/ /
* | Merge branch 'baserock/richardmaw/yarn-git-daemon'Richard Maw2014-08-2811-150/+143
|\ \ | |/ |/| | | | | Reviewed-by: Lars Wirzenius (git-daemon) Reviewed-by: Sam Thursfield (yarn fixup)
| * yarns: Use a more realistic simulation of a git serverbaserock/richardmaw/yarn-git-daemonRichard Maw2014-08-287-11/+62
| | | | | | | | | | | | | | | | | | | | | | Previously we would use file: URIs to point to the git repositories. This was fast and simple, but had the drawback that it bypassed all the git cache logic, so changes to the git cache weren't adequately covered by the test suite. Now we spool up a simulated git server per scenario, and shut it down at the end.
| * Add script for launching git daemon on random portRichard Maw2014-08-281-0/+46
| | | | | | | | | | | | | | This starts the git daemon procvess in inetd mode, so we can bind to an ephemeral port, and still be able to report which port was used, so we can construct git URIs using a non-standard port to talk to simulated git servers.
| * Add yarn for building uncommitted changesRichard Maw2014-08-282-56/+33
| | | | | | | | This replaces an old cmdtest.
| * yarns: Use run_morph for pushing branchesRichard Maw2014-08-281-1/+2
| | | | | | | | We're using the wrong morph if we don't use run_morph
| * Remove cmdtests deprecated by yarnsRichard Maw2014-08-282-82/+0
|/ | | | | "morph makes temporary build branches for uncommitted changes when necessary" makes these redundant.
* Merge remote-tracking branch 'origin/baserock/richardmaw/bugfix/http-fail'Richard Maw2014-08-262-5/+12
|\ | | | | | | | | Reviewed-by: Francisco Redondo Marchena Reviewed-by: Sam Thursfield
| * Only autodetect morphology when result is 404baserock/richardmaw/bugfix/http-failRichard Maw2014-08-192-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The MorphologyFactory class will use a RemoteRepoCache to see if a morphology file exists, and if it doesn't, uses a file listing to see if it can detect what build-system is uses, hence what the default morphology should be. However, it was overly generic in what error cases it would accept as the morphology not being found, so if the RemoteRepoCache was suddenly un-resolvable for a brief period, then it would assume the morphology didn't exist, and use the default one. This happened to a user, and the result was a full rebuild. So we now fix this by only raising the exception that means the file didn't exist, if we got a HTTP 404.
* | Fix `morph edit` for non-file URIsRichard Maw2014-08-261-1/+1
| | | | | | | | | | | | | | | | | | The clone_into function is non-functional when you pass it a sha1 ref. If you have a file:// URI then this doesn't get used, which is how it slipped past the tests. Reviewed-by: Lars Wirzenius +2
* | Prevent git-replace refs affecting git operationsRichard Maw2014-08-2111-139/+171
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We assumed that the sha1 of the tree of the commit of the ref we care about was sufficient to cache, but `git replace` means that you need to know the state of other branches too, since anything in refs/replace can completely change what the tree you check-out is. This behaviour can be disabled globally by setting GIT_NO_REPLACE_OBJECTS, so we're going to do that. If we need to integrate a project that uses git-replace to change the contents of their git trees then we could support that by: if any(refs/replace/*): potentially_replacable_objects = [ `git rev-parse HEAD`, `git rev-parse HEAD^{commit}`, `git rev-parse HEAD^{tree}`] potentially_replacable_objects.extend( `git ls-tree -r HEAD | awk '{print $3}'`) # NOTE: doesn't handle submodules, and you'd need to expand this # set whenever you process a replacement for object in refs/replace/*: if basename(object) not in potentially_replacable_objects: continue cache_key['replacements'][basename(object)] = `git rev-parse $object` If we were to support this would need to traverse the tree anyway, doing replacements, so we may as well use libgit to do the checkout anyway, and list which replacements were used. However, since the expected use-case of `git replace` is as a better way to do history grafting, we're unlikely to need it, as it would only have any effect if it replaced the commit we were using with a different one. Rubber-stamped-by: Daniel Silverstone
* | deploy: Check correct usage of --upgrade for rawdisk deploymentsSam Thursfield2014-08-191-0/+21
| | | | | | | | | | | | | | | | | | | | This avoids confusion when the user expected to be doing an initial deployment and wasn't aware that a file with the same name as the target already existed. Previously rawdisk.write would try to mount the file and upgrade it. Now we require the user to pass '--upgrade' when they intend to upgrade, as with other deployment extensions.
* | Cache configuration values in GitDirectory.Daniel Silverstone2014-08-191-2/+6
|/ | | | | | | | This reduces the vast number of 'git config -z core.bare' which we used to get a lot of, to one or two per run. It doesn't save much time, but it does make logs less full of confusion. Reviewed-By: Richard maw <richard.maw@codethink.co.uk> +2
* Merge branch 'baserock/straycat/distbuild-trivia'Sam Thursfield2014-08-181-30/+9
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * Rename for consistencyRichard Ipsum2014-08-141-10/+9
| |
| * Remove dead functionsRichard Ipsum2014-08-141-20/+0
| |
* | Merge branch 'baserock/franred/fixes-needed-for-organize-definitions'Francisco Redondo Marchena2014-08-152-26/+40
|\ \ | |/ |/| | | | | Reviewed by: Richard Maw <richard.maw@codethink.co.uk> Merged by: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk>
| * Fix morphloader to unset_defaults for chunksFrancisco Redondo Marchena2014-08-151-2/+13
| |
| * Update morphloader_test.pyFrancisco Redondo Marchena2014-08-151-12/+13
| |
| * Set chunk static default to Nonebaserock/franred/fixes-needed-for-organize-definitionsFrancisco Redondo Marchena2014-08-151-12/+12
| |
| * Add system-integration to chunk _static_defaultsFrancisco Redondo Marchena2014-08-151-0/+1
| |
| * Add deploy-defaults before deploy in the MorphologyDumper keyorderFrancisco Redondo Marchena2014-08-151-0/+1
|/
* Merge branch 'baserock/richardmaw/james/writeexts_support_jetson'James Thomas2014-08-142-22/+83
|\ | | | | | | | | Reviewed by: Richard Maw <richard.maw@codethink.co.uk> Merged by: James Thomas <james.thomas@codethink.co.uk>
| * Merge remote-tracking branch 'origin/baserock/james/writeexts_support_jetson'baserock/richardmaw/james/writeexts_support_jetsonRichard Maw2014-08-122-22/+83
| |\
| | * Add support for a device tree to be set using DTB_PATHbaserock/james/writeexts_support_jetsonJames Thomas2014-08-031-0/+21
| | |
| | * Make bootloader config/install more genericJames Thomas2014-08-031-17/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove the BOOTLOADER environment variable and instead favour BOOTLOADER_CONFIG_FORMAT to set the desired bootloader format, and BOOTLOADER_INSTALL to set the type of bootloader to install. For example, since u-boot can boot using extlinux.conf files, it's conceivable that someone might want to do CONFIG_FORMAT=extlinux.conf, INSTALL=u-boot. However, for most platforms you would want to set INSTALL to "none"
| | * Support setting a different root device using ROOT_DEVICEJames Thomas2014-08-031-2/+5
| |/
* | Merge branch 'baserock/adamcoldrick/remove-morph2-tweaked'Adam Coldrick2014-08-1450-2212/+1016
|\ \ | | | | | | | | | | | | Reviewed-by: Richard Maw
| * | Rename morph3 to morphologyAdam Coldrick2014-08-146-57/+57
| | | | | | | | | | | | | | | | | | Instead of leaving morph3 with a potentially confusing name, rename it to `morphology` since it is now the only implementation of the Morphology class.
| * | Revert "Ensure that none of the _orig_* fields are in the cache key"Adam Coldrick2014-08-141-4/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit 4124c50b8dc3dfb0ffb933153d0fe6385edf389c. This commit is no longer needed now that morph2 is gone. Conflicts: morphlib/cachekeycomputer.py
| * | Remove morph2 and its testsAdam Coldrick2014-08-143-705/+0
| | | | | | | | | | | | This commit removes the now unneeded morph2 and its associated tests.
| * | edit-morph: Use morphloader rather than morph2Adam Coldrick2014-08-141-52/+8
| | | | | | | | | | | | | | | | | | Update the edit-morph script used in the test suite to use morphloader for saving/loading morphologies rather than morph2. Also remove some unused code.
| * | cmdtests: Use YAML not JSON to represent morphologiesAdam Coldrick2014-08-1421-867/+549
| | |
| * | cmdtests: Change the error messages to be detectedAdam Coldrick2014-08-144-4/+4
| | |
| * | unittests: Make the unittests use morphloaderAdam Coldrick2014-08-146-297/+254
| | | | | | | | | | | | | | | This commit removes all use of morph2 from the unittests, replacing it with morphloader/morph3. It also converts the test morphologies to YAML.
| * | buildsystem: Generate a Morphology not textAdam Coldrick2014-08-144-30/+29
| | | | | | | | | | | | | | | Rather than generating the text of a morphology which is later loaded, generate a Morphology object and return that.
| * | morphologyfactory: Use morphloader not morph2Adam Coldrick2014-08-142-166/+75
| | | | | | | | | | | | | | | | | | | | | | | | This commit reworks morphologyfactory to stop using morph2. It removes the validation that is already handled by morphloader, and uses morphloader to load the morphology rather than using morph2. The unit tests are also changed to turn the example morphologies into YAML, and also to check for the correct exceptions.
| * | morphloader: Get commands when loading morphologyAdam Coldrick2014-08-144-3/+14
| | | | | | | | | | | | | | | | | | Rather than having a `get_commands` method to obtain missing commands from the build system when they are needed, get the commands when loading a morphology.
| * | morphloader: Add and remove some default valuesAdam Coldrick2014-08-142-12/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | This commit stops morphloader setting the `morph` field in chunk specs, and also makes it set defaults for the prefix and build-mode fields. Not setting the `morph` field is necessary as its presence in chunk specs is used by `traverse_morphs` to mean that the morphology file is in the definitions repository, not the chunk source repository. If we set a default value here, we end up looking for files which do not exist.
| * | morphset: Don't expect to be able to find `morph` or `name`Adam Coldrick2014-08-141-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | Currently, morphloader sets the morph field in chunk specs by default. A later commit will remove this behaviour, so morphset needs to stop expecting the `morph` field to exist in morphology specs. However, stratum specs don't have a `name` field so we can't expect to be able to find that either.
| * | show-dependencies: Use sanitise_morphology_path instead of stripping `.morph`Adam Coldrick2014-08-142-3/+3
| | |