summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
...
| * Create multiple sources per stratum morphologyRichard Maw2014-09-196-66/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Building per-artifact results in undesirable behaviour, as multiple artifacts are produced for every chunk build. It therefore makes more sense to build per-source. This implies that actually, the model of one source per morphology is wrong and we should move the dependencies into the source. Unlike chunks however, where every chunk artifact has the same dependencies, stratum artifacts can have different dependencies. So before we can move the dependencies into the Source, we need to have as many Sources as Stratum Artifacts.
| * Make morphologies hashableRichard Maw2014-09-191-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | I need to be able to deduplicate a list of morphologies. Putting it in a set is the easiest way, but it needs to be hashable. It's not included in dicts by default, since they're stored by reference, and you can change them while they're in the dict, so the hash value can change. I don't need to deduplicate morphologies by their contents, just by reference though, so using `id` as the hash function is sufficient.
| * Turn Morphology.needs_artifact_metadata_cached into a propertyRichard Maw2014-09-193-40/+4
| | | | | | | | | | This means we can remove some complication from the MorphologyFactory class.
| * Remove Morphology.builds_artifactsRichard Maw2014-09-195-44/+0
| | | | | | | | | | This was used before the Artifact splitting code landed to determine which artifacts should be produced.
| * Tidy morphlib.artifact.Artifact.metadata_basenameRichard Maw2014-09-191-4/+1
| |
| * Rename BuildCommand.get_sources to fetch_sourcesRichard Maw2014-09-192-3/+3
| | | | | | | | | | There's other methods called get_sources in other modules, and fetch_sources explains more about what it does in the context.
| * Move metadata_version value to cachekeycomputerRichard Maw2014-09-192-5/+1
| | | | | | | | | | | | | | | | | | | | | | | | metadata_version is for when the format of the /baserock metadata files changes. This means it would make sense for it to either live with the code for generating the metadata, or the cache key code so it lives with the rest of the compatibility values. Since the code for generating the metadata isn't in a nice module anywhere, I've put it in the cachekeycomputer module.
| * Eliminate BuilderBase.new_artifactRichard Maw2014-09-192-14/+3
| | | | | | | | | | | | | | | | | | | | It was an odd thing to have, when Artifact objects are part of your input. Its purpose appears to have been to allow the build step to produce an artifact called $morphology_name-rootfs, but since the split rules decide that the artifact is called that anyway, the new_artifact step is redundant.
| * Validate build-depends in MorphologyLoaderSam Thursfield2014-09-192-5/+56
| |
| * Don't validate morphologies during ArtifactResolverRichard Maw2014-09-172-49/+9
| |
| * Add __repr__ method to Source and SplitRulesRichard Maw2014-09-173-0/+23
| | | | | | | | | | This helps debugging issues with rule matching, since SplitRules can be print-statemented
| * Make remoteartifactcache tests define sensible artifactsRichard Maw2014-09-171-3/+3
| | | | | | | | | | | | | | | | It rather peculiarly defines artifacts that have different cache keys, but the same source. This flies in the face of how real artifacts get cache keys, and our ability to move the cache key to being per-source.
| * Be more lenient in Artifact dependency unit testsRichard Maw2014-09-171-4/+6
| | | | | | | | | | We don't care if add_dependency causes there to be multiple dependents, just that our artifacts are properly included.
| * Stop creating a separate -kernel artifact for systemsRichard Maw2014-09-171-20/+0
| | | | | | | | | | | | | | This pre-dates deployment, and if we need the kernel, we can always copy it out of the rootfs. It also uses new_artifact, which is a method I want to remove.
* | Prevent cliapp from logging env. variables with 'PASSWORD' in their nameSam Thursfield2014-09-193-0/+19
| | | | | | | | | | The upstream cliapp project is not interested in this functionality right now.
* | Don't log environment variables with 'PASSWORD' in their name.Sam Thursfield2014-09-193-21/+23
|/ | | | | | | | | | | | | | | | | | | This involved rewriting the util.log_dict_diff() function. It has been renamed to log_environment_changes() to better reflect its purpose. It no longer logs both the old and new values in the event of an environment variable changing. It now just logs the new value. This makes the code simpler and seems like it should not be a big problem. Some projects recommend passing credentials through the environment. OpenStack does this, for example, see: <http://docs.openstack.org/user-guide/content/cli_openrc.html> It's unlikely that users would be happy about applications saving these passwords in log files all over their system. I do not recommend ever storing valuable passwords in the environment.
* Fix tarfile.chown for versions of python smaller than 2.7.3baserock/franred/bugfix/python-tarfile-chownFrancisco Redondo Marchena2014-09-121-9/+38
| | | | | | | | This workaround fix http://bugs.python.org/issue12841 The code added in this patch is from tarfile.py from python 2.7.3. Also only use the workaround for tarfile.makefile when the ptyhon is smaller than 2.7.4
* Fix `morph branch-from-image` to work for deployed imagesbaserock/richardmaw/bugfix/branch-from-deployed-imageRichard Maw2014-09-111-1/+2
| | | | | | It broke when we added /baserock/deployment.meta. We didn't notice this because our test suite was looking at the artifact produced by morph build, and listed on the terminal.
* Fix up openstack.write extensionbaserock/pedroalvarez/openstack-exts-fixupsPedro Alvarez2014-09-091-1/+0
| | | | | | The openstack.write extension was calling a nonexistent method 'check_location'. This metod was moved to openstack.check in the commit ba7d1d1ed3bad002ce36e5d4adf4e3794625091a.
* Fix openstack write/check exts to pass the testsPedro Alvarez2014-09-092-2/+2
|
* Merge branch 'baserock/pedroalvarez/openstack-check-improvements'Pedro Alvarez2014-09-052-13/+49
|\ | | | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Lars Wirzenius
| * Check OpenStack credentials in openstack.checkbaserock/pedroalvarez/openstack-check-improvementsPedro Alvarez2014-09-052-13/+49
| | | | | | | | | | | | | | | | If the credentials are wrong, then morph will fail before attempting the OpenStack deployment. To achieve that openstack.check will attempt to run `glance image-list`.
* | Fix shell quoting for ssh remote command linesLars Wirzenius2014-09-052-6/+11
| | | | | | | | Found by Richard Maw.
* | Use $((...))) instead of $(... | bc)Lars Wirzenius2014-09-051-4/+4
| |
* | Document why VBoxManage is run in recv-hole instead of callerLars Wirzenius2014-09-051-0/+25
| |
* | Transfer sparse files faster for kvm, vbox deploymentLars Wirzenius2014-09-055-8/+311
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The KVM and VirtualBox deployments use sparse files for raw disk images. This means they can store a large disk (say, tens or hundreds of gigabytes) without using more disk space than is required for the actual content (e.g., a gigabyte or so for the files in the root filesystem). The kernel and filesystem make the unwritten parts of the disk image look as if they are filled with zero bytes. This is good. However, during deployment those sparse files get transferred as if there really are a lot of zeroes. Those zeroes take a lot of time to transfer. rsync, for example, does not handle large holes efficiently. This change introduces a couple of helper tools (morphlib/xfer-hole and morphlib/recv-hole), which transfer the holes more efficiently. The xfer-hole program reads a file and outputs records like these: DATA 123 binary data (exaclyt 123 bytes and no newline at the end) HOLE 3245 xfer-hole can do this efficiently, without having to read through all the zeroes in the holes, using the SEEK_DATA and SEEK_HOLE arguments to lseek. Using this, the holes take only take a few bytes each, making it possible to transfer a disk image faster. In my benchmarks, transferring a 100G byte disk image took about 100 seconds for KVM, and 220 seconds for VirtualBox (which needs to more work at the receiver to convert the raw disk to a VDI). Both benchmarks were from a VM on my laptop to the laptop itself. The interesting bit here is that the receiver (recv-hole) is simple enough that it can be implemented in a bit of shell script, and the text of the shell script can be run on the remote end by giving it to ssh as a command line argument. This means there is no need to install any special tools on the receiver, which makes using this improvement much simpler.
* Use git's remotes to determine unpushed branchesbaserock/richardmaw/less-ls-remoteRichard Maw2014-09-022-10/+23
| | | | | | | | | | | | | | | | | | | | | | Rather than using `git ls-remote` every time to see if there are changes at the remote end, use a local cache. Git already solves this problem with its refs/remotes/$foo branch namespace, so we can just use that instead. In addition, the detection of which upstream branch to use has been improved; it now uses git's notion of which the upstream branch of your current branch is, which saves effort in the implementation, and allows the name of the local branch to differ from that of the remote branch. This now won't notice if the branch you currently have checked out had commits pushed from another source, but for some use-cases this is preferable, as the result equivalent to if you had built before the other push. It may make sense to further extend this logic to check that the local branch is not ahead of the remote branch, instead of requiring them to be equal.
* Merge branch 'sam/less-cache-key-logging'Sam Thursfield2014-09-021-15/+5
|\ | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * Remove some cache key related log messagesSam Thursfield2014-08-291-15/+5
| | | | | | | | | | | | | | | | | | There's no need to log every time we look something up in a dict. This just makes log files huge. The CacheKeyComputer.compute_key() function still logs the first time it calculates a cache key for an Artifact instance. This message now includes the hex string that is used to identify the artifact.
* | deploy: Make Python extensions log debug messages to MORPH_LOG_FD by defaultSam Thursfield2014-09-011-1/+24
| | | | | | | | | | | | | | | | | | Previously logging was disabled for Python deploy extensions. We get a lot of useful information for free in the log file by doing this: the environment will be written when the subprocess starts, and if it crashes the full backtrace will be written there too. Subcommand execution with cliapp.runcmd() will also be logged.
* | deploy: Allow extensions to write to Morph log fileRichard Maw2014-09-012-26/+143
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is achieved by passing it the write end of a pipe, so that the extension has somewhere to write debug messages without clobbering either stdout or stderr. Previously deployment extensions could only display status messages on stdout, which is good for telling the user what is happening but is not useful when trying to do post-mortem debugging, when more information is usually required. This uses the asyncore asynchronous event framework, which is rather specific to sockets, but can be made to work with file descriptors, and has the advantage that it's part of the python standard library. It relies on polling file descriptors, but there's a trick with pipes to use them as a notification that a process has exited: 1. Ensure the write-end of the pipe is only open in the process you want to know when it exits 2. Make sure the pipe's FDs are set in non-blocking read/write mode 3. Call select or poll on the read-end of the file descriptor 4. If select/poll says you can read from that FD, but you get an EOF, then the process has closed it, and if you know it doesn't do that in normal operation, then the process has terminated. It took a few weird hacks to get the async_chat module to unregister its event handlers on EOF, but the result is an event loop that is asleep until the kernel tells it that it has to do something.
* | Add `morph upgrade` command, deprecate `morph deploy --upgrade`Sam Thursfield2014-09-012-6/+28
| | | | | | | | | | | | The arguments to `morph deploy` can get quite long, any way we can make it shorter and clearer is useful. We can also avoid having the strange --no-upgrade flag in future.
* | Clarify that multiple images can be deployed at once by `morph deploy`Sam Thursfield2014-09-011-1/+1
|/
* Merge remote-tracking branch 'origin/baserock/richardmaw/bugfix/http-fail'Richard Maw2014-08-262-5/+12
|\ | | | | | | | | Reviewed-by: Francisco Redondo Marchena Reviewed-by: Sam Thursfield
| * Only autodetect morphology when result is 404baserock/richardmaw/bugfix/http-failRichard Maw2014-08-192-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The MorphologyFactory class will use a RemoteRepoCache to see if a morphology file exists, and if it doesn't, uses a file listing to see if it can detect what build-system is uses, hence what the default morphology should be. However, it was overly generic in what error cases it would accept as the morphology not being found, so if the RemoteRepoCache was suddenly un-resolvable for a brief period, then it would assume the morphology didn't exist, and use the default one. This happened to a user, and the result was a full rebuild. So we now fix this by only raising the exception that means the file didn't exist, if we got a HTTP 404.
* | Fix `morph edit` for non-file URIsRichard Maw2014-08-261-1/+1
| | | | | | | | | | | | | | | | | | The clone_into function is non-functional when you pass it a sha1 ref. If you have a file:// URI then this doesn't get used, which is how it slipped past the tests. Reviewed-by: Lars Wirzenius +2
* | Prevent git-replace refs affecting git operationsRichard Maw2014-08-2111-139/+171
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We assumed that the sha1 of the tree of the commit of the ref we care about was sufficient to cache, but `git replace` means that you need to know the state of other branches too, since anything in refs/replace can completely change what the tree you check-out is. This behaviour can be disabled globally by setting GIT_NO_REPLACE_OBJECTS, so we're going to do that. If we need to integrate a project that uses git-replace to change the contents of their git trees then we could support that by: if any(refs/replace/*): potentially_replacable_objects = [ `git rev-parse HEAD`, `git rev-parse HEAD^{commit}`, `git rev-parse HEAD^{tree}`] potentially_replacable_objects.extend( `git ls-tree -r HEAD | awk '{print $3}'`) # NOTE: doesn't handle submodules, and you'd need to expand this # set whenever you process a replacement for object in refs/replace/*: if basename(object) not in potentially_replacable_objects: continue cache_key['replacements'][basename(object)] = `git rev-parse $object` If we were to support this would need to traverse the tree anyway, doing replacements, so we may as well use libgit to do the checkout anyway, and list which replacements were used. However, since the expected use-case of `git replace` is as a better way to do history grafting, we're unlikely to need it, as it would only have any effect if it replaced the commit we were using with a different one. Rubber-stamped-by: Daniel Silverstone
* | deploy: Check correct usage of --upgrade for rawdisk deploymentsSam Thursfield2014-08-191-0/+21
| | | | | | | | | | | | | | | | | | | | This avoids confusion when the user expected to be doing an initial deployment and wasn't aware that a file with the same name as the target already existed. Previously rawdisk.write would try to mount the file and upgrade it. Now we require the user to pass '--upgrade' when they intend to upgrade, as with other deployment extensions.
* | Cache configuration values in GitDirectory.Daniel Silverstone2014-08-191-2/+6
|/ | | | | | | | This reduces the vast number of 'git config -z core.bare' which we used to get a lot of, to one or two per run. It doesn't save much time, but it does make logs less full of confusion. Reviewed-By: Richard maw <richard.maw@codethink.co.uk> +2
* Fix morphloader to unset_defaults for chunksFrancisco Redondo Marchena2014-08-151-2/+13
|
* Update morphloader_test.pyFrancisco Redondo Marchena2014-08-151-12/+13
|
* Set chunk static default to Nonebaserock/franred/fixes-needed-for-organize-definitionsFrancisco Redondo Marchena2014-08-151-12/+12
|
* Add system-integration to chunk _static_defaultsFrancisco Redondo Marchena2014-08-151-0/+1
|
* Add deploy-defaults before deploy in the MorphologyDumper keyorderFrancisco Redondo Marchena2014-08-151-0/+1
|
* Merge branch 'baserock/richardmaw/james/writeexts_support_jetson'James Thomas2014-08-142-22/+83
|\ | | | | | | | | Reviewed by: Richard Maw <richard.maw@codethink.co.uk> Merged by: James Thomas <james.thomas@codethink.co.uk>
| * Merge remote-tracking branch 'origin/baserock/james/writeexts_support_jetson'baserock/richardmaw/james/writeexts_support_jetsonRichard Maw2014-08-122-22/+83
| |\
| | * Add support for a device tree to be set using DTB_PATHbaserock/james/writeexts_support_jetsonJames Thomas2014-08-031-0/+21
| | |
| | * Make bootloader config/install more genericJames Thomas2014-08-031-17/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove the BOOTLOADER environment variable and instead favour BOOTLOADER_CONFIG_FORMAT to set the desired bootloader format, and BOOTLOADER_INSTALL to set the type of bootloader to install. For example, since u-boot can boot using extlinux.conf files, it's conceivable that someone might want to do CONFIG_FORMAT=extlinux.conf, INSTALL=u-boot. However, for most platforms you would want to set INSTALL to "none"
| | * Support setting a different root device using ROOT_DEVICEJames Thomas2014-08-031-2/+5
| |/
* | Rename morph3 to morphologyAdam Coldrick2014-08-146-57/+57
| | | | | | | | | | | | Instead of leaving morph3 with a potentially confusing name, rename it to `morphology` since it is now the only implementation of the Morphology class.
View Lorry Controller Status