summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
* Escape special characters and handle whitespace in gitattributesbaserock/adamcoldrick/git-fat-handle-whitespace-rebaseAdam Coldrick2014-04-021-1/+21
| | | | | | | | | | | | | | | | Since git's parsing of .gitattributes can't deal with spaces, or even with quotes, replace any whitespace with [[:space:]]. Also, make sure to escape any glob metacharacters to avoid accidental pattern matches. Note: This approach could cause a file "foo<tab>bar" to be treated as a binary if a file "foo bar" has already been added with morph add-binary. However this is as good of a workaround as I can find. Signed-off by: Adam Coldrick
* Merge branch 'baserock/richardipsum/merge_distbuild'Richard Ipsum2014-03-275-10/+368
|\ | | | | | | | | | | | | | | | | | | Conflicts: morphlib/plugins/deploy_plugin.py without-test-modules Reviewed by: Richard Maw Lars Wirzenius
| * UI change: distbuild has its own subcommandRichard Ipsum2014-03-261-7/+39
| |
| * Remove Initiator and InitiatorBuildCommandRichard Ipsum2014-03-261-49/+0
| |
| * Bring InitiatorBuildCommand into buildcommandRichard Ipsum2014-03-261-0/+30
| |
| * Fix deployRichard Ipsum2014-03-261-2/+3
| | | | | | | | | | | | | | | | | | tarfile's open needs the file-like object to have a tell() method, objects returned from sockets don't have this method. So instead we fetch the artifact from the remote and cache it locally
| * Make InitiatorBuildCommand construct BuildCommandRichard Ipsum2014-03-261-0/+1
| | | | | | | | | | | | | | The constructor for BuildCommand sets up the caches, we want the caches to be set up for distbuild too so that we can deploy a system (system artifact will be fetched from the artifact cache)
| * Fix push_build_branchesRichard Ipsum2014-03-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | A semantic error in the BuildBranch class meant that it was not possible to push temporary branches. This escaped testing since BuildBranch interacts too tightly with other components to be easily unit-tested, so testing was deferred to a yarn test. However, coverage isn't measured in yarn tests, so this code path was forgotten.
| * Add the distbuild pluginRichard Ipsum2014-03-211-0/+343
| |
* | Stop using GitDirectory().list_files() to check for .gitfatAdam Coldrick2014-03-251-1/+1
| | | | | | | | | | This function causes a UnicodeDecodeError for some repositories when building. Use os.path.isfile() when looking for .gitfat instead.
* | Use os.environ[] instead of os.environ.get() to read a required variableSam Thursfield2014-03-241-1/+1
| |
* | Be consistent with nfsboot_root pathSam Thursfield2014-03-241-3/+4
| | | | | | | | | | We were building it from a variable in some places and hardcoding it in others; now we build it from a variable everywhere.
* | Merge branch 'sam/nfs-deploy-fixes'Richard Ipsum2014-03-203-60/+74
|\ \ | | | | | | | | | | | | | | | Reviewed by: Richard Maw Lars Wirzenius
| * | Make sanity checks for nfsboot deployments run earlierSam Thursfield2014-03-182-55/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Move some code to the '.check' extension to verify that the deployment can happen *before* spending 5 minutes unpacking and configuring the rootfs. This is not a perfect solution yet because when multiple systems are being deployed in a cluster, we do not check all systems and then deploy them all. Instead, we check one, then deploy it, then check the second, etc.
| * | Improve comment in nfsboot.configureSam Thursfield2014-03-181-1/+3
| | |
| * | Remove fstab hack from nfsboot.configureSam Thursfield2014-03-181-4/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | There is no longer a default /etc/fstab in the Baserock fhs-dirs chunk, and the nfsboot.write extension does not use the default Btrfs system layout so no entry is added for / to /etc/fstab at deploy-time. We cannot have / in /etc/fstab for nfsboot deployments because it causes systemd to remount / during bootup, which breaks everything.
* | | Merge branch 'baserock/pedroalvarez/system-integration5'Pedro Alvarez2014-03-203-7/+96
|\ \ \ | | | | | | | | | | | | Reviewed-by: Richard Maw
| * | | Run the system integration commands when buildingPedro Alvarez2014-03-201-0/+46
| | | |
| * | | Generate scripts with the system-integration-commands of chunk morphologies.Pedro Alvarez2014-03-201-3/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The scripts will be created into the folder: /baserock/sytem-integration/ The system integration commands have the following syntax: name: linux kind: chunk ... install-commands: ... system-integration: linux-libs: 00-depmod: - depmod -a 70-more-integration: - touch /baserock/FILE - | for FOLDER in $(ls /) do echo "$FOLDER" done linux-misc: 70-more-integration: - echo "Hello world" In this concrete example, the following files will be created: $DESTDIR/baserock/system-integration/ 00-depmod-linux-libs-0000: #!/bin/sh set -xev depmod -a 70-more-integration-linux-libs-0000 #!/bin/sh set -xev touch /baserock/FILE 70-more-integration-linux-libs-0001 #!/bin/sh set -xev for FOLDER in $(ls /) do echo "$FOLDER" done 70-more-integration-linux-misc-0000 #!/bin/sh set -xev echo "Hello world"
| * | | Use morphlib.fsutils.mount when possiblePedro Alvarez2014-03-181-1/+1
| | | |
| * | | Extend morphlib.fsutils.mount to take a fstypePedro Alvarez2014-03-181-3/+7
| |/ /
* | | Implement morph push and morph pulladamcoldrick/ingest-binaries-v5-rebaseAdam Coldrick2014-03-201-0/+93
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a plugin to implement both `morph push` and `morph pull`. These commands are wrappers around the corresponding git commands push and pull, which also implement the functionality of pushing and pulling large files provided by git-fat. For example, running `morph pull` will pull any commits from the remote branch not on your local branch, and then pull any large files from the separate git-fat/rsync store on the Trove.
* | | Implement morph add-binary using git-fat to store large filesAdam Coldrick2014-03-201-0/+110
| | | | | | | | | | | | | | | | | | | | | Add a plugin which implements the morph add-binary command. This command is used to add large files to a git repository. It sets up the files needed to use git-fat, and then runs `git add` with git-fat initiated.
* | | Make existing morph commands use git-fatAdam Coldrick2014-03-203-1/+22
| | | | | | | | | | | | | | | | | | When cloning a repository, the files stored using git-fat need to be pulled. This situation occurs in `morph branch`, `morph edit`, and `morph checkout`.
* | | Add functionality for doing git commands in a directoryAdam Coldrick2014-03-201-2/+32
|/ / | | | | | | | | | | | | | | | | | | | | | | These commands are: * git fat <init|push|pull> * git pull They are required for the morph add-binary and push/pull plugins. Also make sure that GitDirectory is working in the root directory of the specified git repository, and add some helper functions for handling paths of files in the working tree.
* | deploy: Set status prefix to show which deployment the status is forRichard Maw2014-03-141-53/+69
| |
* | deploy: Make extension output display which it isRichard Maw2014-03-141-1/+2
| | | | | | | | This also makes it obey status prefix
* | Add sysroot write extensionRichard Maw2014-03-141-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | This moves the deployed system to somewhere on the host. Any existing contents of the directory is deleted, so don't try to be clever and deploy a new system on top of / in place of a proper upgrade. It can be used to deploy a chroot, sysroot or container, but its current use is to allow for nested deployments to include another system in itself, since the parent deployment's "$1" is prepended to the sub-deployment's "$2".
* | Add the ability to do nested deploymentsRichard Maw2014-03-141-6/+18
| |
* | Move deploy logic into multiple methodsRichard Maw2014-03-141-14/+28
| | | | | | | | | | Check is now separate from setup, which is now separate from running the commands.
* | Move tempdir creation out of the loopRichard Maw2014-03-141-21/+27
| | | | | | | | | | | | We don't need to remove the whole thing every time, and for nested deployments, we want to keep the directories around, since there will be multiple deploys happening in it concurrently.
* | Patch buildcommand once, rather that once per systemRichard Maw2014-03-141-3/+5
| |
* | Remove ugly loading of old b&m pluginRichard Maw2014-03-141-10/+0
| |
* | Fix copyright date in stagingarea_testsRichard Maw2014-03-141-1/+1
| |
* | Merge branch 'liw/quieten-staging-area'Lars Wirzenius2014-03-133-6/+10
|\ \ | | | | | | | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Adam Coldrick
| * | Reduce spam from setting up staging areaLars Wirzenius2014-03-103-6/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make the message "Installing chunk..." be chatty, i.e., only displayed when the user turns verbosity higher. Most of the time the chunk is already unpacked in the cache, so installing it takes a fraction of a second. Add a new message, at default verbosity, when a chunk needs to be unpacked. This can take a while, so a message is appropriate, so the user knows what is happening.
* | | Give a useful error when attempting to build a cluster morphSam Thursfield2014-03-121-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously this resulted in a confusing traceback. This is a quick fix. I'd rather insert the error in the buildcommand module instead, but that code assumes the source being built is a system in several places before it actually checks the kind. Those would all need to be changed, or the code would need to reworked to call _validate_root_kind() much earlier. The Application.traverse_morphs() method is rather ugly anyway, so I'm happy to add further ugliness to it for the time being.
* | | Fix creating a Source() from a cluster morphologySam Thursfield2014-03-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | The Source.__init__() function assumes that the artifact.split_rules attribute is not None. Rather than complicating that code with error checks, let's make it always be correct. Avoids traceback from Source.__init__() when passing a cluster morph to `morph build`.
* | | Merge branch 'sam/branching-fixes'Sam Thursfield2014-03-113-5/+10
|\ \ \ | |/ / |/| | | | | Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * | Fix `morph petrify` in cases where root repo URL has a trailing /Sam Thursfield2014-03-072-3/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | gitdir._list_work_tree_files() needs to use os.relpath() instead of direct string manipulation to avoid chopping off the first line of every filename in cases where the base gitdir path string includes the trailing /. Unit test updated to catch this.
| * | Raise correct error on `morph checkout|branch` of repo with no morphsSam Thursfield2014-03-071-2/+2
| | |
* | | Fix system branch dirname generation to avoid colonsLars Wirzenius2014-03-062-4/+12
| | |
* | | Fix assert in unit testLars Wirzenius2014-03-061-2/+2
|/ / | | | | | | | | | | We're clearly comparing two values for equality rather than asserting that the type of an object is not False, since that would make no sense at all.
* | Merge branch 'baserock/markdoffman/s10382/add-help-option-v3'Mark Doffman2014-03-041-16/+52
|\ \ | | | | | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
| * | Make '--help' and help subcommand the same.Mark Doffman2014-03-041-16/+52
| | | | | | | | | | | | | | | '--help' when used with a subcommand will show the subcommand help. Do not reflow the help text by using a custom formatter.
* | | deploy: Record deployment information in deployed systemSam Thursfield2014-03-043-2/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit introduces a new requirement: USERS MUST NOT HAVE SENSITIVE DATA IN THEIR ENVIRONMENT. Otherwise it will be leaked into the system. Note that configuration fields with 'PASSWORD' in their name are stripped before writing the /baserock/deployment.meta file, so the OpenStack OS_PASSWORD field is not leaked. We want this so that we can run hooks at upgrade-time in the future. These hooks might need to know how the system was configured and what releaseuu it was. I'm not quite sure how we will define 'release' yet, but by using `git tag` and `git describe` we are able to textually label a time period in the history of the system's source code. We already have the specific SHA1 of definitions.git stored in the system metadata, so this should give us enough to be able to implement specific hooks that work around any awkward upgrade complications we encounter in the future.
* | | deploy: Fix double exception in rawdisk.writeSam Thursfield2014-03-041-1/+2
| | | | | | | | | | | | | | | | | | | | | If the disk image was not yet created then the os.remove() call fails and the original exception gets lost, causing confusion and sadness. Also print status earlier on failure
* | | deploy: Check the --upgrade flag has been used correctly.Sam Thursfield2014-03-046-0/+199
| | | | | | | | | | | | | | | Most write extensions don't handle both initial deployments and upgrades of a system.
* | | deploy: Add optional 'check' extensionsSam Thursfield2014-03-041-1/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A write extension will have various kinds of sanity checks to do before actually performing the write. The current architecture of 'morph deploy' means that several minutes pass between the user starting the command and the write extension actually executing. It would be rage-inducing watching `morph deploy` spend 3 minutes unpacking a system only to then abort due to a silly error such as forgetting the --upgrade switch. Therefore it's better for now to split the sanity checks out into separate extensions that can be run as soon as possible and abort if the write extension is not going to be able to operate. For now this will just be used to validate usage of the --upgrade flag but in future checking connectivity to remote servers and the like should be done here too.
* | | deploy: Honour AUTOSTART in ssh-rsync extensionSam Thursfield2014-03-041-0/+5
| | | | | | | | | | | | | | | Now you can deploy an upgrade, set it to be the default version and reboot into it all with one call to `morph deploy`.
View Lorry Controller Status