summaryrefslogtreecommitdiff
path: root/morphlib/exts
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'baserock/richardmaw/fix-ssh-rsync-destroying-versions'Richard Maw2014-10-101-77/+105
|\ | | | | | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Jim MacArthur Reviewed-by: Richard Ipsum
| * ssh-rsync: Don't delete version if it existsRichard Maw2014-10-101-77/+105
|/ | | | | | | | | | | | | | | | A quirk in the resource cleanup code meant that if you gave the same version label when deploying a new version, then it would fail, then remove the old version, as it had assumed that it was the one to create those directories. This patch fixes this issue by making short context managers for all the resource allocation, so cleanup is done by walking up the context managers, so only the mount and the temporary directory need to be cleaned up if the `mkdir "$VERSION_ROOT"` fails. I've tested this with a deploy of a version that doesn't already exist, and the version I'm currently running, so I can conclusively say it's fixed that problem.
* deploy: Make ssh-rsync upgrade extension handle unset VERSION_LABELSam Thursfield2014-10-082-2/+7
| | | | | | | | | | | | | | | | It now gives an error message. Previously it would fail with a backtrace like this: 2014-10-08 09:51:37 [systems/genivi-baseline-system-armv7lhf-jetson.morph][self]Removing temporary mounts Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/exts/ssh-rsync.write", line 54, in process_args self.upgrade_remote_system(location, temp_root) File "/src/morph/morphlib/exts/ssh-rsync.write", line 107, in upgrade_remote_system location, ['btrfs', 'subvolume', 'delete', orig_dir]) UnboundLocalError: local variable 'orig_dir' referenced before assignment
* ssh-rsync: gett UUID of the disk before writing fstabPedro Alvarez2014-10-011-2/+5
| | | | | | | | | With this patch, the fstab of the system to be deployed as an upgrade will be conifgured using the UUID of the disk. Now when doing an upgrade is not needed to specify the ROOT_DEVICE.
* Fix up openstack.write extensionbaserock/pedroalvarez/openstack-exts-fixupsPedro Alvarez2014-09-091-1/+0
| | | | | | The openstack.write extension was calling a nonexistent method 'check_location'. This metod was moved to openstack.check in the commit ba7d1d1ed3bad002ce36e5d4adf4e3794625091a.
* Fix openstack write/check exts to pass the testsPedro Alvarez2014-09-092-2/+2
|
* Merge branch 'baserock/pedroalvarez/openstack-check-improvements'Pedro Alvarez2014-09-052-13/+49
|\ | | | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Lars Wirzenius
| * Check OpenStack credentials in openstack.checkbaserock/pedroalvarez/openstack-check-improvementsPedro Alvarez2014-09-052-13/+49
| | | | | | | | | | | | | | | | If the credentials are wrong, then morph will fail before attempting the OpenStack deployment. To achieve that openstack.check will attempt to run `glance image-list`.
* | Fix shell quoting for ssh remote command linesLars Wirzenius2014-09-052-6/+11
| | | | | | | | Found by Richard Maw.
* | Transfer sparse files faster for kvm, vbox deploymentLars Wirzenius2014-09-052-8/+21
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The KVM and VirtualBox deployments use sparse files for raw disk images. This means they can store a large disk (say, tens or hundreds of gigabytes) without using more disk space than is required for the actual content (e.g., a gigabyte or so for the files in the root filesystem). The kernel and filesystem make the unwritten parts of the disk image look as if they are filled with zero bytes. This is good. However, during deployment those sparse files get transferred as if there really are a lot of zeroes. Those zeroes take a lot of time to transfer. rsync, for example, does not handle large holes efficiently. This change introduces a couple of helper tools (morphlib/xfer-hole and morphlib/recv-hole), which transfer the holes more efficiently. The xfer-hole program reads a file and outputs records like these: DATA 123 binary data (exaclyt 123 bytes and no newline at the end) HOLE 3245 xfer-hole can do this efficiently, without having to read through all the zeroes in the holes, using the SEEK_DATA and SEEK_HOLE arguments to lseek. Using this, the holes take only take a few bytes each, making it possible to transfer a disk image faster. In my benchmarks, transferring a 100G byte disk image took about 100 seconds for KVM, and 220 seconds for VirtualBox (which needs to more work at the receiver to convert the raw disk to a VDI). Both benchmarks were from a VM on my laptop to the laptop itself. The interesting bit here is that the receiver (recv-hole) is simple enough that it can be implemented in a bit of shell script, and the text of the shell script can be run on the remote end by giving it to ssh as a command line argument. This means there is no need to install any special tools on the receiver, which makes using this improvement much simpler.
* Add `morph upgrade` command, deprecate `morph deploy --upgrade`Sam Thursfield2014-09-011-2/+2
| | | | | | The arguments to `morph deploy` can get quite long, any way we can make it shorter and clearer is useful. We can also avoid having the strange --no-upgrade flag in future.
* deploy: Check correct usage of --upgrade for rawdisk deploymentsSam Thursfield2014-08-191-0/+21
| | | | | | | | | | This avoids confusion when the user expected to be doing an initial deployment and wasn't aware that a file with the same name as the target already existed. Previously rawdisk.write would try to mount the file and upgrade it. Now we require the user to pass '--upgrade' when they intend to upgrade, as with other deployment extensions.
* Merge remote-tracking branch 'origin/baserock/james/writeexts_support_jetson'baserock/richardmaw/james/writeexts_support_jetsonRichard Maw2014-08-121-3/+4
|
* Check for presence of btrfs before trying to use itSam Thursfield2014-06-114-0/+37
| | | | | | | | | | | | | | | | If btrfs is not present in the kernel we end up with strange output like this: Error creating disk image2014-06-10 16:00:40 [devel-system-x86_64-generic][my-raw-disk-image][rawdisk.write]Failure to create disk image at /src/tmp/testdev.img ERROR: Command failed: mount -o loop /src/tmp/testdev.img /src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE mount: mounting /dev/loop0 on /src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE failed: Device or resource busy To avoid this confusing error, Morph should explicitly check first.
* initramfs.write: create parent directories of locationRichard Maw2014-06-061-2/+2
|
* Add initramfs support to write extensions that produce disksRichard Maw2014-06-023-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If INITRAMFS_PATH is specified and the file exists, then the produced kernel command line will use root=UUID=$uuid_of_created_disk rather than root=/dev/sda, which may be incorrect. Help files have been updated to mention the new option. This leads to an unfortunate duplication of the path to the initramfs, in both the location field of the nested deployment and the INITRAMFS_PATH of the disk image creation. However, an initramfs could be produced by a chunk and put in the same place, so it doesn't make sense to couple the rawdisk and initramfs write extensions to remove this duplication. Similarly, there may be multiple valid initramfs in the rootfs e.g. extlinux loads a hypervisor, which is Linux + initramfs, and the initramfs then boots a guest Linux system, which uses a different initramfs. This makes it important to explicitly let the rootfs write extensions know which to use, or not as the case may be. util-linux's blkid is required, since the busybox version ignores the options to filter its output, and parsing the output is undesirable. Because syslinux's btrfs subvolume support is limited to being able to use a non-0 default subvolume, the initramfs has to be copied out of the run-time rootfs subvolume and into the boot subvolume. This pushed the required disk space of a minimal system over the 512M threshold because we do not have the userland tooling support to be able to do a btrfs file contents clone.
* Add initramfs write extensionRichard Maw2014-06-022-0/+62
| | | | | | | | | | | | | | | This creates a gzipped cpio archive that may be used as an initramfs. It is hard-coded to use gzip to compress the initramfs, since it's the most common way to do it. This is unfortunate, since the busybox gzip utility only allows maximum compression, which is rather slow and doesn't give progress reporting, so you can easily think it's gotten stuck. It's possible to use other compression formats, but they need the kernel to be built with them supported, and in the case of lz4, unusual userland tools to create it, since the version of lz4 supported in the kernel is not what the standard lz4 tools produce.
* Merge remote-tracking branch 'origin/sam/ssh-rsync-check'Sam Thursfield2014-05-232-26/+24
|\ | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * deploy: Do sanity checks earlier in ssh-rsync (upgrade) extensionSam Thursfield2014-05-202-26/+24
| |
* | Fix long lineDaniel Silverstone2014-05-161-1/+2
| |
* | VirtualBox Write Extension: Vagrant supportDaniel Silverstone2014-05-161-7/+15
|/ | | | | | | Add support to the VirtualBox write extension to notice if we are doing a Vagrant Basebox installation and not do the clever network setup we normally do to allow machines to talk to one another since this confuses Vagrant quite a bit if it is left in.
* Add help for install-files.configureRichard Ipsum2014-05-092-28/+60
| | | | | Move the help out of the comment and into a help file, and add a clearer example.
* fix the Virtualbox version checkingPaul Sherwood2014-04-201-2/+2
|
* deploy: Extra checks for KVM deploymentsSam Thursfield2014-04-151-0/+21
| | | | | Test that a VM with the given name does not already exist, and check that the files specified in ATTACH_DISKS do already exist.
* deploy: Share SSH connectivity check in the common writeexts.py codeSam Thursfield2014-04-152-10/+2
| | | | | Also, change it to log the real error message in morph.log before raising a more general exception to the user.
* deploy: Check SSH connection for KVM deployment before startingSam Thursfield2014-04-142-6/+32
| | | | | | Slight duplication is necessary, but it's only a few lines. We could move the duplicated code into the base class in 'morphlib.writeexts' if there was more duplication.
* Add help files for nfsboot, rawdisk and tar.baserock/markdoffman/s10617/add-yaml-help-optionMark Doffman2014-04-013-0/+24
| | | | | | Begin adding help documentation for configuration and write extensions starting with nfsboot.write, rawdisk.write and tar.write.
* Use os.environ[] instead of os.environ.get() to read a required variableSam Thursfield2014-03-241-1/+1
|
* Be consistent with nfsboot_root pathSam Thursfield2014-03-241-3/+4
| | | | | We were building it from a variable in some places and hardcoding it in others; now we build it from a variable everywhere.
* Make sanity checks for nfsboot deployments run earlierSam Thursfield2014-03-182-55/+70
| | | | | | | | | | | Move some code to the '.check' extension to verify that the deployment can happen *before* spending 5 minutes unpacking and configuring the rootfs. This is not a perfect solution yet because when multiple systems are being deployed in a cluster, we do not check all systems and then deploy them all. Instead, we check one, then deploy it, then check the second, etc.
* Improve comment in nfsboot.configureSam Thursfield2014-03-181-1/+3
|
* Remove fstab hack from nfsboot.configureSam Thursfield2014-03-181-4/+1
| | | | | | | | | There is no longer a default /etc/fstab in the Baserock fhs-dirs chunk, and the nfsboot.write extension does not use the default Btrfs system layout so no entry is added for / to /etc/fstab at deploy-time. We cannot have / in /etc/fstab for nfsboot deployments because it causes systemd to remount / during bootup, which breaks everything.
* Add sysroot write extensionRichard Maw2014-03-141-0/+29
| | | | | | | | | | | | This moves the deployed system to somewhere on the host. Any existing contents of the directory is deleted, so don't try to be clever and deploy a new system on top of / in place of a proper upgrade. It can be used to deploy a chroot, sysroot or container, but its current use is to allow for nested deployments to include another system in itself, since the parent deployment's "$1" is prepended to the sub-deployment's "$2".
* deploy: Fix double exception in rawdisk.writeSam Thursfield2014-03-041-1/+2
| | | | | | | If the disk image was not yet created then the os.remove() call fails and the original exception gets lost, causing confusion and sadness. Also print status earlier on failure
* deploy: Check the --upgrade flag has been used correctly.Sam Thursfield2014-03-046-0/+199
| | | | | Most write extensions don't handle both initial deployments and upgrades of a system.
* deploy: Honour AUTOSTART in ssh-rsync extensionSam Thursfield2014-03-041-0/+5
| | | | | Now you can deploy an upgrade, set it to be the default version and reboot into it all with one call to `morph deploy`.
* Make parse_autostart() into more general get_environment_boolean()Sam Thursfield2014-03-043-4/+4
| | | | | Also, be more flexible when parsing environment booleans -- convert to lower case and match 0/1 and true/false as well as yes/no.
* deploy: Always set new system as defaultSam Thursfield2014-03-041-0/+5
|
* deploy: Depend on client OS version manager to deploy upgradesSam Thursfield2014-03-041-91/+41
| | | | | | We now have a OS version manager tool in Baserock (in tbdiff.git). The code to deploy a new base OS version should live there, to minimise duplication between write extensions.
* deploy: Finish off the Btrfs system layout implementationSam Thursfield2014-03-042-0/+4
| | | | | | | | | | | | | | | The shared state directories defined in writeexts.py (/var, /home etc.) are now separate Btrfs subvolumes that are mounted in place using fstab. There are some warnings on mounting /var and /srv about the mountpoint not being empty. Not yet investigated. If a configure extension has already added / to the fstab, use the device it chose rather than assuming /dev/sda. This is required for the vdaboot.configure extension that we use for OpenStack deployments. Similarly, if a configure extension has added an entry for a state directory in /etc/fstab already, we don't replace it with a /state/xxx directory. That's only done as a default behaviour.
* Fix copyright year in previous commitRichard Maw2014-01-281-1/+1
|
* virtualbox-ssh: Work around change in VBox optionsRichard Maw2014-01-281-1/+19
| | | | | | | | | | | | VirtualBox changed a command line option in 4.3 incompatibly, so we now have to check the version number and change an option from --sataportcount to --portcount if the version of VirtualBox running on the target is at least 4.3 This turns the version into a tuple and compares it against another, since it's more reliable than comparing strings, which will count '1.10' as earlier than '1.2', and more convenient than comparing the digits individually.
* Add optional overwrite optionrichardipsum/install_files_modRichard Ipsum2014-01-161-11/+31
| | | | | | | | | | | This option lets the install-files config extension overwrite existing files. A file will only be overwritten if the overwrite flag is specified for that file. Since the overwrite arg is optionally prepended to the manifest line, this patch should not break existing manifests With this patch default config files can be replaced with project specific config files
* Merge remote-tracking branch 'refs/remotes/origin/danielfirth/S9122'Lars Wirzenius2013-11-061-162/+0
|\
| * Removed ssh.configure extensionDan Firth2013-10-081-162/+0
| |
* | Merge remote-tracking branch 'origin/liw/fstab.configure'Lars Wirzenius2013-10-241-0/+40
|\ \ | | | | | | | | | | | | | | | | | | Reviewed-by: Richard Maw At his suggestion, fixed the call to sorted() to be a call to asciibetical().
| * | Add fstab.configureLars Wirzenius2013-10-241-0/+40
|/ / | | | | | | | | This will allow the user to append text to /etc/fstab during a deployment, without having to write custom configuration extensions.
* | Deployment failures will now remove the disk imageDan Firth2013-10-143-3/+10
|/
* exts: Remove tab from vdaboot.configureRichard Maw2013-09-161-1/+1
| | | | | This snuck in since the test suite could not be run when TMPDIR was on a tmpfs.
* exts: Add openstack configure/write extsPedro Alvarez2013-09-112-0/+174
| | | | | | | | | openstackssh.write: Write extension which deploy a raw image of baserock directly to an OpenStack machine using python-glanceclient. The raw image deployed has modified its bootloader to use virtio disks. vdaboot.configure: Configuration extension to change the mount point of "/" to use virtio disks (/dev/vda).
View Lorry Controller Status