summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Move xfer-hole and recv-hole into extensions/baserock/adamcoldrick/writeexts-in-definitionsAdam Coldrick2015-06-042-0/+0
| | | | Change-Id: I46ed5f3ec85f9662bebac592eff7a6eb6d628f28
* Put recv-hole and xfer-hole into definitionsAdam Coldrick2015-06-042-0/+295
|\ | | | | | | This merge commit adds the history of the two scripts as well.
| * Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-162-6/+4
| | | | | | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
| * xfer-hole: Fix bug in copy_slice_from_file.Lars Wirzenius2014-10-021-2/+8
| | | | | | | | | | os.read is limited to an int in size. copy_slice_from_file was trying to os.read more than that causing an OverflowError.
| * Use $((...))) instead of $(... | bc)Lars Wirzenius2014-09-051-4/+4
| |
| * Document why VBoxManage is run in recv-hole instead of callerLars Wirzenius2014-09-051-0/+25
| |
| * Transfer sparse files faster for kvm, vbox deploymentLars Wirzenius2014-09-052-0/+266
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The KVM and VirtualBox deployments use sparse files for raw disk images. This means they can store a large disk (say, tens or hundreds of gigabytes) without using more disk space than is required for the actual content (e.g., a gigabyte or so for the files in the root filesystem). The kernel and filesystem make the unwritten parts of the disk image look as if they are filled with zero bytes. This is good. However, during deployment those sparse files get transferred as if there really are a lot of zeroes. Those zeroes take a lot of time to transfer. rsync, for example, does not handle large holes efficiently. This change introduces a couple of helper tools (morphlib/xfer-hole and morphlib/recv-hole), which transfer the holes more efficiently. The xfer-hole program reads a file and outputs records like these: DATA 123 binary data (exaclyt 123 bytes and no newline at the end) HOLE 3245 xfer-hole can do this efficiently, without having to read through all the zeroes in the holes, using the SEEK_DATA and SEEK_HOLE arguments to lseek. Using this, the holes take only take a few bytes each, making it possible to transfer a disk image faster. In my benchmarks, transferring a 100G byte disk image took about 100 seconds for KVM, and 220 seconds for VirtualBox (which needs to more work at the receiver to convert the raw disk to a VDI). Both benchmarks were from a VM on my laptop to the laptop itself. The interesting bit here is that the receiver (recv-hole) is simple enough that it can be implemented in a bit of shell script, and the text of the shell script can be run on the remote end by giving it to ssh as a command line argument. This means there is no need to install any special tools on the receiver, which makes using this improvement much simpler.
* Stop extensions/writeexts.py from depending on morphlib and cliappAdam Coldrick2015-06-031-49/+121
| | | | Change-Id: I2a53e250f1782e23f7be20e7a1514c392180ec02
* Put writeexts.py in extensions subdirectoryAdam Coldrick2015-06-031-0/+0
| | | | Change-Id: I6eb8fe69416bbf483ffa8c1c317c8f8cea56ef0e
* Put writeexts.py from morphlib in definitionsAdam Coldrick2015-06-031-0/+624
|\ | | | | | | | | | | This is only used by write extensions, all of which are now in definitions. This merge commit also adds all the history of the writeexts.py file from morphlib.
| * Merge "deploy: Tighten SSH connectivity check"Javier Jardón2015-03-191-1/+5
| |\
| | * deploy: Tighten SSH connectivity checkSam Thursfield2015-03-191-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I accidentally tried to deploy a Baserock upgrade to a Fedora cloud machine. Every SSH command that Morph ran got the following output: Please login as the user "fedora" rather than the user "root". The existing implementation of check_ssh_connectivity() didn't raise an exception, leading to confusing errors further down. The new implementation produces this error: ERROR: Unexpected output from remote machine: Please login as the user "fedora" rather than the user "root". Change-Id: Ida5a82b25d759167aa842194b0d833d0565b4acf
| * | Merge "Use python3 compatible notation for catching exceptions"Javier Jardón2015-03-171-2/+2
| |\ \
| | * | Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-2/+2
| | | | | | | | | | | | | | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
| * | | Merge branch 'jjardon/simple-network-networkd2'Javier Jardón2015-03-160-0/+0
| |\ \ \ | | |/ / | |/| | | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| | * | Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | | | | | | | | | | | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
| * | | Merge "Use the modern way of the GPL copyright header: URL instead real address"Javier Jardón2015-03-161-2/+1
| |\ \ \ | | |/ / | |/| |
| | * | Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | |/ | | | | | | | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
| * | Use python3 compatible notation for octal constantsJavier Jardón2015-03-131-1/+1
| |/ | | | | | | Change-Id: I771c3de9cecda7a503f4d36ae5d9fabc040892e4
| * Merge branch 'jjardon/no_build_depends'Javier Jardón2015-03-060-0/+0
| |\ | | | | | | | | | | | | | | | Reviewed-By: Paul Sherwood <paul.sherwood@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk> Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
| | * Fix copyright yearsSam Thursfield2015-03-041-1/+1
| | |
| * | Fix copyright yearsSam Thursfield2015-02-271-1/+1
| |/
| * Fix Morph producing unbootable systemsSam Thursfield2015-02-261-1/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since the version of btrfs-progs in the Baserock reference system definitions was updated to v3.18.2, Morph has produced unbootable x86 systems. This is down to lack of support for new Btrfs features in SYSLINUX. This patch disables the new features so that deployed systems will boot. Although I generally don't want to have compatibility code in Morph, this patch keeps support for the old mkfs.btrfs (which doesn't support the commandline options that we need to pass to new mkfs.btrfs). This will hopefully save people from suffering 'I need to use new Morph to upgrade my devel system, but new Morph doesn't run on my devel system'.
| * Document KERNEL_ARGS write extension parameterPete Fotheringham2014-12-151-0/+6
| |
| * Whitespace removalPete Fotheringham2014-12-151-8/+8
| |
| * Document BOOTLOADER_INSTALL and BOOTLOADER_CONFIG_FORMAT write extension ↵Pete Fotheringham2014-12-151-3/+2
| | | | | | | | parameters
| * writeexts.py: convert 'mount' to context managerPedro Alvarez2014-12-011-38/+27
| |
| * Split create_local_system on various methodsPedro Alvarez2014-12-011-4/+19
| | | | | | | | | | | | This way we can still use create_local_system to create a raw disk, but also reuse bits of it to be able e.g. to deploy to devices.
| * Don't loop mount when mounting a devicePedro Alvarez2014-12-011-1/+4
| |
| * Add force flag to 'mkfs.btrfs'Pedro Alvarez2014-12-011-1/+1
| | | | | | | | | | | | This way when deploying to a device it will format it without asking the user if the device already has format.
| * Add 'is_device' function to check if we are deploying to a devicePedro Alvarez2014-12-011-0/+11
| |
| * Prevent cliapp from logging env. variables with 'PASSWORD' in their nameSam Thursfield2014-09-191-0/+4
| | | | | | | | | | The upstream cliapp project is not interested in this functionality right now.
| * deploy: Make Python extensions log debug messages to MORPH_LOG_FD by defaultSam Thursfield2014-09-011-1/+24
| | | | | | | | | | | | | | | | | | Previously logging was disabled for Python deploy extensions. We get a lot of useful information for free in the log file by doing this: the environment will be written when the subprocess starts, and if it crashes the full backtrace will be written there too. Subcommand execution with cliapp.runcmd() will also be logged.
| * Merge remote-tracking branch 'origin/baserock/james/writeexts_support_jetson'Richard Maw2014-08-121-1/+2
| |
| * Add support for a device tree to be set using DTB_PATHJames Thomas2014-08-031-0/+21
| |
| * Make bootloader config/install more genericJames Thomas2014-08-031-17/+52
| | | | | | | | | | | | | | | | | | | | | | | | Remove the BOOTLOADER environment variable and instead favour BOOTLOADER_CONFIG_FORMAT to set the desired bootloader format, and BOOTLOADER_INSTALL to set the type of bootloader to install. For example, since u-boot can boot using extlinux.conf files, it's conceivable that someone might want to do CONFIG_FORMAT=extlinux.conf, INSTALL=u-boot. However, for most platforms you would want to set INSTALL to "none"
| * Support setting a different root device using ROOT_DEVICEJames Thomas2014-08-031-2/+5
| |
| * Check for presence of btrfs before trying to use itSam Thursfield2014-06-111-1/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If btrfs is not present in the kernel we end up with strange output like this: Error creating disk image2014-06-10 16:00:40 [devel-system-x86_64-generic][my-raw-disk-image][rawdisk.write]Failure to create disk image at /src/tmp/testdev.img ERROR: Command failed: mount -o loop /src/tmp/testdev.img /src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE mount: mounting /dev/loop0 on /src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE failed: Device or resource busy To avoid this confusing error, Morph should explicitly check first.
| * Use UUID in fstab entries if providedRichard Maw2014-06-041-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes systems use the UUID of the disk in the fstab when there is no pre-existing fstab entry for /. This happens whether the system has an initramfs or not, since it should be harmless, and by this point we're in userland, so can know what the UUIDs are. Passing the UUID to `complete_fstab_for_btrfs_layout` is optional, and defaults to the old behaviour of using /dev/sda, since it is called directly by some write extensions for doing upgrades, and upgrading systems that use an initramfs will be part of a later patch.
| * Make uuid mandatory when calling create_btrfs_system_layoutRichard Maw2014-06-041-1/+1
| | | | | | | | | | | | | | | | | | | | It's only ever called from the core of the write extensions library, so we can change it to be mandatory in all cases. install_extlinux is left with the uuid being optional and defaulting to /dev/sda, since it is called directly from the rawdisk write extension currently, and upgrades of systems that have an initramfs will be part of a later patch series.
| * Add initramfs support to write extensions that produce disksRichard Maw2014-06-021-8/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If INITRAMFS_PATH is specified and the file exists, then the produced kernel command line will use root=UUID=$uuid_of_created_disk rather than root=/dev/sda, which may be incorrect. Help files have been updated to mention the new option. This leads to an unfortunate duplication of the path to the initramfs, in both the location field of the nested deployment and the INITRAMFS_PATH of the disk image creation. However, an initramfs could be produced by a chunk and put in the same place, so it doesn't make sense to couple the rawdisk and initramfs write extensions to remove this duplication. Similarly, there may be multiple valid initramfs in the rootfs e.g. extlinux loads a hypervisor, which is Linux + initramfs, and the initramfs then boots a guest Linux system, which uses a different initramfs. This makes it important to explicitly let the rootfs write extensions know which to use, or not as the case may be. util-linux's blkid is required, since the busybox version ignores the options to filter its output, and parsing the output is undesirable. Because syslinux's btrfs subvolume support is limited to being able to use a non-0 default subvolume, the initramfs has to be copied out of the run-time rootfs subvolume and into the boot subvolume. This pushed the required disk space of a minimal system over the 512M threshold because we do not have the userland tooling support to be able to do a btrfs file contents clone.
| * Fix state subvolume generator to preserve permissionsDaniel Silverstone2014-05-161-1/+1
| | | | | | | | | | | | shutil.move() does not preserve permissions, file modes, ownerships etc, resulting in much confusion when prepopulating a non-root user during deployment. This change to `mv` fixes that.
| * deploy: Share SSH connectivity check in the common writeexts.py codeSam Thursfield2014-04-151-0/+9
| | | | | | | | | | Also, change it to log the real error message in morph.log before raising a more general exception to the user.
| * Adding support to add extra kernel args in extlinux.confPedro Alvarez2014-04-041-1/+5
| |
| * Make parse_autostart() into more general get_environment_boolean()Sam Thursfield2014-03-041-7/+7
| | | | | | | | | | Also, be more flexible when parsing environment booleans -- convert to lower case and match 0/1 and true/false as well as yes/no.
| * deploy: Finish off the Btrfs system layout implementationSam Thursfield2014-03-041-42/+141
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The shared state directories defined in writeexts.py (/var, /home etc.) are now separate Btrfs subvolumes that are mounted in place using fstab. There are some warnings on mounting /var and /srv about the mountpoint not being empty. Not yet investigated. If a configure extension has already added / to the fstab, use the device it chose rather than assuming /dev/sda. This is required for the vdaboot.configure extension that we use for OpenStack deployments. Similarly, if a configure extension has added an entry for a state directory in /etc/fstab already, we don't replace it with a /state/xxx directory. That's only done as a default behaviour.
| * Adding syslinux 'menu.c32' file during the deployment.Pedro Alvarez2014-03-041-1/+19
| | | | | | | | | | We will need this file to enable a bootloader menu to choose between OS after an upgrade.
| * Import morphlib as we are using morphlib.ErrorTiago Gomes2013-07-171-0/+1
| |
| * Allow to set the number of cpus for virtualbox and kvm deployments.Tiago Gomes2013-07-021-0/+4
| |
| * Write extensions: Flush output when using status()Richard Maw2013-06-101-0/+1
| | | | | | | | | | This is required to get real-time output and the timestamps meaning anything useful.
View Lorry Controller Status