diff options
author | Adam Coldrick <adam.coldrick@codethink.co.uk> | 2015-03-19 09:34:58 +0000 |
---|---|---|
committer | Morph (on behalf of Adam Coldrick) <adam.coldrick@codethink.co.uk> | 2015-03-19 09:34:58 +0000 |
commit | 7db4ee53fb5398dd8f4ae8f56778735fe6531178 (patch) | |
tree | 01513d77326acd03b2da356ec2cd7f4761901b6b /morphlib/exts | |
parent | 211d6317d22bace089da58875d280ae5e54d5d54 (diff) | |
download | morph-7db4ee53fb5398dd8f4ae8f56778735fe6531178.tar.gz |
Morph build 2ee8190abe87461992f5b7ed85fe2ee9
System branch: master
Diffstat (limited to 'morphlib/exts')
33 files changed, 847 insertions, 268 deletions
diff --git a/morphlib/exts/add-config-files.configure b/morphlib/exts/add-config-files.configure index 0094cf6b..2cf96fd1 100755 --- a/morphlib/exts/add-config-files.configure +++ b/morphlib/exts/add-config-files.configure @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2013 Codethink Limited +# Copyright (C) 2013,2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # Copy all files located in $SRC_CONFIG_DIR to the image /etc. diff --git a/morphlib/exts/fstab.configure b/morphlib/exts/fstab.configure index a1287ea4..3bbc9102 100755 --- a/morphlib/exts/fstab.configure +++ b/morphlib/exts/fstab.configure @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2013 Codethink Limited +# Copyright (C) 2013,2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # # =*= License: GPL-2 =*= diff --git a/morphlib/exts/initramfs.write b/morphlib/exts/initramfs.write index f8af6d84..1059defa 100755 --- a/morphlib/exts/initramfs.write +++ b/morphlib/exts/initramfs.write @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # # =*= License: GPL-2 =*= diff --git a/morphlib/exts/initramfs.write.help b/morphlib/exts/initramfs.write.help index 29a9d266..54d3ae8c 100644 --- a/morphlib/exts/initramfs.write.help +++ b/morphlib/exts/initramfs.write.help @@ -1,4 +1,19 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + help: | + Create an initramfs for a system by taking an existing system and converting it to the appropriate format. @@ -33,3 +48,8 @@ help: | initramfs: type: initramfs location: boot/initramfs.gz + + Parameters: + + * location: the path where the initramfs will be installed (e.g. + `boot/initramfs.gz`) in the above example diff --git a/morphlib/exts/install-files.configure b/morphlib/exts/install-files.configure index 04dc5f18..58cf373a 100755 --- a/morphlib/exts/install-files.configure +++ b/morphlib/exts/install-files.configure @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2013-2014 Codethink Limited +# Copyright (C) 2013-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. ''' A Morph configuration extension for adding arbitrary files to a system diff --git a/morphlib/exts/install-files.configure.help b/morphlib/exts/install-files.configure.help index eb3aab0c..991c26c8 100644 --- a/morphlib/exts/install-files.configure.help +++ b/morphlib/exts/install-files.configure.help @@ -1,3 +1,17 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + help: | Install a set of files onto a system diff --git a/morphlib/exts/kvm.check b/morphlib/exts/kvm.check index 1bb4007a..62d76453 100755 --- a/morphlib/exts/kvm.check +++ b/morphlib/exts/kvm.check @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,12 +11,12 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''Preparatory checks for Morph 'kvm' write extension''' import cliapp +import os import re import urlparse @@ -43,8 +43,10 @@ class KvmPlusSshCheckExtension(morphlib.writeexts.WriteExtension): ssh_host, vm_name, vm_path = self.check_and_parse_location(location) self.check_ssh_connectivity(ssh_host) + self.check_can_create_file_at_given_path(ssh_host, vm_path) self.check_no_existing_libvirt_vm(ssh_host, vm_name) self.check_extra_disks_exist(ssh_host, self.parse_attach_disks()) + self.check_virtual_networks_are_started(ssh_host) def check_and_parse_location(self, location): '''Check and parse the location argument to get relevant data.''' @@ -73,6 +75,26 @@ class KvmPlusSshCheckExtension(morphlib.writeexts.WriteExtension): 'write extension to deploy upgrades to existing machines.' % (ssh_host, vm_name)) + def check_can_create_file_at_given_path(self, ssh_host, vm_path): + + def check_can_write_to_given_path(): + try: + cliapp.ssh_runcmd(ssh_host, ['touch', vm_path]) + except cliapp.AppException as e: + raise cliapp.AppException("Can't write to location %s on %s" + % (vm_path, ssh_host)) + else: + cliapp.ssh_runcmd(ssh_host, ['rm', vm_path]) + + try: + cliapp.ssh_runcmd(ssh_host, ['test', '-e', vm_path]) + except cliapp.AppException as e: + # vm_path doesn't already exist, so let's test we can write + check_can_write_to_given_path() + else: + raise cliapp.AppException('%s already exists on %s' + % (vm_path, ssh_host)) + def check_extra_disks_exist(self, ssh_host, filename_list): for filename in filename_list: try: @@ -81,4 +103,50 @@ class KvmPlusSshCheckExtension(morphlib.writeexts.WriteExtension): raise cliapp.AppException('Did not find file %s on host %s' % (filename, ssh_host)) + def check_virtual_networks_are_started(self, ssh_host): + + def check_virtual_network_is_started(network_name): + cmd = ['virsh', '-c', 'qemu:///system', 'net-info', network_name] + net_info = cliapp.ssh_runcmd(ssh_host, cmd).split('\n') + + def pretty_concat(lines): + return '\n'.join(['\t%s' % line for line in lines]) + + for line in net_info: + m = re.match('^Active:\W*(\w+)\W*', line) + if m: + break + else: + raise cliapp.AppException( + "Got unexpected output parsing output of `%s':\n%s" + % (' '.join(cmd), pretty_concat(net_info))) + + network_active = m.group(1) == 'yes' + + if not network_active: + raise cliapp.AppException("Network '%s' is not started" + % network_name) + + def name(nic_entry): + if ',' in nic_entry: + # NETWORK_NAME,mac=12:34,model=e1000... + return nic_entry[:nic_entry.find(',')] + else: + return nic_entry # NETWORK_NAME + + if 'NIC_CONFIG' in os.environ: + nics = os.environ['NIC_CONFIG'].split() + + # --network bridge= is used to specify a bridge + # --network user is used to specify a form of NAT + # (see the virt-install(1) man page) + networks = [name(n) for n in nics if not n.startswith('bridge=') + and not n.startswith('user')] + else: + networks = ['default'] + + for network in networks: + check_virtual_network_is_started(network) + + KvmPlusSshCheckExtension().run() diff --git a/morphlib/exts/kvm.write b/morphlib/exts/kvm.write index 16f188b5..0d0c095b 100755 --- a/morphlib/exts/kvm.write +++ b/morphlib/exts/kvm.write @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2012-2014 Codethink Limited +# Copyright (C) 2012-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,11 +11,14 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. -'''A Morph deployment write extension for deploying to KVM+libvirt.''' +'''A Morph deployment write extension for deploying to KVM+libvirt. + +See file kvm.write.help for documentation + +''' import cliapp @@ -30,40 +33,20 @@ import morphlib.writeexts class KvmPlusSshWriteExtension(morphlib.writeexts.WriteExtension): - '''Create a KVM/LibVirt virtual machine during Morph's deployment. - - The location command line argument is the pathname of the disk image - to be created. The user is expected to provide the location argument - using the following syntax: - - kvm+ssh://HOST/GUEST/PATH - - where: - - * HOST is the host on which KVM/LibVirt is running - * GUEST is the name of the guest virtual machine on that host - * PATH is the path to the disk image that should be created, - on that host - - The extension will connect to HOST via ssh to run libvirt's - command line management tools. - - ''' - location_pattern = '^/(?P<guest>[^/]+)(?P<path>/.+)$' def process_args(self, args): if len(args) != 2: raise cliapp.AppException('Wrong number of command line args') - + temp_root, location = args ssh_host, vm_name, vm_path = self.parse_location(location) autostart = self.get_environment_boolean('AUTOSTART') - + fd, raw_disk = tempfile.mkstemp() os.close(fd) self.create_local_system(temp_root, raw_disk) - + try: self.transfer(raw_disk, ssh_host, vm_path) self.create_libvirt_guest(ssh_host, vm_name, vm_path, autostart) @@ -105,7 +88,7 @@ class KvmPlusSshWriteExtension(morphlib.writeexts.WriteExtension): def create_libvirt_guest(self, ssh_host, vm_name, vm_path, autostart): '''Create the libvirt virtual machine.''' - + self.status(msg='Creating libvirt/kvm virtual machine') attach_disks = self.parse_attach_disks() @@ -135,4 +118,3 @@ class KvmPlusSshWriteExtension(morphlib.writeexts.WriteExtension): ['virsh', '--connect', 'qemu:///system', 'autostart', vm_name]) KvmPlusSshWriteExtension().run() - diff --git a/morphlib/exts/kvm.write.help b/morphlib/exts/kvm.write.help index 8b5053a5..812a5309 100644 --- a/morphlib/exts/kvm.write.help +++ b/morphlib/exts/kvm.write.help @@ -1,4 +1,90 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + help: | - The INITRAMFS_PATH option can be used to specify the location of an - initramfs for syslinux to tell Linux to use, rather than booting - the rootfs directly. + + Deploy a Baserock system as a *new* KVM/LibVirt virtual machine. + + Use the `ssh-rsync` write extension to deploy upgrades to an *existing* VM + + Parameters: + + * location: a custom URL scheme of the form `kvm+ssh://HOST/GUEST/PATH`, + where: + * HOST is the name of the host on which KVM/LibVirt is running + * GUEST is the name of the guest VM on that host + * PATH is the path to the disk image that should be created, + on that host. For example, + `kvm+ssh://alice@192.168.122.1/testsys/home/alice/testys.img` where + * `alice@192.168.122.1` is the target host as given to ssh, + **from within the development host** (which may be + different from the target host's normal address); + * `testsys` is the name of the new guest VM'; + * `/home/alice/testys.img` is the pathname of the disk image files + on the target host. + + * HOSTNAME=name: the hostname of the **guest** VM within the network into + which it is being deployed + + * DISK_SIZE=X: the size of the VM's primary virtual hard disk. `X` should + use a suffix of `K`, `M`, or `G` (in upper or lower case) to indicate + kilo-, mega-, or gigabytes. For example, `DISK_SIZE=100G` would create a + 100 gigabyte disk image. **This parameter is mandatory**. + + * RAM_SIZE=X: The amount of RAM that the virtual machine should allocate + for itself from the host. `X` is interpreted in the same was as for + DISK_SIZE`, and defaults to `1G` + + * VCPUS=n: the number of virtual CPUs for the VM. Allowed values 1-32. Do + not use more CPU cores than you have available physically (real cores, no + hyperthreads) + + * INITRAMFS_PATH=path: the location of an initramfs for the bootloader to + tell Linux to use, rather than booting the rootfs directly. + + * AUTOSTART=<VALUE>` - boolean. If it is set, the VM will be started when + it has been deployed. + + * DTB_PATH=path: **(MANDATORY)** for systems that require a device tree + binary - Give the full path (without a leading /) to the location of the + DTB in the built system image . The deployment will fail if `path` does + not exist. + + * BOOTLOADER_INSTALL=value: the bootloader to be installed + **(MANDATORY)** for non-x86 systems + + allowed values = + - 'extlinux' (default) - the extlinux bootloader will + be installed + - 'none' - no bootloader will be installed by `morph deploy`. A + bootloader must be installed manually. This value must be used when + deploying non-x86 systems such as ARM. + + * BOOTLOADER_CONFIG_FORMAT=value: the bootloader format to be used. + If not specified for x86-32 and x86-64 systems, 'extlinux' will be used + + allowed values = + - 'extlinux' + + * KERNEL_ARGS=args: optional additional kernel command-line parameters to + be appended to the default set. The default set is: + + 'rw init=/sbin/init rootfstype=btrfs \ + rootflags=subvol=systems/default/run \ + root=[name or UUID of root filesystem]' + + (See https://www.kernel.org/doc/Documentation/kernel-parameters.txt) + + (See `morph help deploy` for details of how to pass parameters to write + extensions) diff --git a/morphlib/exts/nfsboot.check b/morphlib/exts/nfsboot.check index 806e560a..e273f61c 100755 --- a/morphlib/exts/nfsboot.check +++ b/morphlib/exts/nfsboot.check @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''Preparatory checks for Morph 'nfsboot' write extension''' diff --git a/morphlib/exts/nfsboot.configure b/morphlib/exts/nfsboot.configure index 660d9c39..6a68dc48 100755 --- a/morphlib/exts/nfsboot.configure +++ b/morphlib/exts/nfsboot.configure @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2013-2014 Codethink Limited +# Copyright (C) 2013-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # Remove all networking interfaces. On nfsboot systems, eth0 is set up diff --git a/morphlib/exts/nfsboot.write b/morphlib/exts/nfsboot.write index 8d3d6df7..d928775e 100755 --- a/morphlib/exts/nfsboot.write +++ b/morphlib/exts/nfsboot.write @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2013-2014 Codethink Limited +# Copyright (C) 2013-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,12 +11,21 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''A Morph deployment write extension for deploying to an nfsboot server +*** DO NOT USE *** +- This was written before 'proper' deployment mechanisms were in place +It is unlikely to work at all and will not work correctly + +Use the pxeboot write extension instead + +*** + + + An nfsboot server is defined as a baserock system that has tftp and nfs servers running, the tftp server is exporting the contents of /srv/nfsboot/tftp/ and the user has sufficient permissions to create nfs roots @@ -125,7 +134,7 @@ class NFSBootWriteExtension(morphlib.writeexts.WriteExtension): self.status(msg='Creating destination directories') try: - cliapp.ssh_runcmd('root@%s' % location, + cliapp.ssh_runcmd('root@%s' % location, ['mkdir', '-p', orig_path, run_path]) except cliapp.AppException: raise cliapp.AppException('Could not create dirs %s and %s on %s' @@ -191,4 +200,3 @@ mv "$temp" "$target" NFSBootWriteExtension().run() - diff --git a/morphlib/exts/nfsboot.write.help b/morphlib/exts/nfsboot.write.help index 598b1b23..186c479a 100644 --- a/morphlib/exts/nfsboot.write.help +++ b/morphlib/exts/nfsboot.write.help @@ -1,6 +1,27 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + help: | + *** DO NOT USE *** + - This was written before 'proper' deployment mechanisms were in place. + It is unlikely to work at all, and will not work correctly. + + Use the pxeboot write extension instead + + *** Deploy a system image and kernel to an nfsboot server. - + An nfsboot server is defined as a baserock system that has tftp and nfs servers running, the tftp server is exporting the contents of /srv/nfsboot/tftp/ and the user has sufficient diff --git a/morphlib/exts/openstack.check b/morphlib/exts/openstack.check index edc37cc1..4c21b604 100755 --- a/morphlib/exts/openstack.check +++ b/morphlib/exts/openstack.check @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''Preparatory checks for Morph 'openstack' write extension''' @@ -77,9 +76,16 @@ class OpenStackCheckExtension(morphlib.writeexts.WriteExtension): '--os-password', password, '--os-auth-url', auth_url, 'image-list'] - try: - cliapp.runcmd(cmdline) - except cliapp.AppException: - raise cliapp.AppException('Wrong OpenStack credentals.') + + exit, out, err = cliapp.runcmd_unchecked(cmdline) + + if exit != 0: + if err.startswith('The request you have made requires ' + 'authentication. (HTTP 401)'): + raise cliapp.AppException('Invalid OpenStack credentials.') + else: + raise cliapp.AppException( + 'Failed to connect to OpenStack instance at %s: %s' % + (auth_url, err)) OpenStackCheckExtension().run() diff --git a/morphlib/exts/openstack.write b/morphlib/exts/openstack.write index 516fe367..67e07c18 100755 --- a/morphlib/exts/openstack.write +++ b/morphlib/exts/openstack.write @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2013 - 2014 Codethink Limited +# Copyright (C) 2013-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''A Morph deployment write extension for deploying to OpenStack.''' @@ -28,40 +27,12 @@ import morphlib.writeexts class OpenStackWriteExtension(morphlib.writeexts.WriteExtension): - '''Configure a raw disk image into an OpenStack host. - - The raw disk image is created during Morph's deployment and the - image is deployed in OpenStack using python-glanceclient. - - The location command line argument is the authentification url - of the OpenStack server using the following syntax: - - http://HOST:PORT/VERSION - - where - - * HOST is the host running OpenStack - * PORT is the port which is using OpenStack for authentifications. - * VERSION is the authentification version of OpenStack (Only v2.0 - supported) - - This extension needs in the environment the following variables: - - * OPENSTACK_USER is the username to use in the deployment. - * OPENSTACK_TENANT is the project name to use in the deployment. - * OPENSTACK_IMAGENAME is the name of the image to create. - * OPENSTACK_PASSWORD is the password of the user. - - - The extension will connect to OpenStack using python-glanceclient - to configure a raw image. - - ''' + '''See openstack.write.help for documentation''' def process_args(self, args): if len(args) != 2: raise cliapp.AppException('Wrong number of command line args') - + temp_root, location = args os_params = self.get_openstack_parameters() @@ -69,7 +40,7 @@ class OpenStackWriteExtension(morphlib.writeexts.WriteExtension): fd, raw_disk = tempfile.mkstemp() os.close(fd) self.create_local_system(temp_root, raw_disk) - self.status(msg='Temporary disk image has been created at %s' + self.status(msg='Temporary disk image has been created at %s' % raw_disk) self.set_extlinux_root_to_virtio(raw_disk) @@ -79,8 +50,7 @@ class OpenStackWriteExtension(morphlib.writeexts.WriteExtension): def set_extlinux_root_to_virtio(self, raw_disk): '''Re-configures extlinux to use virtio disks''' self.status(msg='Updating extlinux.conf') - mp = self.mount(raw_disk) - try: + with self.mount(raw_disk) as mp: path = os.path.join(mp, 'extlinux.conf') with open(path) as f: @@ -91,9 +61,6 @@ class OpenStackWriteExtension(morphlib.writeexts.WriteExtension): with open(path, "w") as f: f.write(extlinux_conf) - finally: - self.unmount(mp) - def get_openstack_parameters(self): '''Get the environment variables needed. @@ -124,4 +91,3 @@ class OpenStackWriteExtension(morphlib.writeexts.WriteExtension): self.status(msg='Image configured.') OpenStackWriteExtension().run() - diff --git a/morphlib/exts/openstack.write.help b/morphlib/exts/openstack.write.help new file mode 100644 index 00000000..26983060 --- /dev/null +++ b/morphlib/exts/openstack.write.help @@ -0,0 +1,51 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + +help: | + + Deploy a Baserock system as a *new* OpenStack virtual machine. + (Use the `ssh-rsync` write extension to deploy upgrades to an *existing* + VM) + + Deploys the system to the OpenStack host using python-glanceclient. + + Parameters: + + * location: the authentication url of the OpenStack server using the + following syntax: + + http://HOST:PORT/VERSION + + where + + * HOST is the host running OpenStack + * PORT is the port which is using OpenStack for authentications. + * VERSION is the authentication version of OpenStack (Only v2.0 + supported) + + * OPENSTACK_USER=username: the username to use in the `--os-username` + argument to `glance`. + + * OPENSTACK_TENANT=tenant: the project name to use in the + `--os-tenant-name` argument to `glance`. + + * OPENSTACK_IMAGENAME=imagename: the name of the image to use in the + `--name` argument to `glance`. + + * OPENSTACK_PASSWORD=password: the password of the OpenStack user. (We + recommend passing this on the command-line, rather than setting an + environment variable or storing it in a cluster cluster definition file.) + + (See `morph help deploy` for details of how to pass parameters to write + extensions) diff --git a/morphlib/exts/rawdisk.check b/morphlib/exts/rawdisk.check index acdc4de1..9be0ce91 100755 --- a/morphlib/exts/rawdisk.check +++ b/morphlib/exts/rawdisk.check @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''Preparatory checks for Morph 'rawdisk' write extension''' @@ -33,10 +32,11 @@ class RawdiskCheckExtension(morphlib.writeexts.WriteExtension): location = args[0] upgrade = self.get_environment_boolean('UPGRADE') if upgrade: - if not os.path.isfile(location): - raise cliapp.AppException( - 'Cannot upgrade %s: it is not an existing disk image' % - location) + if not self.is_device(location): + if not os.path.isfile(location): + raise cliapp.AppException( + 'Cannot upgrade %s: it is not an existing disk image' % + location) version_label = os.environ.get('VERSION_LABEL') if version_label is None: @@ -44,9 +44,10 @@ class RawdiskCheckExtension(morphlib.writeexts.WriteExtension): 'VERSION_LABEL was not given. It is required when ' 'upgrading an existing system.') else: - if os.path.exists(location): - raise cliapp.AppException( - 'Target %s already exists. Use `morph upgrade` if you ' - 'want to update an existing image.' % location) + if not self.is_device(location): + if os.path.exists(location): + raise cliapp.AppException( + 'Target %s already exists. Use `morph upgrade` if you ' + 'want to update an existing image.' % location) RawdiskCheckExtension().run() diff --git a/morphlib/exts/rawdisk.write b/morphlib/exts/rawdisk.write index 12db4398..6f2d45ba 100755 --- a/morphlib/exts/rawdisk.write +++ b/morphlib/exts/rawdisk.write @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2012-2014 Codethink Limited +# Copyright (C) 2012-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''A Morph deployment write extension for raw disk images.''' @@ -29,81 +28,77 @@ import morphlib.writeexts class RawDiskWriteExtension(morphlib.writeexts.WriteExtension): - '''Create a raw disk image during Morph's deployment. - - If the image already exists, it is upgraded. - - The location command line argument is the pathname of the disk image - to be created/upgraded. - - ''' + '''See rawdisk.write.help for documentation''' def process_args(self, args): if len(args) != 2: raise cliapp.AppException('Wrong number of command line args') - + temp_root, location = args - if os.path.isfile(location): + upgrade = self.get_environment_boolean('UPGRADE') + + if upgrade: self.upgrade_local_system(location, temp_root) else: try: - self.create_local_system(temp_root, location) - self.status(msg='Disk image has been created at %s' % location) + if not self.is_device(location): + with self.created_disk_image(location): + self.format_btrfs(location) + self.create_system(temp_root, location) + self.status(msg='Disk image has been created at %s' % + location) + else: + self.format_btrfs(location) + self.create_system(temp_root, location) + self.status(msg='System deployed to %s' % location) except Exception: - self.status(msg='Failure to create disk image at %s' % + self.status(msg='Failure to deploy system to %s' % location) - if os.path.exists(location): - os.remove(location) raise def upgrade_local_system(self, raw_disk, temp_root): self.complete_fstab_for_btrfs_layout(temp_root) - mp = self.mount(raw_disk) + with self.mount(raw_disk) as mp: + version_label = self.get_version_label(mp) + self.status(msg='Updating image to a new version with label %s' % + version_label) - version_label = self.get_version_label(mp) - self.status(msg='Updating image to a new version with label %s' % - version_label) + version_root = os.path.join(mp, 'systems', version_label) + os.mkdir(version_root) - version_root = os.path.join(mp, 'systems', version_label) - os.mkdir(version_root) + old_orig = os.path.join(mp, 'systems', 'default', 'orig') + new_orig = os.path.join(version_root, 'orig') + cliapp.runcmd( + ['btrfs', 'subvolume', 'snapshot', old_orig, new_orig]) - old_orig = os.path.join(mp, 'systems', 'default', 'orig') - new_orig = os.path.join(version_root, 'orig') - cliapp.runcmd( - ['btrfs', 'subvolume', 'snapshot', old_orig, new_orig]) + cliapp.runcmd( + ['rsync', '-a', '--checksum', '--numeric-ids', '--delete', + temp_root + os.path.sep, new_orig]) - cliapp.runcmd( - ['rsync', '-a', '--checksum', '--numeric-ids', '--delete', - temp_root + os.path.sep, new_orig]) + self.create_run(version_root) - self.create_run(version_root) + default_path = os.path.join(mp, 'systems', 'default') + if os.path.exists(default_path): + os.remove(default_path) + else: + # we are upgrading and old system that does + # not have an updated extlinux config file + if self.bootloader_config_is_wanted(): + self.generate_bootloader_config(mp) + self.install_bootloader(mp) + os.symlink(version_label, default_path) - default_path = os.path.join(mp, 'systems', 'default') - if os.path.exists(default_path): - os.remove(default_path) - else: - # we are upgrading and old system that does - # not have an updated extlinux config file if self.bootloader_config_is_wanted(): - self.generate_bootloader_config(mp) - self.install_bootloader(mp) - os.symlink(version_label, default_path) - - if self.bootloader_config_is_wanted(): - self.install_kernel(version_root, temp_root) - - self.unmount(mp) + self.install_kernel(version_root, temp_root) def get_version_label(self, mp): version_label = os.environ.get('VERSION_LABEL') if version_label is None: - self.unmount(mp) raise cliapp.AppException('VERSION_LABEL was not given') if os.path.exists(os.path.join(mp, 'systems', version_label)): - self.unmount(mp) raise cliapp.AppException('VERSION_LABEL %s already exists' % version_label) @@ -111,4 +106,3 @@ class RawDiskWriteExtension(morphlib.writeexts.WriteExtension): RawDiskWriteExtension().run() - diff --git a/morphlib/exts/rawdisk.write.help b/morphlib/exts/rawdisk.write.help index 298d441c..52ed73fb 100644 --- a/morphlib/exts/rawdisk.write.help +++ b/morphlib/exts/rawdisk.write.help @@ -1,11 +1,82 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + help: | - Create a raw disk image during Morph's deployment. - - If the image already exists, it is upgraded. - The `location` argument is a pathname to the image to be - created or upgraded. + Write a system produced by Morph to a physical disk, or to a file that can + be used as a virtual disk. The target will be formatted as a single Btrfs + partition, with the system image written to a subvolume in /systems, and + other subvolumes created for /home, /opt, /root, /srv and /var. + + When written to a physical drive, the drive can be used as the boot device + for a 'real' machine. + + When written to a file, the file can be used independently of `morph` to + create virtual machines with KVM / libvirt, OpenStack or, after converting + it to VDI format, VirtualBox. + + `morph deploy` will fail if the file specified by `location` already + exists. + + If used in `morph upgrade`, the rootfs produced by 'morph build' is added + to the existing raw disk image or device as an additional btrfs sub-volume. + `morph upgrade` will fail if the file specified by `location` does not + exist, or is not a Baserock raw disk image. (Most users are unlikely to + need or use this functionality: it is useful mainly for developers working + on the Baserock tools.) + + Parameters: + + * location: the pathname of the disk image to be created/upgraded, or the + path to the physical device. + + * VERSION_LABEL=label - should contain only alpha-numeric + characters and the '-' (hyphen) character. Mandatory if being used with + `morph update` + + * INITRAMFS_PATH=path: the location of an initramfs for the bootloader to + tell Linux to use, rather than booting the rootfs directly. + + * DTB_PATH=path: **(MANDATORY)** for systems that require a device tree + binary - Give the full path (without a leading /) to the location of the + DTB in the built system image . The deployment will fail if `path` does + not exist. + + * BOOTLOADER_INSTALL=value: the bootloader to be installed + **(MANDATORY)** for non-x86 systems + + allowed values = + - 'extlinux' (default) - the extlinux bootloader will + be installed + - 'none' - no bootloader will be installed by `morph deploy`. A + bootloader must be installed manually. This value must be used when + deploying non-x86 systems such as ARM. + + * BOOTLOADER_CONFIG_FORMAT=value: the bootloader format to be used. + If not specified for x86-32 and x86-64 systems, 'extlinux' will be used + + allowed values = + - 'extlinux' + + * KERNEL_ARGS=args: optional additional kernel command-line parameters to + be appended to the default set. The default set is: + + 'rw init=/sbin/init rootfstype=btrfs \ + rootflags=subvol=systems/default/run \ + root=[name or UUID of root filesystem]' + + (See https://www.kernel.org/doc/Documentation/kernel-parameters.txt) - The INITRAMFS_PATH option can be used to specify the location of an - initramfs for syslinux to tell Linux to use, rather than booting - the rootfs directly. + (See `morph help deploy` for details of how to pass parameters to write + extensions) diff --git a/morphlib/exts/set-hostname.configure b/morphlib/exts/set-hostname.configure index e44c5d56..4b2424d8 100755 --- a/morphlib/exts/set-hostname.configure +++ b/morphlib/exts/set-hostname.configure @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2013 Codethink Limited +# Copyright (C) 2013,2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # Set hostname on system from HOSTNAME. diff --git a/morphlib/exts/simple-network.configure b/morphlib/exts/simple-network.configure index b98b202c..61113325 100755 --- a/morphlib/exts/simple-network.configure +++ b/morphlib/exts/simple-network.configure @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2013 Codethink Limited +# Copyright (C) 2013,2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,16 +11,17 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. -'''A Morph deployment configuration extension to handle /etc/network/interfaces +'''A Morph deployment configuration extension to handle network configutation -This extension prepares /etc/network/interfaces with the interfaces specified -during deployment. +This extension prepares /etc/network/interfaces and networkd .network files +in /etc/systemd/network/ with the interfaces specified during deployment. If no network configuration is provided, eth0 will be configured for DHCP -with the hostname of the system. +with the hostname of the system in the case of /etc/network/interfaces. +In the case of networkd, any interface starting by e* will be configured +for DHCP ''' @@ -37,20 +38,74 @@ class SimpleNetworkError(morphlib.Error): class SimpleNetworkConfigurationExtension(cliapp.Application): - '''Configure /etc/network/interfaces + '''Configure /etc/network/interfaces and generate networkd .network files - Reading NETWORK_CONFIG, this extension sets up /etc/network/interfaces. + Reading NETWORK_CONFIG, this extension sets up /etc/network/interfaces + and .network files in /etc/systemd/network/. ''' def process_args(self, args): - network_config = os.environ.get( - "NETWORK_CONFIG", "lo:loopback;eth0:dhcp,hostname=$(hostname)") + network_config = os.environ.get("NETWORK_CONFIG") - self.status(msg="Processing NETWORK_CONFIG=%(nc)s", nc=network_config) + self.rename_networkd_chunk_file(args) - stanzas = self.parse_network_stanzas(network_config) - iface_file = self.generate_iface_file(stanzas) + if network_config is None: + self.generate_default_network_config(args) + else: + self.status(msg="Processing NETWORK_CONFIG=%(nc)s", + nc=network_config) + + stanzas = self.parse_network_stanzas(network_config) + + self.generate_interfaces_file(args, stanzas) + self.generate_networkd_files(args, stanzas) + + def rename_networkd_chunk_file(self, args): + """Rename the 10-dchp.network file generated in the systemd chunk + The systemd chunk will place something in 10-dhcp.network, which will + have higher precedence than anything added in this extension (we + start at 50-*). + + We should check for that file and rename it instead remove it in + case the file is being used by the user. + + Until both the following happen, we should continue to rename that + default config file: + + 1. simple-network.configure is always run when systemd is included + 2. We've been building systems without systemd including that default + networkd config for long enough that nobody should be including + that config file. + """ + file_path = os.path.join(args[0], "etc", "systemd", "network", + "10-dhcp.network") + try: + os.rename(file_path, file_path + ".morph") + self.status(msg="Renaming networkd file from systemd chunk: %(f)s \ + to %(f)s.morph", f=file_path) + except OSError: + pass + + def generate_default_network_config(self, args): + """Generate default network config: DHCP in all the interfaces""" + + default_network_config_interfaces = "lo:loopback;" \ + "eth0:dhcp,hostname=$(hostname)" + default_network_config_networkd = "e*:dhcp" + + stanzas_interfaces = self.parse_network_stanzas( + default_network_config_interfaces) + stanzas_networkd = self.parse_network_stanzas( + default_network_config_networkd) + + self.generate_interfaces_file(args, stanzas_interfaces) + self.generate_networkd_files(args, stanzas_networkd) + + def generate_interfaces_file(self, args, stanzas): + """Generate /etc/network/interfaces file""" + + iface_file = self.generate_iface_file(stanzas) with open(os.path.join(args[0], "etc/network/interfaces"), "w") as f: f.write(iface_file) @@ -83,6 +138,74 @@ class SimpleNetworkConfigurationExtension(cliapp.Application): lines += [""] return "\n".join(lines) + def generate_networkd_files(self, args, stanzas): + """Generate .network files""" + + for i, stanza in enumerate(stanzas, 50): + iface_file = self.generate_networkd_file(stanza) + + if iface_file is None: + continue + + path = os.path.join(args[0], "etc", "systemd", "network", + "%s-%s.network" % (i, stanza['name'])) + + with open(path, "w") as f: + f.write(iface_file) + + def generate_networkd_file(self, stanza): + """Generate an .network file from the provided data.""" + + name = stanza['name'] + itype = stanza['type'] + pairs = stanza['args'].items() + + if itype == "loopback": + return + + lines = ["[Match]"] + lines += ["Name=%s\n" % name] + lines += ["[Network]"] + if itype == "dhcp": + lines += ["DHCP=yes"] + else: + lines += self.generate_networkd_entries(pairs) + + return "\n".join(lines) + + def generate_networkd_entries(self, pairs): + """Generate networkd configuration entries with the other parameters""" + + address = None + netmask = None + gateway = None + lines = [] + for pair in pairs: + if pair[0] == 'address': + address = pair[1] + elif pair[0] == 'netmask': + netmask = pair[1] + elif pair[0] == 'gateway': + gateway = pair[1] + + if address and netmask: + network_suffix = self.convert_net_mask_to_cidr_suffix (netmask); + address_line = address + '/' + str(network_suffix) + lines += ["Address=%s" % address_line] + elif address or netmask: + raise Exception('address and netmask must be specified together') + + if gateway is not None: + lines += ["Gateway=%s" % gateway] + + return lines + + def convert_net_mask_to_cidr_suffix(self, mask): + """Convert dotted decimal form of a subnet mask to CIDR suffix notation + + For example: 255.255.255.0 -> 24 + """ + return sum(bin(int(x)).count('1') for x in mask.split('.')) def parse_network_stanzas(self, config): """Parse a network config environment variable into stanzas. diff --git a/morphlib/exts/ssh-rsync.check b/morphlib/exts/ssh-rsync.check index 11446c28..c3bdfd29 100755 --- a/morphlib/exts/ssh-rsync.check +++ b/morphlib/exts/ssh-rsync.check @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''Preparatory checks for Morph 'ssh-rsync' write extension''' diff --git a/morphlib/exts/ssh-rsync.write b/morphlib/exts/ssh-rsync.write index 2d7258ba..6d596500 100755 --- a/morphlib/exts/ssh-rsync.write +++ b/morphlib/exts/ssh-rsync.write @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2013-2014 Codethink Limited +# Copyright (C) 2013-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''A Morph deployment write extension for upgrading systems over ssh.''' @@ -37,14 +36,8 @@ def ssh_runcmd_ignore_failure(location, command, **kwargs): class SshRsyncWriteExtension(morphlib.writeexts.WriteExtension): - '''Upgrade a running baserock system with ssh and rsync. - - It assumes the system is baserock-based and has a btrfs partition. - - The location command line argument is the 'user@hostname' string - that will be passed to ssh and rsync - - ''' + '''See ssh-rsync.write.help for documentation''' + def find_root_disk(self, location): '''Read /proc/mounts on location to find which device contains "/"''' diff --git a/morphlib/exts/ssh-rsync.write.help b/morphlib/exts/ssh-rsync.write.help new file mode 100644 index 00000000..f3f79ed5 --- /dev/null +++ b/morphlib/exts/ssh-rsync.write.help @@ -0,0 +1,50 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + +help: | + + Upgrade a Baserock system which is already deployed: + - as a KVM/LibVirt, OpenStack or vbox-ssh virtual machine; + - on a Jetson board. + + Copies a binary delta over to the target system and arranges for it + to be bootable. + + The recommended way to use this extension is by calling `morph upgrade`. + Using `morph deploy --upgrade` is deprecated. + + The upgrade will fail if: + - no VM is deployed and running at `location`; + - the target system is not a Baserock system; + - the target's filesystem and its layout are not compatible with that + created by `morph deploy`." + + See also the 'Upgrading a Baserock installation' section of the 'Using + Baserock` page at wiki.baserock.org + http://wiki.baserock.org/devel-with/#index8h2 + + Parameters: + + * location: the 'user@hostname' string that will be used by ssh and rsync. + 'user' will always be `root` and `hostname` the hostname or address of + the system being upgraded. + + * VERSION_LABEL=label - **(MANDATORY)** should contain only alpha-numeric + characters and the '-' (hyphen) character. + + * AUTOSTART=<VALUE>` - boolean. If it is set, the VM will be started when + it has been deployed. + + (See `morph help deploy` for details of how to pass parameters to write + extensions) diff --git a/morphlib/exts/sysroot.check b/morphlib/exts/sysroot.check new file mode 100755 index 00000000..8ed965bd --- /dev/null +++ b/morphlib/exts/sysroot.check @@ -0,0 +1,29 @@ +#!/bin/sh +# Copyright (C) 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see <http://www.gnu.org/licenses/>. + +# Preparatory checks for Morph 'sysroot' write extension + +set -eu + +location="$1" +if [ -d "$location" ]; then + echo >&2 "ERROR: Deployment directory already exists: $location" + exit 1 +fi + +if [ "$UPGRADE" == "yes" ]; then + echo >&2 "ERROR: Cannot upgrade a sysroot deployment" + exit 1 +fi diff --git a/morphlib/exts/sysroot.write b/morphlib/exts/sysroot.write index 1ae4864f..0ad8d630 100755 --- a/morphlib/exts/sysroot.write +++ b/morphlib/exts/sysroot.write @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014,2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,16 +11,13 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # A Morph write extension to deploy to another directory set -eu -# Ensure the target is an empty directory mkdir -p "$2" -find "$2" -mindepth 1 -delete # Move the contents of our source directory to our target # Previously we would (cd "$1" && find -print0 | cpio -0pumd "$absolute_path") diff --git a/morphlib/exts/tar.check b/morphlib/exts/tar.check index cbeaf163..f2304d46 100755 --- a/morphlib/exts/tar.check +++ b/morphlib/exts/tar.check @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # Preparatory checks for Morph 'tar' write extension diff --git a/morphlib/exts/tar.write b/morphlib/exts/tar.write index 333626b5..01b545b4 100755 --- a/morphlib/exts/tar.write +++ b/morphlib/exts/tar.write @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2013 Codethink Limited +# Copyright (C) 2013,2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # A Morph write extension to deploy to a .tar file diff --git a/morphlib/exts/tar.write.help b/morphlib/exts/tar.write.help index f052ac03..b45c61fa 100644 --- a/morphlib/exts/tar.write.help +++ b/morphlib/exts/tar.write.help @@ -1,5 +1,19 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + help: | Create a .tar file of the deployed system. - + The `location` argument is a pathname to the .tar file to be created. diff --git a/morphlib/exts/vdaboot.configure b/morphlib/exts/vdaboot.configure index b88eb3a8..60de925b 100755 --- a/morphlib/exts/vdaboot.configure +++ b/morphlib/exts/vdaboot.configure @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2013 Codethink Limited +# Copyright (C) 2013,2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. # Change the "/" mount point to /dev/vda to use virtio disks. diff --git a/morphlib/exts/virtualbox-ssh.check b/morphlib/exts/virtualbox-ssh.check index 57d54db1..a97f3294 100755 --- a/morphlib/exts/virtualbox-ssh.check +++ b/morphlib/exts/virtualbox-ssh.check @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2014 Codethink Limited +# Copyright (C) 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''Preparatory checks for Morph 'virtualbox-ssh' write extension''' diff --git a/morphlib/exts/virtualbox-ssh.write b/morphlib/exts/virtualbox-ssh.write index 39ea8f86..774f2b4f 100755 --- a/morphlib/exts/virtualbox-ssh.write +++ b/morphlib/exts/virtualbox-ssh.write @@ -1,5 +1,5 @@ #!/usr/bin/python -# Copyright (C) 2012-2014 Codethink Limited +# Copyright (C) 2012-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -11,8 +11,7 @@ # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# with this program. If not, see <http://www.gnu.org/licenses/>. '''A Morph deployment write extension for deploying to VirtualBox via ssh. @@ -20,6 +19,8 @@ VirtualBox is assumed to be running on a remote machine, which is accessed over ssh. The machine gets created, but not started. +See file virtualbox-ssh.write.help for documentation + ''' @@ -36,30 +37,10 @@ import morphlib.writeexts class VirtualBoxPlusSshWriteExtension(morphlib.writeexts.WriteExtension): - '''Create a VirtualBox virtual machine during Morph's deployment. - - The location command line argument is the pathname of the disk image - to be created. The user is expected to provide the location argument - using the following syntax: - - vbox+ssh://HOST/GUEST/PATH - - where: - - * HOST is the host on which VirtualBox is running - * GUEST is the name of the guest virtual machine on that host - * PATH is the path to the disk image that should be created, - on that host - - The extension will connect to HOST via ssh to run VirtualBox's - command line management tools. - - ''' - def process_args(self, args): if len(args) != 2: raise cliapp.AppException('Wrong number of command line args') - + temp_root, location = args ssh_host, vm_name, vdi_path = self.parse_location(location) autostart = self.get_environment_boolean('AUTOSTART') @@ -88,7 +69,7 @@ class VirtualBoxPlusSshWriteExtension(morphlib.writeexts.WriteExtension): def parse_location(self, location): '''Parse the location argument to get relevant data.''' - + x = urlparse.urlparse(location) if x.scheme != 'vbox+ssh': raise cliapp.AppException( @@ -169,11 +150,11 @@ class VirtualBoxPlusSshWriteExtension(morphlib.writeexts.WriteExtension): attach_disks = self.parse_attach_disks() for device_no, disk in enumerate(attach_disks, 1): - cmd = ['storageattach', vm_name, + cmd = ['storageattach', vm_name, '--storagectl', 'SATA Controller', '--port', str(device_no), '--device', '0', - '--type', 'hdd', + '--type', 'hdd', '--medium', disk] commands.append(cmd) @@ -187,20 +168,6 @@ class VirtualBoxPlusSshWriteExtension(morphlib.writeexts.WriteExtension): def get_host_interface(self, ssh_host): host_ipaddr = os.environ.get('HOST_IPADDR') netmask = os.environ.get('NETMASK') - network_config = os.environ.get("NETWORK_CONFIG") - - if network_config is None: - raise cliapp.AppException('NETWORK_CONFIG was not given') - - if "eth0:" not in network_config: - raise cliapp.AppException( - 'NETWORK_CONFIG does not contain ' - 'the eth0 configuration') - - if "eth1:" not in network_config: - raise cliapp.AppException( - 'NETWORK_CONFIG does not contain ' - 'the eth1 configuration') if host_ipaddr is None: raise cliapp.AppException('HOST_IPADDR was not given') @@ -242,4 +209,3 @@ class VirtualBoxPlusSshWriteExtension(morphlib.writeexts.WriteExtension): return iface VirtualBoxPlusSshWriteExtension().run() - diff --git a/morphlib/exts/virtualbox-ssh.write.help b/morphlib/exts/virtualbox-ssh.write.help index 8b5053a5..2dbf988c 100644 --- a/morphlib/exts/virtualbox-ssh.write.help +++ b/morphlib/exts/virtualbox-ssh.write.help @@ -1,4 +1,135 @@ +# Copyright (C) 2014, 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, see <http://www.gnu.org/licenses/>. + help: | - The INITRAMFS_PATH option can be used to specify the location of an - initramfs for syslinux to tell Linux to use, rather than booting - the rootfs directly. + + Deploy a Baserock system as a *new* VirtualBox virtual machine. + (Use the `ssh-rsync` write extension to deploy upgrades to an *existing* + VM) + + Connects to HOST via ssh to run VirtualBox's command line management tools. + + Parameters: + + * location: a custom URL scheme of the form `vbox+ssh://HOST/GUEST/PATH`, + where: + * HOST is the name of the host on which VirtualBox is running + * GUEST is the name of the guest VM on that host + * PATH is the path to the disk image that should be created, + on that host. For example, + `vbox+ssh://alice@192.168.122.1/testsys/home/alice/testys.img` where + * `alice@192.168.122.1` is the target host as given to ssh, + **from within the development host** (which may be + different from the target host's normal address); + * `testsys` is the name of the new guest VM'; + * `/home/alice/testys.img` is the pathname of the disk image files + on the target host. + + * HOSTNAME=name: the hostname of the **guest** VM within the network into + which it is being deployed. + + * DISK_SIZE=X: **(MANDATORY)** the size of the VM's primary virtual hard + disk. `X` should use a suffix of `K`, `M`, or `G` (in upper or lower + case) to indicate kilo-, mega-, or gigabytes. For example, + `DISK_SIZE=100G` would create a 100 gigabyte virtual hard disk. + + * RAM_SIZE=X: The amount of RAM that the virtual machine should allocate + for itself from the host. `X` is interpreted in the same as for + DISK_SIZE, and defaults to `1G`. + + * VCPUS=n: the number of virtual CPUs for the VM. Allowed values 1-32. Do + not use more CPU cores than you have available physically (real cores, + no hyperthreads). + + * INITRAMFS_PATH=path: the location of an initramfs for the bootloader to + tell Linux to use, rather than booting the rootfs directly. + + * DTB_PATH=path: **(MANDATORY)** for systems that require a device tree + binary - Give the full path (without a leading /) to the location of the + DTB in the built system image . The deployment will fail if `path` does + not exist. + + * BOOTLOADER_INSTALL=value: the bootloader to be installed + **(MANDATORY)** for non-x86 systems + + allowed values = + - 'extlinux' (default) - the extlinux bootloader will + be installed + - 'none' - no bootloader will be installed by `morph deploy`. A + bootloader must be installed manually. This value must be used when + deploying non-x86 systems such as ARM. + + * BOOTLOADER_CONFIG_FORMAT=value: the bootloader format to be used. + If not specified for x86-32 and x86-64 systems, 'extlinux' will be used + + allowed values = + - 'extlinux' + + * KERNEL_ARGS=args: optional additional kernel command-line parameters to + be appended to the default set. The default set is: + + 'rw init=/sbin/init rootfstype=btrfs \ + rootflags=subvol=systems/default/run \ + root=[name or UUID of root filesystem]' + + (See https://www.kernel.org/doc/Documentation/kernel-parameters.txt) + + * AUTOSTART=<VALUE> - boolean. If it is set, the VM will be started when + it has been deployed. + + * VAGRANT=<VALUE> - boolean. If it is set, then networking is configured + so that the VM will work with Vagrant. Otherwise networking is + configured to run directly in VirtualBox. + + * HOST_IPADDR=<ip_address> - the IP address of the VM host. + + * NETMASK=<netmask> - the netmask of the VM host. + + * NETWORK_CONFIG=<net_config> - `net_config` is used to set up the VM's + network interfaces. It is a string containing semi-colon separated + 'stanzas' where each stanza provides information about a network + interface. Each stanza is of the form name:type[,arg=value] e.g. + + lo:loopback + eth0:dhcp + eth1:static,address=10.0.0.1,netmask=255.255.0.0 + + An example of the NETWORK_CONFIG parameter (It should be in one line) + + `"lo:loopback;eth0:static,address=192.168.100.2,netmask=255.255.255.0; + eth1:dhcp,hostname=$(hostname)"` + + It is useful to configure one interface to use NAT to give the VM access + to the outside world and another interface to use the Virtual Box host + adapter to allow you to access the Trove from the host machine. + + The NAT interface eth1 is set up to use dhcp, the host-only adapter + interface is configured statically. + + Note: you must give the host-only adapter interface an address that lies + **on the same network** as the host adapter. So if the host adapter has + an IP of 192.168.100.1 eth0 should have an address such as + 192.168.100.42. + + The settings of the host adapter, including its IP can be changed either + in the VirtualBox manager UI + (https://www.virtualbox.org/manual/ch03.html#settings-network) + or via the VBoxManage command line + (https://www.virtualbox.org/manual/ch08.html#idp57572192) + + See Chapter 6 of the VirtualBox User Manual for more information about + virtual networking (https://www.virtualbox.org/manual/ch06.html) + + (See `morph help deploy` for details of how to pass parameters to write + extensions) |