Ugly unfinished mess.

13 files changed, 444 insertions, 427 deletions

diff --git a/README b/README
index 186cdf9..8168e10 100644
--- a/README
+++ b/README
@@ -1,14 +1,4 @@
 CIAT-Tester | Continuous Integration and Automated Test: Tester
 ===============================================================
 
-This is for testing system images.  As parameters, it takes
-the URL for the system image to be tested, the testing backend
-to use, and a list of test command files to run.  For example:
-
-    ./test-runner.sh file:///test.img openstack tests/uname.test
-
-Given this it will fetch the image, in this case `test.img`,
-and it will try to test it on openstack.
-
-See the README file for the backend you intend to use, for
-backend-specific information.
+TODO
diff --git a/examples/system.setup b/examples/system.setup
new file mode 100644
index 0000000..2a60e9e
--- /dev/null
+++ b/examples/system.setup
@@ -0,0 +1,19 @@
+- image: file:///home/mdrake/test-font-size.html
+  systems:
+  - name: foo
+    platform: !openstack
+      flavour: m1.medium
+      net_id: 190842934298742749
+      host: dc
+  - name: bar
+    platform: !openstack
+      flavour: m1.large
+      net_id: 190842934298742749
+      host: dc
+- image: file:///home/mdrake/test-writing-mode-table-cell.html
+  systems:
+  - name: baz
+    platform: !openstack
+      flavour: m1.medium
+      net_id: 398739587398579335
+      host: nebula
diff --git a/openstack.setup b/openstack.setup
new file mode 100644
index 0000000..25b489a
--- /dev/null
+++ b/openstack.setup
@@ -0,0 +1,6 @@
+- host: example
+  variables:
+    - OS_USERNAME: user
+    - OS_PASSWORD: password
+    - OS_TENANT_NAME: tennant
+    - OS_AUTH_URL: https://example.net:5000/v2.0
diff --git a/openstack/README b/openstack/README
deleted file mode 100644
index d066b2b..0000000
--- a/openstack/README
+++ /dev/null
@@ -1,35 +0,0 @@
-CIAT-Tester: Openstack backend
-==============================
-
-You should ensure that the openstack environment variables are set
-up correctly before running this, as the nova and glance clients
-require them.
-
-You will need to configure the script for your environment editing
-the config section of `do-tests.sh`.  Specifically you need to set
-up the flavour and network ID to use.
-
-Currently the openstack script assumes that there are floating ips
-available.  It also assumes that the default security group allows
-SSH access.
-
-This script also assumes that the test image's root user's
-authorized_keys contains the id_rsa.pub from the machine/user
-on which you are running CIAT-Tester.
-
-
-Configuring an openstack host
------------------------------
-
-Your openstack host may need some configuration.
-
-### Networking
-
-The following commands may be of use to set up a network.
-
-    neutron net-create test-net
-    neutron subnet-create --name test-subnet --gateway 192.168.1.1 \
-        test-net 192.168.1.0/24
-    neutron router-create test-router
-    neutron router-interface-add test-router test-subnet
-    neutron router-gateway-set test-router ext-net
diff --git a/openstack/do-tests.sh b/openstack/do-tests.sh
deleted file mode 100755
index ec319c7..0000000
--- a/openstack/do-tests.sh
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/bin/bash
-
-# Config
-OS_TEST_SCRIPT="openstack/tester"
-OS_NET_ID="b635e4b7-40f2-4ca2-8d35-644c479f6ba6"
-OS_FLAVOUR="dc1.1x2"
-
-# Inputs
-FILE="$1"
-shift
-TEST_FILES=( "$@" )
-
-# Run the test
-echo "Running: $OS_TEST_SCRIPT --net-id $OS_NET_ID --flavour $OS_FLAVOUR --image-file $FILE ${TEST_FILES[@]}"
-python -u "$OS_TEST_SCRIPT" \
-    --net-id "$OS_NET_ID" \
-    --flavour "$OS_FLAVOUR" \
-    --image-file "$FILE" \
-    "${TEST_FILES[@]}"
diff --git a/openstack/tester b/openstack/tester
deleted file mode 100755
index cf60385..0000000
--- a/openstack/tester
+++ /dev/null
@@ -1,306 +0,0 @@
-#!/usr/bin/env python
-#
-# Copyright 2015 Codethink Ltd
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; version 2 of the License.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-'''openstack/tester
-
-This script tests an image on openstack.
-
-'''
-
-import os
-import pipes
-import socket
-import time
-import uuid
-import yaml
-
-import cliapp
-
-from novaclient import client
-
-class TimeoutError(cliapp.AppException):
-
-    """Error to be raised when a connection waits too long"""
-
-    def __init__(self, msg):
-        super(TimeoutError, self).__init__(msg)
-
-
-def delete_image(hostname):
-    # TODO: Do all this stuff properly with python novaclient
-    # Remove an image from the openstack tenancy
-    print "Deleting %s test disc image" % (hostname)
-    try:
-        cliapp.runcmd(['nova', 'image-delete', hostname])
-    except cliapp.AppException as e:
-        # TODO: Stop assuming that image-delete failed because it was
-        #       already removed
-        print "- Failed"
-        pass
-
-def delete_instance_and_image(hostname):
-    # TODO: Do all this stuff properly with python novaclient
-    # Stop and remove VM, and its image
-    print "Deleting %s test instance" % (hostname)
-    try:
-        cliapp.runcmd(['nova', 'delete', hostname])
-    except cliapp.AppException as e:
-        # TODO: Stop assuming that delete failed because the instance
-        #       wasn't running
-        print "- Failed"
-        pass
-
-    # Sleep for a bit, or nova silently fails to delete the image
-    # TODO: Test whether the image has been deleted in a retry loop.
-    time.sleep(20)
-
-    delete_image(hostname)
-
-
-class DeployedSystemInstance(object):
-
-    def __init__(self, deployment, ip_addr, hostname):
-        self.deployment = deployment
-        self.ip_address = ip_addr
-        self.hostname = hostname
-
-    @property
-    def ssh_host(self):
-        # TODO: Stop assuming we ssh into test instances as root
-        return 'root@{host}'.format(host=self.ip_address)
-
-    def runcmd(self, argv, chdir='.', **kwargs):
-        ssh_cmd = ['ssh', '-o', 'StrictHostKeyChecking=no',
-                   '-o', 'UserKnownHostsFile=/dev/null', self.ssh_host]
-        cmd = ['sh', '-c', 'cd "$1" && shift && exec "$@"', '-', chdir]
-        cmd += argv
-        ssh_cmd.append(' '.join(map(pipes.quote, cmd)))
-        print ssh_cmd
-        return cliapp.runcmd(ssh_cmd, **kwargs)
-
-    def _wait_for_dhcp(self, timeout):
-        '''Block until given hostname resolves successfully.
-
-        Raises TimeoutError if the hostname has not appeared in 'timeout'
-        seconds.
-
-        '''
-        start_time = time.time()
-        while True:
-            try:
-                socket.gethostbyname(self.ip_address)
-                return
-            except socket.gaierror:
-                pass
-            if time.time() > start_time + timeout:
-                raise TimeoutError("Host %s did not appear after %i seconds" %
-                                   (self.ip_address, timeout))
-            time.sleep(0.5)
-
-    def _wait_for_ssh(self, timeout):
-        """Wait until the deployed VM is responding via SSH"""
-        print('Attempting to connect to test instance via SSH:')
-        start_time = time.time()
-        while True:
-            try:
-                self.runcmd(['true'], stdin=None, stdout=None, stderr=None)
-                print('SSH connection established.')
-                return
-            except cliapp.AppException:
-                # TODO: Stop assuming the ssh part of the command is what failed
-                if time.time() > start_time + timeout:
-                    raise TimeoutError("%s sshd did not start after %i seconds"
-                                       % (self.ip_address, timeout))
-                time.sleep(0.5)
-
-    def _wait_for_cloud_init(self, timeout):
-        """Wait until cloud init has resized the disc"""
-        start_time = time.time()
-        while True:
-            try:
-                out = self.runcmd(['sh', '-c',
-                              'test -e "$1" && echo exists || echo does not exist',
-                              '-',
-                              '/root/cloud-init-finished'])
-            except:
-                import traceback
-                traceback.print_exc()
-                raise
-            if out.strip() == 'exists':
-                return
-            if time.time() > start_time + timeout:
-                raise TimeoutError("Disc size not increased after %i seconds"
-                                   % (timeout))
-            time.sleep(3)
-
-    def wait_until_online(self, timeout=180):
-        self._wait_for_dhcp(timeout)
-        self._wait_for_ssh(timeout)
-        #self._wait_for_cloud_init(timeout)
-        print "Test system %s ready to run tests." % (self.hostname)
-
-    def delete(self):
-        delete_instance_and_image(self.hostname)
-
-
-class Deployment(object):
-
-    def __init__(self, net_id, image_file, flavour):
-        self.net_id = net_id
-        self.image_file = image_file
-        self.flavour = flavour
-
-    def deploy(self):
-        hostname = str(uuid.uuid4())
-
-        # Deploy the image to openstack
-        # TODO: Ensure this is cleaned up when something raises an exception
-        print('Uploading image to openstack host')
-        args = ['glance', 'image-create', '--progress',
-                '--name', hostname,
-                '--disk-format', 'raw',
-                '--container-format', 'bare',
-                '--file', self.image_file]
-        output = cliapp.runcmd(args)
-        print(output)
-
-        # Get a novaclient object
-        nc = client.Client(2,
-                           os.environ['OS_USERNAME'],
-                           os.environ['OS_PASSWORD'],
-                           os.environ['OS_TENANT_NAME'],
-                           os.environ['OS_AUTH_URL'])
-
-        # Boot an instance from the image
-        # TODO: Ensure this is cleaned up when something raises an exception
-        # TODO: use python-novaclient
-        print('Booting an instance from the image')
-        args = ['nova', 'boot',
-                '--flavor', self.flavour,
-                '--image', hostname,
-                #'--user-data', '/usr/lib/mason/os-init-script',
-                '--nic', "net-id=%s" % (self.net_id),
-                hostname]
-        output = cliapp.runcmd(args)
-
-        # Print nova boot output, with adminPass line removed
-        output_lines = output.split('\n')
-        for line in output_lines:
-            if line.find('adminPass') != -1:
-                password_line = line
-        output_lines.remove(password_line)
-        output = '\n'.join(output_lines)
-        print output
-
-        # Sleep for a bit, or nova explodes when trying to assign IP address
-        time.sleep(20)
-
-        # Assign a floating IP address
-        print('Requesting a floating IP address')
-        for _ in xrange(5):
-            ip_list = nc.floating_ips.list()
-            free_ip = None
-            for ip in ip_list:
-                if ip.instance_id == None:
-                    free_ip = ip
-                    break
-            else:
-                free_ip = nc.floating_ips.create(
-                              nc.floating_ip_pools.list()[0].name)
-            if free_ip != None:
-                instance = nc.servers.find(name=hostname)
-                # TODO: switch back to cli tool, as python
-                #       approach gave error.
-                instance.add_floating_ip(free_ip)
-                ip_addr = free_ip.ip
-                break
-        else:
-            delete_instance_and_image(hostname)
-            raise cliapp.AppException('Could not get a floating IP')
-
-        # Print the IP address
-        print "IP address for instance %s: %s" % (hostname, ip_addr)
-
-        return DeployedSystemInstance(self, ip_addr, hostname)
-
-class ReleaseApp(cliapp.Application):
-
-    """Cliapp application which handles automatic builds and tests"""
-
-    def add_settings(self):
-        """Add the command line options needed"""
-        group_main = 'Program Options'
-        self.settings.string(['net-id'],
-                             'Openstack network ID',
-                             default=None,
-                             group=group_main)
-        self.settings.string(['image-file'],
-                             'Path to system image to test',
-                             default=None,
-                             group=group_main)
-        self.settings.string(['flavour'],
-                             'Name of flavour to use for instance',
-                             default=None,
-                             group=group_main)
-
-    def run_tests(self, instance, tests):
-        instance.wait_until_online()
-
-        for test in tests:
-            with open(test, 'r') as stream:
-                data = yaml.load(stream)
-
-            if data == None:
-                continue
-            if 'name' not in data:
-                print('Bad test: no name')
-                continue
-            if 'commands' not in data:
-                print('Bad test: no commands')
-                continue
-
-            print('Running test: ' + data['name'])
-            for cmd in data['commands']:
-                print('$ ' + cmd)
-                instance.runcmd(['sh', '-c', cmd], stdout=self.output)
-
-    def deploy_and_test_systems(self, tests):
-        """Run the deployments and tests"""
-
-        deployment = Deployment(self.settings['net-id'],
-                                self.settings['image-file'],
-                                self.settings['flavour'])
-
-        instance = deployment.deploy()
-        try:
-            self.run_tests(instance, tests)
-        finally:
-            instance.delete()
-
-    def process_args(self, args):
-        """Process the command line args and kick off the builds/tests"""
-        for setting in ('net-id', 'image-file', 'flavour'):
-            self.settings.require(setting)
-
-        if len(args) == 0:
-            print('Warning: No tests specified.')
-
-        self.deploy_and_test_systems(args)
-
-
-if __name__ == '__main__':
-    ReleaseApp().run()
diff --git a/test-runner.sh b/test-runner.sh
deleted file mode 100755
index 17309ab..0000000
--- a/test-runner.sh
+++ /dev/null
@@ -1,40 +0,0 @@
-#!/bin/bash
-
-# Usage:
-#  ./test-runner.sh <IMG_URL> <BACKEND> [<TEST> [<TEST> ...]]
-#
-# Example:
-#  ./test-runner.sh file:///test.img openstack tests/uname.test
-
-set -e
-
-# Inputs
-URL="$1"
-FILE="$(basename "$URL")"
-shift
-BACKEND="$1"
-shift
-TEST_FILES=( "$@" )
-
-# Write log header
-echo "==============================================================================="
-echo "Testing image $FILE on $(date)"
-echo "  with test command files: ${TEST_FILES[@]}"
-echo "==============================================================================="
-
-# Fetch the image
-echo "Fetching image $FILE"
-curl -o "$FILE" "$URL"
-
-# Run the test
-echo "Launching $BACKEND test backend"
-"$BACKEND/do-tests.sh" "$FILE" "${TEST_FILES[@]}" 2>&1
-
-# Delete the test image
-echo "Deleting local image $FILE"
-rm "$FILE"
-
-# Write log footer
-echo "==============================================================================="
-echo "$(date): Finished testing: SUCCESS!"
-echo "==============================================================================="
diff --git a/tester b/tester
new file mode 100755
index 0000000..58a04f2
--- /dev/null
+++ b/tester
@@ -0,0 +1,401 @@
+#!/usr/bin/env python
+#
+# Copyright 2015 Codethink Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+'''tester
+
+Test system images.
+
+'''
+
+import os
+import pipes
+import socket
+import time
+import uuid
+import yaml
+import shutil
+import urllib2
+
+import cliapp
+
+from novaclient import client
+
+class Host(object):
+    def __init__(self):
+        self.type = None
+        self.systems = []
+
+    def get_images_for_host(self):
+        images_for_host = []
+        for system in self.systems:
+            images_for_host.append(system.image)
+        return list(set(images_for_host))
+
+    def upload_images(self, files):
+        print('Dummy image upload')
+
+    def delete_images(self, files):
+        print('Dummy image remove')
+
+    def deploy_systems(self, files):
+        print('Dummy deploy systems')
+
+    def cleanup_systems(self, files):
+        print('Dummy cleanup systems')
+
+class OpenstackHost(Host):
+    def __init__(self, name, host_type):
+        self.name = name
+        self.type = host_type
+        self.systems = []
+        self.env_vars = {}
+        print(str(self) + 'New host: ' + name)
+        self.load_host_config()
+
+    def __str__(self):
+     return '[' + self.type + '|' + self.name + '] '
+
+    def load_host_config(self):
+        with open('openstack.setup', 'r') as stream:
+            data = yaml.load(stream)
+            for host in data:
+                if host['host'] != self.name:
+                    continue
+                self.env_vars = { k: v for d in host['variables']
+                                       for k, v in d.items() }
+                print(str(self) + 'Loaded config:')
+                # TODO: Is it only password we want to hide?
+                for var in self.env_vars:
+                    if var in ['OS_PASSWORD', 'OS_USERNAME', 'OS_TENANT_NAME']:
+                        print('  ' + var + ': <hidden>')
+                    else:
+                        print('  ' + var + ': ' + self.env_vars[var])
+                break
+            else:
+                # Not a fatal error, since we might be used for single openstack
+                # host, with environment already set up.
+                print(str(self) + 'WARNING: No config found')
+
+    def set_env_vars(self):
+        for var in self.env_vars:
+            os.environ[var] = self.env_vars[var]
+
+    def remove_env_vars(self):
+        for var in self.env_vars:
+            del os.environ[var]
+
+    def add_system(self, name, image, flavour, net_id):
+        system = OpenstackSystem(name, image, flavour, net_id)
+        self.systems.append(system)
+        return system
+
+    def upload_images(self, files):
+        images = self.get_images_for_host()
+        self.set_env_vars()
+        for image in images:
+            print(str(self) + 'Uploading image: ' + files[image])
+            args = ['glance', 'image-create', '--progress',
+                    '--name', files[image],
+                    '--disk-format', 'raw',
+                    '--container-format', 'bare',
+                    '--file', files[image]]
+            output = cliapp.runcmd(args)
+            print(output)
+        self.remove_env_vars()
+
+    def delete_images(self, files):
+        images = self.get_images_for_host()
+        self.set_env_vars()
+        for image in images:
+            print(str(self) + 'Removing image: ' + files[image])
+            cliapp.runcmd(['nova', 'image-delete', files[image]])
+        self.remove_env_vars()
+
+    def deploy_systems(self, files):
+        self.set_env_vars()
+        for system in self.systems:
+            print(str(self) + 'Deploying system: ' + system.name)
+            system.deploy(files)
+        self.remove_env_vars()
+
+    def cleanup_systems(self, files):
+        self.set_env_vars()
+        print(str(self) + 'Terminating systems:')
+        for system in self.systems:
+            system.terminate()
+        self.remove_env_vars()
+
+class System(object):
+    def __init__(self):
+        self.name = None
+        self.id = None
+        self.instance = None
+        self.ip_addr = None
+
+    def __str__(self):
+     return '[' + self.name + '] '
+
+    def ssh_host(self, user):
+        # TODO: Stop assuming we ssh into test instances as root
+        return '{user}@{host}'.format(host=self.ip_addr, user=user)
+
+    def runcmd(self, argv, user='root', chdir='.', **kwargs):
+        print(str(self) + 'Running command as ' + user + ': ' + argv[-1])
+        ssh_cmd = ['ssh', '-o', 'StrictHostKeyChecking=no',
+                   '-o', 'UserKnownHostsFile=/dev/null', self.ssh_host(user)]
+        cmd = ['sh', '-c', 'cd "$1" && shift && exec "$@"', '-', chdir]
+        cmd += argv
+        ssh_cmd.append(' '.join(map(pipes.quote, cmd)))
+        return cliapp.runcmd(ssh_cmd, **kwargs)
+
+    def wait_for_ssh(self, timeout=180):
+        """Wait until the deployed VM is responding via SSH"""
+        start_time = time.time()
+        while True:
+            try:
+                self.runcmd(['true'], stdin=None, stdout=None, stderr=None)
+                print(str(self) + 'SSH connection established.')
+                return
+            except cliapp.AppException:
+                # TODO: Stop assuming the ssh part of the command is what failed
+                if time.time() > start_time + timeout:
+                    raise TimeoutError("%s sshd did not start after %i seconds"
+                                       % (self.ip_address, timeout))
+                time.sleep(0.5)
+
+class OpenstackSystem(System):
+    def __init__(self, name, image, flavour, net_id):
+        self.name = name
+        self.image = image
+        self.flavour = flavour
+        self.net_id = net_id
+        self.id = str(uuid.uuid4()) + '-' + name
+        self.instance = None
+        self.ip_addr = None
+
+    def deploy(self, files):
+        print(str(self) + 'Launching instance with image ' + files[self.image])
+
+        # Get a novaclient object
+        nc = client.Client(2,
+                           os.environ['OS_USERNAME'],
+                           os.environ['OS_PASSWORD'],
+                           os.environ['OS_TENANT_NAME'],
+                           os.environ['OS_AUTH_URL'])
+
+        args = ['nova', 'boot',
+                '--flavor', self.flavour,
+                '--image', files[self.image],
+                '--nic', 'net-id', self.net_id,
+                self.id]
+        output = cliapp.runcmd(args)
+
+        # Print nova boot output, with adminPass line removed
+        output_lines = output.split('\n')
+        for line in output_lines:
+            if line.find('adminPass') != -1:
+                password_line = line
+        output_lines.remove(password_line)
+        output = '\n'.join(output_lines)
+        print output
+
+        # Sleep for a bit, or nova explodes when trying to assign IP address
+        time.sleep(20)
+
+        print(str(self) + 'Requesting a floating IP address')
+
+        for _ in xrange(5):
+            ip_list = nc.floating_ips.list()
+            free_ip = None
+            for ip in ip_list:
+                if ip.instance_id == None:
+                    free_ip = ip
+                    break
+            else:
+                free_ip = nc.floating_ips.create(
+                              nc.floating_ip_pools.list()[0].name)
+            if free_ip != None:
+                instance = nc.servers.find(name=hostname)
+                # TODO: switch back to cli tool, as python
+                #       approach gave error.
+                instance.add_floating_ip(free_ip)
+                ip_addr = free_ip.ip
+                break
+        else:
+            delete_instance_and_image(hostname)
+            raise cliapp.AppException('Could not get a floating IP')
+
+        print(str(self) + 'Obtained IP address: ' + self.ip_addr)
+
+    def terminate(self):
+        print(str(self) + 'Terminating instance')
+        cliapp.runcmd(['nova', 'delete', self.id])
+
+class OpenstackYAML(yaml.YAMLObject):
+    yaml_tag = u'!openstack'
+    def __init__(self, flavour, host, net_id):
+        self.flavour = flavour
+        self.net_id = net_id
+        self.host = host
+    def __repr__(self):
+        return "%s(flavour=%r, net-id=%r, host=%r)" % (
+            self.__class__.__name__, self.flavour, self.net_id, self.host)
+
+def download_image(url):
+    # TODO: Improve progress indication.
+    path = str(uuid.uuid4()) + '-' + url.rpartition('/')[2]
+    print('[Local] Fetching: ' + url)
+
+    response = urllib2.urlopen(url)
+    CHUNK = 16 * 1024
+    with open(path, 'wb') as f:
+        print('[Local] Progress:'),
+        while True:
+            chunk = response.read(CHUNK)
+            if not chunk: break
+            f.write(chunk)
+            print('-'),
+    print('\n[Local] Saved as: ' + path)
+    return path
+
+class ReleaseApp(cliapp.Application):
+
+    """Cliapp application which handles automatic builds and tests"""
+
+    def add_settings(self):
+        """Add the command line options needed"""
+        group_main = 'Program Options'
+        self.settings.string(['setup'],
+                             'Test setup description file',
+                             default=None,
+                             group=group_main)
+
+    def add_openstack_system(self, image_url, system_info):
+
+        system_name = system_info['name']
+        system_platform = system_info['platform']
+        if system_name in self.systems.keys():
+            print('WARNING: Ignoring duplicate system: ' + system_name)
+            return
+
+        # Create host, if needed
+        host_type = 'openstack'
+        host_name = system_platform.host
+        host_id = 'openstack-' + host_name
+
+        if host_id not in self.hosts.keys():
+            host = OpenstackHost(host_name, host_type)
+            self.hosts[host_id] = host
+
+        # Create system
+        host = self.hosts[host_id]
+        system = host.add_system(system_name, image_url,
+                                 system_platform.flavour,
+                                 system_platform.net_id)
+        self.systems[system_name] = system
+
+    def load_setup(self, path):
+        self.systems = {}
+        self.images = {}
+        self.hosts = {}
+        with open(path, 'r') as stream:
+            data = yaml.load(stream)
+            for d in data:
+                image_url = d['image']
+                self.images[image_url] = download_image(image_url)
+                for s in d['systems']:
+                    if (type(s['platform']).__name__ == 'OpenstackYAML'):
+                        self.add_openstack_system(image_url, s)
+
+    def load_tests(self, tests):
+        self.tests = {}
+
+        for test in tests:
+            print('[Local] Loading test set: ' + test)
+            with open(test, 'r') as stream:
+                data = yaml.load(stream)
+
+            if data == None:
+                continue
+            if not isinstance(data, list):
+                print('Bad test: no array of steps')
+            for step in data:
+                if 'systems' not in step:
+                    print('Bad test step: no systems')
+                    continue
+                if not isinstance(step['systems'], list):
+                    print('Bad test step: systems should be an array')
+                    continue
+                if 'commands' not in step:
+                    print('Bad test step: no commands')
+                    continue
+                if not isinstance(step['commands'], list):
+                    print('Bad test step: commands should be an array')
+                    continue
+                self.tests[test] = data
+
+    def start(self):
+        print('[Local] Upload images to test hosts:')
+        for host in self.hosts:
+            self.hosts[host].upload_images(self.images)
+
+        print('[Local] Deploy systems on test hosts:')
+        for host in self.hosts:
+            self.hosts[host].deploy_systems(self.images)
+
+        print('[Local] Check SSH connection to systems:')
+        for system in self.systems:
+            self.systems[system].wait_for_ssh()
+
+        for test in self.tests:
+            print('[Local] Running test: ' + test)
+            for step in self.tests[test]:
+                for cmd in step['commands']:
+                    for system in step['systems']:
+                        # TODO: Get user for each command from test file
+                        s = self.systems[system]
+                        s.runcmd(['sh', '-c', cmd], 'root')
+
+    def clean_up(self):
+        print('[Local] Clean up systems on remote hosts:')
+        for host in self.hosts:
+            self.hosts[host].cleanup_systems(self.images)
+
+        print('[Local] Remove images from remote hosts:')
+        for host in self.hosts:
+            self.hosts[host].delete_images(self.images)
+
+        print('[Local] Delete local images:')
+        for url in self.images:
+             path = self.images[url]
+             print('[Local] Deleting: ' + path)
+             os.remove(path)
+
+    def process_args(self, args):
+        """Process the command line args and kick off the tests"""
+        self.settings.require('setup')
+
+        if len(args) == 0:
+            print('WARNING: No tests specified.')
+
+        self.load_tests(args)
+        self.load_setup(self.settings['setup'])
+        self.start()
+        self.clean_up()
+
+if __name__ == '__main__':
+    ReleaseApp().run()
diff --git a/tests/baserock-meta.test b/tests/baserock-meta.test
deleted file mode 100644
index b12e5ee..0000000
--- a/tests/baserock-meta.test
+++ /dev/null
@@ -1,5 +0,0 @@
-name: Simple Baserock metadata tests
-commands:
-    - cd /baserock && ls -l | grep "systemd"
-    - pwd
-    - find /baserock | grep "systemd" | xargs cat
diff --git a/tests/multiple.test b/tests/multiple.test
new file mode 100644
index 0000000..b34347d
--- /dev/null
+++ b/tests/multiple.test
@@ -0,0 +1,17 @@
+- systems:
+  - foo
+  - bar
+  commands:
+  - cmd one
+  - cmd two
+- systems:
+  - bar
+  commands:
+  - cmd three
+  - cmd four
+  - cmd five
+- systems:
+  - baz
+  commands:
+  - cmd six
+  - cmd seven
diff --git a/tests/python.test b/tests/python.test
deleted file mode 100644
index 1ead294..0000000
--- a/tests/python.test
+++ /dev/null
@@ -1,4 +0,0 @@
-name: Simple python tests
-commands:
-    - python -c 'print "Hello world!"'
-    - python -c 'print 1+1'
diff --git a/tests/systemd.test b/tests/systemd.test
deleted file mode 100644
index 0ed1b2f..0000000
--- a/tests/systemd.test
+++ /dev/null
@@ -1,4 +0,0 @@
-name: Simple systemd output tests
-commands:
-    - systemctl -l
-    - journalctl
diff --git a/tests/uname.test b/tests/uname.test
deleted file mode 100644
index 25a94b3..0000000
--- a/tests/uname.test
+++ /dev/null
@@ -1,3 +0,0 @@
-name: uname test
-commands:
-    - uname -a