diff options
316 files changed, 37190 insertions, 27 deletions
@@ -1 +1,2 @@ *.pyc +*~ @@ -0,0 +1,124 @@ +NEWS for Morph +============== + +This file contains high-level summaries of user-visible changes in +each Morph release. + +Version 14.28, released 2014-07-10 +---------------------------------- + +* Fix failure to update some cached git repos +* Fix and clarify chunk splitting + +Version 14.26, released 2014-06-27 +---------------------------------- + +* Smarter git caching behaviour, cached repos will only be updated when + necessary +* Morph deploy now lets the user specify which systems they want to deploy + within a cluster +* Various bug fixes + +Version 14.24, released 2014-06-13 +---------------------------------- + +* rawdisk deployments check that they have the btrfs module loaded first +* distbuild should busy-wait less +* fetching of artifacts should be atomic, so a failure to fetch the + metadata of an artifact doesn't confuse the build system when we have + the chunk, but no metadata +* `morph deploy` now defaults to `--no-git-update` +* `morph gc` now cleans up failed deployments, so they aren't left around + if morph terminates uncleanly +* `morph edit` now only takes the name of the chunk, rather than the + name of the system and stratum that chunk is in + +Version 14.23, released 2014-06-06 +---------------------------------- + +New feature: + +* Initramfs support + +There have also been a number of fixes to distbuild, and the +`morph copy-artifacts` command has been replaced by `morph list-artifacts`. + +Version 14.22, released 2014-05-29 +---------------------------------- + +New features: + +* VirtualBox deployment now supports Vagrant. See: + <http://wiki.baserock.org/guides/vagrant-basebox/> + +* Additional checks when deploying upgrades with the 'ssh-rsync' extension. + +Additional bug fixes described in the git log. + +Version 14.20, released 2014-05-14 +---------------------------------- + +New features include: + +* New CPU architecture: armv7lhf (ARM hard float). + +* Artifact splitting for chunk and stratum artifacts. + +* Components that can be used to set up a distributed build network of Morph + build workers. + +* Built-in documentation for some extensions, see `morph help-extensions` and + `morph help <extension>`. + +* Nested deployment by `morph deploy`. + +* Support for adding binaries to Git repos when used with Trove. See the `morph + add-binary` and `morph push` commands. + +Many additional changes are described in the Git log. + +Version 13, released 2014-01-10 +------------------------------- + +New features added: + +* New CPU architecture: ppc64 architecture (POWER PC 64-bit). This is + the change specific for Morph. There are changes to the morphologies + (in a different git repository) to actually build such systems. + +* `morph build` and `morph deploy` now allow `.morph` suffixes in + command line arguments. The suffixes are stripped internally, so + Morph behaves as if they suffix wasn't there in the first place. + +* The `morph build` command is now a new implementation. The old + implementation is still available as `morph old-build`, just in case + the new code is buggy, but will be removed in a future release. + Likewise, `morph deploy` has a new implementation, but no + `old-deploy`. Both new implementations should work exactly as the + old ones, except for bugs. + +Bugs fixed: + +* When Morph reads git configuration files, it now correctly handles + whitespace at the end of configuration values. + +* `morph deploy` no longer creates and pushes a temporary build + branch. Pushing it wasn't useful, merely wasteful. + +* `morph deploy` now allows cross-architecture deployments, and + and `morph cross-bootstrap` checks that the system is being built + supports the target architecture. + +Other user-visible changes: + +* When preparing to build (when construcing the build graph), Morph + now reports the ref (SHA1) it uses for each stratum. + +* Systems being built must now have at least one stratum, and the + strata in a system must have at least one chunk that is built using + the normal (staging area) mode, rather than bootstrap mode. + +Version 12, released 2013-11-15 +------------------------------- + +* NEWS file added. @@ -0,0 +1,290 @@ +README for morph +================ + +> **NOTA BENE:** This document is very much work-in-progress, and anything +> and everything may and will change at little or no notice. If you see +> problems, mail baserock-dev@baserock.org. + +`morph` builds binaries for [Baserock](http://www.baserock.org/), +an appliance Linux solution. Please see the website for overall information. + + +Usage +----- + +The Baserock builds are controlled by **morphology** files, +which are build recipes. See below for their syntax. Everything +in Baserock is built from git commits. +Morphologies must be committed in git before building. The `morph` tool is +used to actually run the build. The usual workflow is this: + +* put the morphology for an upstream project with its source code +* put other morphologies in the `morphs` (note plural) repository +* run `morph` to build stuff + +`morph --help` will provide some information, though a full guide is +really required. Meanwhile a short usage to build a disk image: + + morph init workspace + cd workspace + morph checkout baserock:baserock/definitions master + cd master/baserock/baserock/definitions + morph build base-system-x86_64-generic + +For deploying you need to create a cluster morphology. Here is an +example to deploy to a raw disk image. + + name: foo + kind: cluster + systems: + - morph: base-system-x86_64-generic + repo: baserock:baserock/definitions + ref: master + deploy: + my-raw-disk-image: + type: rawdisk + location: /src/tmp/testdev.img + DISK_SIZE: 4G + +To deploy it, you only need to run `morph deploy` with the cluster morphology +created: + + morph deploy foo + +You can write a configuration file to avoid having to write options on +the command line every time. Put it in `~/.morph.conf` and make it look +something like this: + + [config] + cachedir = /home/username/baserock/cache + log = /home/username/baserock/morph.log + log-max = 200M + trove-host = git.baserock.org + +All of the above settings apart from `log` are the defaults, so may be omitted. + + +Morphology file syntax +---------------------- + +YAML is used for the morphology syntax. For example, to build a chunk: + + name: foo + kind: chunk + configure-commands: + - ./configure --prefix="$PREFIX" + build-commands: + - make + test-commands: + - make check + install-commands: + - make DESTDIR="$DESTDIR" install + +For all morphologies, use the following fields: + +* `name`: the name of the morphology; it must currently match the filename + (without the `.morph` suffix); **required** +* `kind`: the kind of thing being built; **required** + +For chunks, use the following fields: + + +* `build-system`: if the program is built using a build system known to + `morph`, you can set this field and avoid having to set the various + `*-commands` fields; the commands that the build system specifies can + be overridden; the following build-systems are known: + + - `autotools` + - `python-distutils` + - `cpan` + - `cmake` + - `qmake` + + optional + +* `pre-configure-commands`: a list of shell commands to run at + the configuration phase of a build, before the list in `configure-commands`; + optional +* `configure-commands`: a list of shell commands to run at the configuraiton + phase of a build; optional +* `post-configure-commands`: a list of shell commands to run at + the configuration phase of a build, after the list in `configure-commands`; + optional + +* `pre-build-commands`: a list of shell commands to run at + the build phase of a build, before the list in `build-commands`; + optional +* `build-commands`: a list of shell commands to run to build (compile) the + project; optional +* `post-build-commands`: a list of shell commands to run at + the build phase of a build, after the list in `build-commands`; + optional + +* `pre-test-commands`: a list of shell commands to run at + the test phase of a build, before the list in `test-commands`; + optional +* `test-commands`: a list of shell commands to run unit tests and other + non-interactive tests on the built but un-installed project; optional +* `post-test-commands`: a list of shell commands to run at + the test phase of a build, after the list in `test-commands`; + optional + +* `pre-install-commands`: a list of shell commands to run at + the install phase of a build, before the list in `install-commands`; + optional +* `install-commands`: a list of shell commands to install the built project; + the install should go into the directory named in the `DESTDIR` environment + variable, not the actual system; optional +* `post-install-commands`: a list of shell commands to run at + the install phase of a build, after the list in `install-commands`; + optional + +* `max-jobs`: a string to be given to `make` as the argument to the `-j` + option to specify the maximum number of parallel jobs; the only sensible + value is `"1"` (including the quotes), to prevent parallel jobs to run + at all; parallel jobs are only used during the `build-commands` phase, + since the other phases are often not safe when run in parallel; `morph` + picks a default value based on the number of CPUs on the host system; + optional + +* `chunks`: a key/value map of lists of regular expressions; + the key is the name + of a binary chunk, the regexps match the pathnames that will be + included in that chunk; the patterns match the pathnames that get installed + by `install-commands` (the whole path below `DESTDIR`); every file must + be matched by at least one pattern; by default, a single chunk gets + created, named according to the morphology, and containing all files; + optional + +For strata, use the following fields: + +* `build-depends`: a list of strings, each of which refers to another + stratum that the current stratum depends on. This list may be omitted + or empty if the stratum does not depend on anything else. +* `chunks`: a list of key/value mappings, where each mapping corresponds + to a chunk to be included in the stratum; the mappings may use the + following keys: `name` is the chunk's name (may be different from the + morphology name), `repo` is the repository in which to find (defaults to + chunk name), `ref` identifies the commit to use (typically a branch + name, but any tree-ish git accepts is ok), and `morph` is the name + of the morphology to use and is optional. In addition to these keys, + each of the sources MUST specify a list of build dependencies using the + `build-depends` field. This field may be omitted to make the source + depend on all other chunks that are listed earlier in the `chunks` + list. The field may be an empty list to indicate that the chunk does + not depend on anything else in the same stratum. To specify one or + more chunk dependencies, `build-depends` needs to be set to a list + that contains the names of chunks that the source depends on in the + same stratum. These names correspond to the values of the `name` + fields of the other chunks. + +For systems, use the following fields: + +* `strata`: a list of names of strata to be included in the system. Unlike + chunks, the stratum morphs must all be in the same Git repository as the + system morphology. The value of the `morph` field will be taken as the + artifact name; if this causes ambiguity then an `alias` may be specified as + well. **required** + +Example chunk (simplified commands): + + name: eglibc + kind: chunk + configure-commands: + - mkdir o + - cd o && ../libc/configure --prefix=/usr + build-commands: + - cd o && make + install-commands: + - cd o && make install_root="$DESTDIR" install + +Example stratum: + + name: foundation + kind: stratum + chunks: + - name: fhs-dirs + repo: upstream:fhs-dirs + ref: baserock/bootstrap + build-depends: [] + - name: linux-api-headers + repo: upstream:linux + ref: baserock/morph + build-depends: + - fhs-dirs + - name: eglibc + repo: upstream:eglibc + ref: baserock/bootstrap + build-depends: + - linux-api-headers + - name: busybox + repo: upstream:busybox + ref: baserock/bootstrap + build-depends: + - fhs-dirs + - linux-api-headers + +Example system: + + name: base + kind: system + strata: + - morph: foundation + - morph: linux-stratum + +Note that currently, unknown keys in morphologies are silently ignored. + + +Build environment +----------------- + +When `morph` runs build commands, it clears the environment of all +variables and creates new ones. This is so that the build will run +more consistently across machines and developers. + +See the `morphlib/buildenvironment.py` file for details on what +environment variables are set. + +Morph also constructs a staging area for every build, composed of its +build-dependencies, so everything that is used for a build is traceable +and reproducible. + + +Hacking morph +------------- + +When running Morph from a Git checkout, remember to set PYTHONPATH to +point to your checkout. This will cause Morph to load the plugins and +write extensions from your checkout correctly. + +Run the test suite with this command: + + ./check --full + +If your /tmp is a tmpfs you may need to set TMPDIR to a different path, +as there are tests for large disk image deploys. + +Install CoverageTestRunner (from <http://liw.fi/coverage-test-runner/>), +and check out the `cmdtest` utility (from <http://liw.fi/cmdtest/>). + +Run the checks before submitting a patch, please. + + +Legalese +-------- + +Copyright (C) 2011-2014 Codethink Limited + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; version 2 of the License. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License along +with this program; if not, write to the Free Software Foundation, Inc., +51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + @@ -0,0 +1,164 @@ +#!/bin/sh +# +# Run test suite for morph. +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +set -e + + +# Parse the command line. + +run_style=false +run_unit_tests=false +run_cmdtests=false +run_slow_cmdtests=false +run_yarns=false +if [ "$#" -eq 0 ]; then + run_style=true + run_unit_tests=true + run_cmdtests=true + run_slow_cmdtests=false + run_yarns=true +fi +while [ "$#" -gt 0 ] +do + case "$1" in + --full) + run_style=true + run_unit_tests=true + run_cmdtests=true + run_slow_cmdtests=true + run_yarns=true + ;; + --style) + run_style=true + ;; + --no-style) + run_style=false + ;; + --unit-tests) + run_unit_tests=true + ;; + --no-unit-tests) + run_unit_tests=false + ;; + --cmdtests) + run_cmdtests=true + ;; + --no-cmdtests) + run_cmdtests=false + ;; + --slow-cmdtests) + run_slow_cmdtests=true + ;; + --no-slow-cmdtests) + run_slow_cmdtests=false + ;; + --yarns) + run_yarns=true + ;; + --no-yarns) + run_yarns=false + ;; + *) echo "ERROR: Unknown argument $1." 1>&2; exit 1 ;; + esac + shift +done + + +# Set PYTHONPATH to start with the current directory so that we always +# find the right version of it for the test suite. + +case "$PYTHONPATH" in + '') PYTHONPATH="$(pwd)" ;; + *) PYTHONPATH="$(pwd):$PYTHONPATH" ;; +esac +export PYTHONPATH + +# Run the style checks + +if "$run_style" && [ -d .git ]; +then + echo "Checking copyright statements" + if ! (git ls-files --cached -z | + xargs -0r scripts/check-copyright-year); then + exit 1 + fi + + echo 'Checking source code for silliness' + if ! (git ls-files --cached | + grep -v '\.gz$' | + grep -Ev 'tests[^/]*/.*\.std(out|err)' | + grep -vF 'tests.build/build-system-autotools.script' | + xargs -r scripts/check-silliness); then + exit 1 + fi +fi + +# Clean up artifacts from previous (possibly failed) runs, build, +# and run the tests. + +if "$run_unit_tests"; then + python setup.py clean check +fi + +# Run scenario tests with yarn, if yarn is available. +# +# Yarn cleans up the environment when it runs tests, and this removes +# PYTHONPATH from the environment. However, we need our tests to have +# the PYTHONPATH, so that we can get them to, for example, use the right +# versions of updated dependencies. The immediate current need is to +# be able to get them to use an updated version of cliapp, but it is +# a general need. +# +# We solve this by using the yarn --env option, allowing us to tell yarn +# explicitly which environment variables to set in addition to the set +# it sets anyway. + +if "$run_yarns" && command -v yarn > /dev/null +then + yarn --env "PYTHONPATH=$PYTHONPATH" -s yarns/morph.shell-lib yarns/*.yarn +fi + +# cmdtest tests. + +HOME="$(pwd)/scripts" + +if "$run_cmdtests" +then + cmdtest tests +else + echo "NOT RUNNING test" +fi + +if "$run_slow_cmdtests" +then + cmdtest tests.branching +else + echo "NOT RUNNING test.branching" +fi + +# Building systems requires the 'filter' parameter of tarfile.TarFile.add(): +# this was introduced in Python 2.7 +if ! "$run_cmdtests"; then + echo "NOT RUNNING tests.build" +elif ! (python --version 2>&1 | grep -q '^Python 2\.[78]'); then + echo "NOT RUNNING tests.build (requires Python 2.7)" +else + cmdtest tests.build +fi diff --git a/distbuild-helper b/distbuild-helper new file mode 100755 index 00000000..cdc1873e --- /dev/null +++ b/distbuild-helper @@ -0,0 +1,328 @@ +#!/usr/bin/python +# +# distbuild-helper -- helper process for Morph distributed building +# +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + +import cliapp +import errno +import fcntl +import httplib +import logging +import os +import signal +import socket +import subprocess +import sys +import time +import urlparse + +import distbuild + + +class FileReadable(object): + + def __init__(self, request_id, p, f): + self.request_id = request_id + self.process = p + self.file = f + + +class FileWriteable(object): + + def __init__(self, request_id, p, f): + self.request_id = request_id + self.process = p + self.file = f + + +class SubprocessEventSource(distbuild.EventSource): + + def __init__(self): + self.procs = [] + self.closed = False + + def get_select_params(self): + r = [] + w = [] + for requst_id, p in self.procs: + if p.stdin_contents is not None: + w.append(p.stdin) + if p.stdout is not None: + r.append(p.stdout) + if p.stderr is not None: + r.append(p.stderr) + return r, w, [], None + + def get_events(self, r, w, x): + events = [] + + for request_id, p in self.procs: + if p.stdin in w: + events.append(FileWriteable(request_id, p, p.stdin)) + if p.stdout in r: + events.append(FileReadable(request_id, p, p.stdout)) + if p.stderr in r: + events.append(FileReadable(request_id, p, p.stderr)) + + return events + + def add(self, request_id, process): + + self.procs.append((request_id, process)) + distbuild.set_nonblocking(process.stdin) + distbuild.set_nonblocking(process.stdout) + distbuild.set_nonblocking(process.stderr) + + def remove(self, process): + self.procs = [t for t in self.procs if t[1] != process] + + def kill_by_id(self, request_id): + logging.debug('SES: Killing all processes for %s', request_id) + for id, process in self.procs: + if id == request_id: + logging.debug('SES: killing %s', repr(process)) + process.kill() + + def close(self): + self.procs = [] + self.closed = True + + def is_finished(self): + return self.closed + + +class HelperMachine(distbuild.StateMachine): + + def __init__(self, conn): + distbuild.StateMachine.__init__(self, 'waiting') + self.conn = conn + self.debug_messages = False + + def setup(self): + distbuild.crash_point() + + jm = self.jm = distbuild.JsonMachine(self.conn) + self.mainloop.add_state_machine(jm) + + p = self.procsrc = SubprocessEventSource() + self.mainloop.add_event_source(p) + + self.send_helper_ready(jm) + + spec = [ + ('waiting', jm, distbuild.JsonNewMessage, 'waiting', self.do), + ('waiting', jm, distbuild.JsonEof, None, self._eofed), + ('waiting', p, FileReadable, 'waiting', self._relay_exec_output), + ('waiting', p, FileWriteable, 'waiting', self._feed_stdin), + ] + self.add_transitions(spec) + + def send_helper_ready(self, jm): + msg = { + 'type': 'helper-ready', + } + jm.send(msg) + logging.debug('HelperMachine: sent: %s', repr(msg)) + + def do(self, parent, event): + distbuild.crash_point() + + logging.debug('JsonMachine: got: %s', repr(event.msg)) + handlers = { + 'http-request': self.do_http_request, + 'exec-request': self.do_exec_request, + 'exec-cancel': self.do_exec_cancel, + } + handler = handlers.get(event.msg['type']) + handler(parent, event.msg) + + def do_http_request(self, parent, msg): + distbuild.crash_point() + + url = msg['url'] + method = msg['method'] + headers = msg['headers'] + body = msg['body'] + assert method in ('HEAD', 'GET', 'POST') + + logging.debug('JsonMachine: http request: %s %s' % (method, url)) + + schema, netloc, path, query, fragment = urlparse.urlsplit(url) + assert schema == 'http' + if query: + path += '?' + query + + try: + conn = httplib.HTTPConnection(netloc) + + if headers: + conn.request(method, path, body, headers) + else: + conn.request(method, path, body) + except (socket.error, httplib.HTTPException), e: + status = 418 # teapot + data = str(e) + else: + res = conn.getresponse() + status = res.status + data = res.read() + conn.close() + + response = { + 'type': 'http-response', + 'id': msg['id'], + 'status': status, + 'body': data, + } + parent.send(response) + logging.debug('JsonMachine: sent to parent: %s', repr(response)) + self.send_helper_ready(parent) + + def do_exec_request(self, parent, msg): + distbuild.crash_point() + + argv = msg['argv'] + stdin_contents = msg.get('stdin_contents', '') + logging.debug('JsonMachine: exec request: argv=%s', repr(argv)) + logging.debug( + 'JsonMachine: exec request: stdin=%s', repr(stdin_contents)) + + p = subprocess.Popen(argv, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + + p.stdin_contents = stdin_contents + + self.procsrc.add(msg['id'], p) + + def do_exec_cancel(self, parent, msg): + distbuild.crash_point() + + self.procsrc.kill_by_id(msg['id']) + + def _relay_exec_output(self, event_source, event): + distbuild.crash_point() + + buf_size = 16 * 1024 + fd = event.file.fileno() + data = os.read(fd, buf_size) + if data: + if event.file == event.process.stdout: + stream = 'stdout' + other = 'stderr' + else: + stream = 'stderr' + other = 'stdout' + msg = { + 'type': 'exec-output', + 'id': event.request_id, + stream: data, + other: '', + } + logging.debug('JsonMachine: sent to parent: %s', repr(msg)) + self.jm.send(msg) + else: + if event.file == event.process.stdout: + event.process.stdout.close() + event.process.stdout = None + else: + event.process.stderr.close() + event.process.stderr = None + + if event.process.stdout == event.process.stderr == None: + event.process.wait() + self.procsrc.remove(event.process) + msg = { + 'type': 'exec-response', + 'id': event.request_id, + 'exit': event.process.returncode, + } + logging.debug('JsonMachine: sent to parent: %s', repr(msg)) + self.jm.send(msg) + self.send_helper_ready(self.jm) + + def _feed_stdin(self, event_source, event): + distbuild.crash_point() + + fd = event.file.fileno() + try: + n = os.write(fd, event.process.stdin_contents) + except os.error, e: + # If other end closed the read end, stop writing. + if e.errno == errno.EPIPE: + logging.debug('JsonMachine: reader closed pipe') + event.process.stdin_contents = '' + else: + raise + else: + logging.debug('JsonMachine: fed %d bytes to stdin', n) + event.process.stdin_contents = event.process.stdin_contents[n:] + if event.process.stdin_contents == '': + logging.debug('JsonMachine: stdin contents finished, closing') + event.file.close() + event.process.stdin_contents = None + + def _eofed(self, event_source, event): + distbuild.crash_point() + logging.info('eof from parent, closing') + event_source.close() + self.procsrc.close() + + +class DistributedBuildHelper(cliapp.Application): + + def add_settings(self): + self.settings.string( + ['parent-address'], + 'address (hostname/ip address) for parent', + metavar='HOSTNAME', + default='localhost') + self.settings.integer( + ['parent-port'], + 'port number for parent', + metavar='PORT', + default=3434) + self.settings.boolean( + ['debug-messages'], + 'log messages that are received?') + self.settings.string_list( + ['crash-condition'], + 'add FILENAME:FUNCNAME:MAXCALLS to list of crash conditions ' + '(this is for testing only)', + metavar='FILENAME:FUNCNAME:MAXCALLS') + + def process_args(self, args): + distbuild.add_crash_conditions(self.settings['crash-condition']) + + # We don't want SIGPIPE, ever. It just kills us. We handle EPIPE + # instead. + signal.signal(signal.SIGPIPE, signal.SIG_IGN) + + addr = self.settings['parent-address'] + port = self.settings['parent-port'] + conn = distbuild.create_socket() + conn.connect((addr, port)) + helper = HelperMachine(conn) + helper.debug_messages = self.settings['debug-messages'] + loop = distbuild.MainLoop() + loop.add_state_machine(helper) + loop.run() + + +DistributedBuildHelper().run() + diff --git a/distbuild/__init__.py b/distbuild/__init__.py new file mode 100644 index 00000000..52ad2cc2 --- /dev/null +++ b/distbuild/__init__.py @@ -0,0 +1,67 @@ +# distbuild/__init__.py -- library for Morph's distributed build plugin +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +from stringbuffer import StringBuffer +from sm import StateMachine +from eventsrc import EventSource +from socketsrc import (SocketError, NewConnection, ListeningSocketEventSource, + SocketReadable, SocketWriteable, SocketEventSource, + set_nonblocking) +from sockbuf import (SocketBufferNewData, SocketBufferEof, + SocketBufferClosed, SocketBuffer) +from mainloop import MainLoop +from sockserv import ListenServer +from jm import JsonMachine, JsonNewMessage, JsonEof + +from serialise import serialise_artifact, deserialise_artifact +from idgen import IdentifierGenerator +from route_map import RouteMap +from timer_event_source import TimerEventSource, Timer +from proxy_event_source import ProxyEventSource +from json_router import JsonRouter +from helper_router import (HelperRouter, HelperRequest, HelperOutput, + HelperResult) +from initiator_connection import (InitiatorConnection, InitiatorDisconnect) +from connection_machine import (ConnectionMachine, InitiatorConnectionMachine, + Reconnect, StopConnecting) +from worker_build_scheduler import (WorkerBuildQueuer, + WorkerConnection, + WorkerBuildRequest, + WorkerCancelPending, + WorkerBuildOutput, + WorkerBuildCaching, + WorkerBuildStepAlreadyStarted, + WorkerBuildWaiting, + WorkerBuildFinished, + WorkerBuildFailed, + WorkerBuildStepStarted) +from build_controller import (BuildController, BuildFailed, BuildProgress, + BuildSteps, BuildStepStarted, + BuildStepAlreadyStarted, BuildOutput, + BuildStepFinished, BuildStepFailed, + BuildFinished, BuildCancel, + build_step_name, map_build_graph) +from initiator import Initiator +from protocol import message + +from crashpoint import (crash_point, add_crash_condition, add_crash_conditions, + clear_crash_conditions) + +from distbuild_socket import create_socket + +__all__ = locals() diff --git a/distbuild/build_controller.py b/distbuild/build_controller.py new file mode 100644 index 00000000..e8a8dc37 --- /dev/null +++ b/distbuild/build_controller.py @@ -0,0 +1,645 @@ +# distbuild/build_controller.py -- control the steps for one build +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import logging +import httplib +import traceback +import urllib +import urlparse +import json + +import distbuild + + +# Artifact build states +UNKNOWN = 'unknown' +UNBUILT = 'not-built' +BUILDING = 'building' +BUILT = 'built' + + +class _Start(object): pass +class _Annotated(object): pass +class _Built(object): pass + +class _AnnotationFailed(object): + + def __init__(self, http_status_code, error_msg): + self.http_status_code = http_status_code + self.error_msg = error_msg + +class _GotGraph(object): + + def __init__(self, artifact): + self.artifact = artifact + + +class _GraphFailed(object): + + pass + + +class BuildCancel(object): + + def __init__(self, id): + self.id = id + + +class BuildFinished(object): + + def __init__(self, request_id, urls): + self.id = request_id + self.urls = urls + + +class BuildFailed(object): + + def __init__(self, request_id, reason): + self.id = request_id + self.reason = reason + + +class BuildProgress(object): + + def __init__(self, request_id, message_text): + self.id = request_id + self.message_text = message_text + + +class BuildSteps(object): + + def __init__(self, request_id, artifact): + self.id = request_id + self.artifact = artifact + + +class BuildStepStarted(object): + + def __init__(self, request_id, step_name, worker_name): + self.id = request_id + self.step_name = step_name + self.worker_name = worker_name + +class BuildStepAlreadyStarted(BuildStepStarted): + + def __init__(self, request_id, step_name, worker_name): + super(BuildStepAlreadyStarted, self).__init__( + request_id, step_name, worker_name) + +class BuildOutput(object): + + def __init__(self, request_id, step_name, stdout, stderr): + self.id = request_id + self.step_name = step_name + self.stdout = stdout + self.stderr = stderr + + +class BuildStepFinished(object): + + def __init__(self, request_id, step_name): + self.id = request_id + self.step_name = step_name + + +class BuildStepFailed(object): + + def __init__(self, request_id, step_name): + self.id = request_id + self.step_name = step_name + + +class _Abort(object): + + pass + + +def build_step_name(artifact): + '''Return user-comprehensible name for a given artifact.''' + return artifact.name + + +def map_build_graph(artifact, callback): + result = [] + done = set() + queue = [artifact] + while queue: + a = queue.pop() + if a not in done: + result.append(callback(a)) + queue.extend(a.dependencies) + done.add(a) + return result + + +class BuildController(distbuild.StateMachine): + + '''Control one build-request fulfillment. + + The initiator sends a build-request message, which causes the + InitiatorConnection to instantiate this class to control the steps + needed to fulfill the request. This state machine builds the + build graph to determine all the artifacts that need building, then + builds anything that is not cached. + + ''' + + _idgen = distbuild.IdentifierGenerator('BuildController') + + def __init__(self, initiator_connection, build_request_message, + artifact_cache_server, morph_instance): + distbuild.crash_point() + distbuild.StateMachine.__init__(self, 'init') + self._initiator_connection = initiator_connection + self._request = build_request_message + self._artifact_cache_server = artifact_cache_server + self._morph_instance = morph_instance + self._helper_id = None + self.debug_transitions = False + self.debug_graph_state = False + + def __repr__(self): + return '<BuildController at 0x%x, request-id %s>' % (id(self), + self._request['id']) + + def setup(self): + distbuild.crash_point() + + spec = [ + # state, source, event_class, new_state, callback + ('init', self, _Start, 'graphing', self._start_graphing), + ('init', self._initiator_connection, + distbuild.InitiatorDisconnect, None, None), + + ('graphing', distbuild.HelperRouter, distbuild.HelperOutput, + 'graphing', self._maybe_collect_graph), + ('graphing', distbuild.HelperRouter, distbuild.HelperResult, + 'graphing', self._maybe_finish_graph), + ('graphing', self, _GotGraph, + 'annotating', self._start_annotating), + ('graphing', self, _GraphFailed, None, None), + ('graphing', self._initiator_connection, + distbuild.InitiatorDisconnect, None, None), + + ('annotating', distbuild.HelperRouter, distbuild.HelperResult, + 'annotating', self._maybe_handle_cache_response), + ('annotating', self, _AnnotationFailed, None, + self._notify_annotation_failed), + ('annotating', self, _Annotated, 'building', + self._queue_worker_builds), + ('annotating', self._initiator_connection, + distbuild.InitiatorDisconnect, None, None), + + # The exact WorkerConnection that is doing our building changes + # from build to build. We must listen to all messages from all + # workers, and choose whether to change state inside the callback. + # (An alternative would be to manage a set of temporary transitions + # specific to WorkerConnection instances that our currently + # building for us, but the state machines are not intended to + # behave that way). + + ('building', distbuild.WorkerConnection, + distbuild.WorkerBuildStepStarted, 'building', + self._maybe_relay_build_step_started), + ('building', distbuild.WorkerConnection, + distbuild.WorkerBuildOutput, 'building', + self._maybe_relay_build_output), + ('building', distbuild.WorkerConnection, + distbuild.WorkerBuildCaching, 'building', + self._maybe_relay_build_caching), + ('building', distbuild.WorkerConnection, + distbuild.WorkerBuildStepAlreadyStarted, 'building', + self._maybe_relay_build_step_already_started), + ('building', distbuild.WorkerConnection, + distbuild.WorkerBuildWaiting, 'building', + self._maybe_relay_build_waiting_for_worker), + ('building', distbuild.WorkerConnection, + distbuild.WorkerBuildFinished, 'building', + self._maybe_check_result_and_queue_more_builds), + ('building', distbuild.WorkerConnection, + distbuild.WorkerBuildFailed, 'building', + self._maybe_notify_build_failed), + ('building', self, _Abort, None, None), + ('building', self, _Built, None, self._notify_build_done), + ('building', distbuild.InitiatorConnection, + distbuild.InitiatorDisconnect, 'building', + self._maybe_notify_initiator_disconnected), + ] + self.add_transitions(spec) + + self.mainloop.queue_event(self, _Start()) + + def _start_graphing(self, event_source, event): + distbuild.crash_point() + + logging.info('Start constructing build graph') + self._artifact_data = distbuild.StringBuffer() + self._artifact_error = distbuild.StringBuffer() + argv = [ + self._morph_instance, + 'serialise-artifact', + '--quiet', + self._request['repo'], + self._request['ref'], + self._request['morphology'], + ] + msg = distbuild.message('exec-request', + id=self._idgen.next(), + argv=argv, + stdin_contents='') + self._helper_id = msg['id'] + req = distbuild.HelperRequest(msg) + self.mainloop.queue_event(distbuild.HelperRouter, req) + + progress = BuildProgress(self._request['id'], 'Computing build graph') + self.mainloop.queue_event(BuildController, progress) + + def _maybe_collect_graph(self, event_source, event): + distbuild.crash_point() + + if event.msg['id'] == self._helper_id: + self._artifact_data.add(event.msg['stdout']) + self._artifact_error.add(event.msg['stderr']) + + def _maybe_finish_graph(self, event_source, event): + distbuild.crash_point() + + def notify_failure(msg_text): + logging.error('Graph creation failed: %s' % msg_text) + + failed = BuildFailed( + self._request['id'], + 'Failed to compute build graph: %s' % msg_text) + self.mainloop.queue_event(BuildController, failed) + + self.mainloop.queue_event(self, _GraphFailed()) + + def notify_success(artifact): + logging.debug('Graph is finished') + + progress = BuildProgress( + self._request['id'], 'Finished computing build graph') + self.mainloop.queue_event(BuildController, progress) + + build_steps = BuildSteps(self._request['id'], artifact) + self.mainloop.queue_event(BuildController, build_steps) + + self.mainloop.queue_event(self, _GotGraph(artifact)) + + if event.msg['id'] == self._helper_id: + self._helper_id = None + + error_text = self._artifact_error.peek() + if event.msg['exit'] != 0 or error_text: + notify_failure('Problem with serialise-artifact: %s' + % error_text) + + if event.msg['exit'] != 0: + return + + text = self._artifact_data.peek() + try: + artifact = distbuild.deserialise_artifact(text) + except ValueError, e: + logging.error(traceback.format_exc()) + notify_failure(str(e)) + return + + notify_success(artifact) + + def _start_annotating(self, event_source, event): + distbuild.crash_point() + + self._artifact = event.artifact + self._helper_id = self._idgen.next() + artifact_names = [] + + def set_state_and_append(artifact): + artifact.state = UNKNOWN + artifact_names.append(artifact.basename()) + + map_build_graph(self._artifact, set_state_and_append) + + url = urlparse.urljoin(self._artifact_cache_server, '/1.0/artifacts') + msg = distbuild.message('http-request', + id=self._helper_id, + url=url, + headers={'Content-type': 'application/json'}, + body=json.dumps(artifact_names), + method='POST') + + request = distbuild.HelperRequest(msg) + self.mainloop.queue_event(distbuild.HelperRouter, request) + logging.debug('Made cache request for state of artifacts ' + '(helper id: %s)' % self._helper_id) + + def _maybe_handle_cache_response(self, event_source, event): + + def set_status(artifact): + is_in_cache = cache_state[artifact.basename()] + artifact.state = BUILT if is_in_cache else UNBUILT + + if self._helper_id != event.msg['id']: + return # this event is not for us + + logging.debug('Got cache response: %s' % repr(event.msg)) + + http_status_code = event.msg['status'] + error_msg = event.msg['body'] + + if http_status_code != httplib.OK: + logging.debug('Cache request failed with status: %s' + % event.msg['status']) + self.mainloop.queue_event(self, + _AnnotationFailed(http_status_code, error_msg)) + return + + cache_state = json.loads(event.msg['body']) + map_build_graph(self._artifact, set_status) + self.mainloop.queue_event(self, _Annotated()) + + count = sum(map_build_graph(self._artifact, + lambda a: 1 if a.state == UNBUILT else 0)) + + progress = BuildProgress( + self._request['id'], + 'Need to build %d artifacts' % count) + self.mainloop.queue_event(BuildController, progress) + + if count == 0: + logging.info('There seems to be nothing to build') + self.mainloop.queue_event(self, _Built()) + + def _find_artifacts_that_are_ready_to_build(self): + def is_ready_to_build(artifact): + return (artifact.state == UNBUILT and + all(a.state == BUILT for a in artifact.dependencies)) + + return [a + for a in map_build_graph(self._artifact, lambda a: a) + if is_ready_to_build(a)] + + def _queue_worker_builds(self, event_source, event): + distbuild.crash_point() + + if self._artifact.state == BUILT: + logging.info('Requested artifact is built') + self.mainloop.queue_event(self, _Built()) + return + + logging.debug('Queuing more worker-builds to run') + if self.debug_graph_state: + logging.debug('Current state of build graph nodes:') + for a in map_build_graph(self._artifact, lambda a: a): + logging.debug(' %s state is %s' % (a.name, a.state)) + if a.state != BUILT: + for dep in a.dependencies: + logging.debug( + ' depends on %s which is %s' % + (dep.name, dep.state)) + + while True: + ready = self._find_artifacts_that_are_ready_to_build() + + if len(ready) == 0: + logging.debug('No new artifacts queued for building') + break + + artifact = ready[0] + + logging.debug( + 'Requesting worker-build of %s (%s)' % + (artifact.name, artifact.cache_key)) + request = distbuild.WorkerBuildRequest(artifact, + self._request['id']) + self.mainloop.queue_event(distbuild.WorkerBuildQueuer, request) + + artifact.state = BUILDING + if artifact.source.morphology['kind'] == 'chunk': + # Chunk artifacts are not built independently + # so when we're building any chunk artifact + # we're also building all the chunk artifacts + # in this source + for a in ready: + if a.source == artifact.source: + a.state = BUILDING + + + def _maybe_notify_initiator_disconnected(self, event_source, event): + if event.id != self._request['id']: + logging.debug('Heard initiator disconnect with event id %d ' + 'but our request id is %d', + event.id, self._request['id']) + return # not for us + + logging.debug("BuildController %r: initiator id %s disconnected", + self, event.id) + + cancel_pending = distbuild.WorkerCancelPending(event.id) + self.mainloop.queue_event(distbuild.WorkerBuildQueuer, cancel_pending) + + cancel = BuildCancel(event.id) + self.mainloop.queue_event(BuildController, cancel) + + self.mainloop.queue_event(self, _Abort) + + def _maybe_relay_build_waiting_for_worker(self, event_source, event): + if event.initiator_id != self._request['id']: + return # not for us + + artifact = self._find_artifact(event.artifact_cache_key) + if artifact is None: + # This is not the event you are looking for. + return + + progress = BuildProgress( + self._request['id'], + 'Ready to build %s: waiting for a worker to become available' + % artifact.name) + self.mainloop.queue_event(BuildController, progress) + + def _maybe_relay_build_step_started(self, event_source, event): + distbuild.crash_point() + if self._request['id'] not in event.initiators: + return # not for us + + logging.debug( + 'BC: _relay_build_step_started: %s' % event.artifact_cache_key) + + artifact = self._find_artifact(event.artifact_cache_key) + if artifact is None: + # This is not the event you are looking for. + return + + logging.debug('BC: got build step started: %s' % artifact.name) + started = BuildStepStarted( + self._request['id'], build_step_name(artifact), event.worker_name) + self.mainloop.queue_event(BuildController, started) + logging.debug('BC: emitted %s' % repr(started)) + + def _maybe_relay_build_step_already_started(self, event_source, event): + if event.initiator_id != self._request['id']: + return # not for us + + artifact = self._find_artifact(event.artifact_cache_key) + + logging.debug('BC: got build step already started: %s' % artifact.name) + started = BuildStepAlreadyStarted( + self._request['id'], build_step_name(artifact), event.worker_name) + self.mainloop.queue_event(BuildController, started) + logging.debug('BC: emitted %s' % repr(started)) + + def _maybe_relay_build_output(self, event_source, event): + distbuild.crash_point() + if self._request['id'] not in event.msg['ids']: + return # not for us + + logging.debug('BC: got output: %s' % repr(event.msg)) + artifact = self._find_artifact(event.artifact_cache_key) + logging.debug('BC: got artifact: %s' % repr(artifact)) + if artifact is None: + # This is not the event you are looking for. + return + + output = BuildOutput( + self._request['id'], build_step_name(artifact), + event.msg['stdout'], event.msg['stderr']) + self.mainloop.queue_event(BuildController, output) + logging.debug('BC: queued %s' % repr(output)) + + def _maybe_relay_build_caching(self, event_source, event): + distbuild.crash_point() + + if self._request['id'] not in event.initiators: + return # not for us + + artifact = self._find_artifact(event.artifact_cache_key) + if artifact is None: + # This is not the event you are looking for. + return + + progress = BuildProgress( + self._request['id'], + 'Transferring %s to shared artifact cache' % artifact.name) + self.mainloop.queue_event(BuildController, progress) + + def _find_artifact(self, cache_key): + artifacts = map_build_graph(self._artifact, lambda a: a) + wanted = [a for a in artifacts if a.cache_key == cache_key] + if wanted: + return wanted[0] + else: + return None + + def _maybe_check_result_and_queue_more_builds(self, event_source, event): + distbuild.crash_point() + if self._request['id'] not in event.msg['ids']: + return # not for us + + artifact = self._find_artifact(event.artifact_cache_key) + if artifact is None: + # This is not the event you are looking for. + return + + logging.debug( + 'Got build result for %s: %s', artifact.name, repr(event.msg)) + + finished = BuildStepFinished( + self._request['id'], build_step_name(artifact)) + self.mainloop.queue_event(BuildController, finished) + + artifact.state = BUILT + + def set_state(a): + if a.source == artifact.source: + a.state = BUILT + + if artifact.source.morphology['kind'] == 'chunk': + # Building a single chunk artifact + # yields all chunk artifacts for the given source + # so we set the state of this source's artifacts + # to BUILT + map_build_graph(self._artifact, set_state) + + self._queue_worker_builds(None, event) + + def _notify_annotation_failed(self, event_source, event): + errmsg = ('Failed to annotate build graph: http request got %d: %s' + % (event.http_status_code, event.error_msg)) + + logging.error(errmsg) + failed = BuildFailed(self._request['id'], errmsg) + self.mainloop.queue_event(BuildController, failed) + + def _maybe_notify_build_failed(self, event_source, event): + distbuild.crash_point() + + if self._request['id'] not in event.msg['ids']: + return # not for us + + artifact = self._find_artifact(event.artifact_cache_key) + + if artifact is None: + logging.error( + 'BuildController %r: artifact %s is not in our build graph!', + self, artifact) + # We abort the build in this case on the grounds that something is + # very wrong internally, and it's best for the initiator to receive + # an error than to be left hanging. + self.mainloop.queue_event(self, _Abort()) + + logging.info( + 'Build step failed for %s: %s', artifact.name, repr(event.msg)) + + step_failed = BuildStepFailed( + self._request['id'], build_step_name(artifact)) + self.mainloop.queue_event(BuildController, step_failed) + + build_failed = BuildFailed( + self._request['id'], + 'Building failed for %s' % artifact.name) + self.mainloop.queue_event(BuildController, build_failed) + + # Cancel any jobs waiting to be executed, since there is no point + # running them if this build has failed, it would just waste + # resources + cancel_pending = distbuild.WorkerCancelPending( + self._request['id']) + self.mainloop.queue_event(distbuild.WorkerBuildQueuer, cancel_pending) + + # Cancel any currently executing jobs for the above reasons, since + # this build will fail and we can't decide whether these jobs will + # be of use to any other build + cancel = BuildCancel(self._request['id']) + self.mainloop.queue_event(BuildController, cancel) + + self.mainloop.queue_event(self, _Abort()) + + def _notify_build_done(self, event_source, event): + distbuild.crash_point() + + logging.debug('Notifying initiator of successful build') + baseurl = urlparse.urljoin( + self._artifact_cache_server, '/1.0/artifacts') + filename = ('%s.%s.%s' % + (self._artifact.cache_key, + self._artifact.source.morphology['kind'], + self._artifact.name)) + url = '%s?filename=%s' % (baseurl, urllib.quote(filename)) + finished = BuildFinished(self._request['id'], [url]) + self.mainloop.queue_event(BuildController, finished) diff --git a/distbuild/connection_machine.py b/distbuild/connection_machine.py new file mode 100644 index 00000000..e75ebe56 --- /dev/null +++ b/distbuild/connection_machine.py @@ -0,0 +1,189 @@ +# distbuild/connection_machine.py -- state machine for connecting to server +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import errno +import logging +import socket + +import distbuild + + +class Reconnect(object): + + pass + + +class StopConnecting(object): + + def __init__(self, exception=None): + self.exception = exception + +class ConnectError(object): + + def __init__(self, exception): + self.exception = exception + + +class ProxyEventSource(object): + + '''Proxy event sources that may come and go.''' + + def __init__(self): + self.event_source = None + + def get_select_params(self): + if self.event_source: + return self.event_source.get_select_params() + else: + return [], [], [], None + + def get_events(self, r, w, x): + if self.event_source: + return self.event_source.get_events(r, w, x) + else: + return [] + + def is_finished(self): + return False + + +class ConnectionMachine(distbuild.StateMachine): + + def __init__(self, addr, port, machine, extra_args, + reconnect_interval=1, max_retries=float('inf')): + super(ConnectionMachine, self).__init__('connecting') + self._addr = addr + self._port = port + self._machine = machine + self._extra_args = extra_args + self._socket = None + self._reconnect_interval = reconnect_interval + self._numof_retries = 0 + self._max_retries = max_retries + + def setup(self): + self._sock_proxy = ProxyEventSource() + self.mainloop.add_event_source(self._sock_proxy) + self._start_connect() + + self._timer = distbuild.TimerEventSource(self._reconnect_interval) + self.mainloop.add_event_source(self._timer) + + spec = [ + # state, source, event_class, new_state, callback + ('connecting', self._sock_proxy, distbuild.SocketWriteable, + 'connected', self._connect), + ('connecting', self, StopConnecting, None, self._stop), + ('connected', self, Reconnect, 'connecting', self._reconnect), + ('connected', self, ConnectError, 'timeout', self._start_timer), + ('connected', self, StopConnecting, None, self._stop), + ('timeout', self._timer, distbuild.Timer, 'connecting', + self._reconnect), + ('timeout', self, StopConnecting, None, self._stop), + ] + self.add_transitions(spec) + + def _start_connect(self): + logging.debug( + 'ConnectionMachine: connecting to %s:%s' % + (self._addr, self._port)) + self._socket = distbuild.create_socket() + distbuild.set_nonblocking(self._socket) + try: + self._socket.connect((self._addr, self._port)) + except socket.error, e: + if e.errno != errno.EINPROGRESS: + raise socket.error( + "%s (attempting connection to distbuild controller " + "at %s:%s)" % (e, self._addr, self._port)) + + src = distbuild.SocketEventSource(self._socket) + self._sock_proxy.event_source = src + + def _connect(self, event_source, event): + try: + self._socket.connect((self._addr, self._port)) + except socket.error, e: + logging.error( + 'Failed to connect to %s:%s: %s' % + (self._addr, self._port, str(e))) + + if self._numof_retries < self._max_retries: + self.mainloop.queue_event(self, ConnectError(e)) + else: + self.mainloop.queue_event(self, StopConnecting(e)) + + return + self._sock_proxy.event_source = None + logging.info('Connected to %s:%s' % (self._addr, self._port)) + m = self._machine(self, self._socket, *self._extra_args) + self.mainloop.add_state_machine(m) + self._socket = None + + def _reconnect(self, event_source, event): + logging.info('Reconnecting to %s:%s' % (self._addr, self._port)) + self._numof_retries += 1 + + if self._socket is not None: + self._socket.close() + self._timer.stop() + self._start_connect() + + def _stop(self, event_source, event): + logging.info( + 'Stopping connection attempts to %s:%s' % (self._addr, self._port)) + self.mainloop.remove_event_source(self._timer) + if self._socket is not None: + self._socket.close() + self._socket = None + + def _start_timer(self, event_source, event): + self._timer.start() + + self._sock_proxy.event_source.close() + self._sock_proxy.event_source = None + +class InitiatorConnectionMachine(ConnectionMachine): + + def __init__(self, app, addr, port, machine, extra_args, + reconnect_interval, max_retries): + + self.cm = super(InitiatorConnectionMachine, self) + self.cm.__init__(addr, port, machine, extra_args, + reconnect_interval, max_retries) + + self.app = app + + def _connect(self, event_source, event): + self.app.status(msg='Connecting to %s:%s' % (self._addr, self._port)) + self.cm._connect(event_source, event) + + def _stop(self, event_source, event): + if event.exception: + self.app.status(msg="Couldn't connect to %s:%s: %s" % + (self._addr, self._port, event.exception.strerror)) + + self.cm._stop(event_source, event) + + def _start_timer(self, event_source, event): + self.app.status(msg="Couldn't connect to %s:%s: %s" % + (self._addr, self._port, event.exception.strerror)) + self.app.status(msg="Retrying in %d seconds" % + self._reconnect_interval) + + self.cm._start_timer(event_source, event) diff --git a/distbuild/crashpoint.py b/distbuild/crashpoint.py new file mode 100644 index 00000000..6e3eb3ef --- /dev/null +++ b/distbuild/crashpoint.py @@ -0,0 +1,126 @@ +# distbuild/crashpoint.py -- user-controlled crashing +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +'''Crash the application. + +For crash testing, it's useful to easily induce crashes, to see how the +rest of the system manages. This module implements user-controllable +crashes. The code will be sprinkled with calls to the ``crash_point`` +function, which crashes the process if call matches a set of user-defined +criteria. + +The criteria consist of: + +* a filename +* a function name +* a maximum call count + +The criterion is fullfilled if ``crash_point`` is called from the named +function defined in the named file more than the given number of times. +Filename matching is using substrings (a filename pattern ``foo.py`` +matches an actual source file path of +``/usr/lib/python2.7/site-packages/distbuild/foo.py``), but function +names must match exactly. It is not possible to match on class names +(since that information is not available from a traceback). + +''' + + +import logging +import os +import sys +import traceback + + +detailed_logging = False + + +def debug(msg): # pragma: no cover + if detailed_logging: + logging.debug(msg) + + +class CrashCondition(object): + + def __init__(self, filename, funcname, max_calls): + self.filename = filename + self.funcname = funcname + self.max_calls = max_calls + self.called = 0 + + def matches(self, filename, funcname): + if self.filename not in filename: + debug( + 'crashpoint: filename mismatch: %s not in %s' % + (repr(self.filename), repr(filename))) + return False + + if self.funcname != funcname: + debug( + 'crashpoint: funcname mismatch: %s != %s' % + (self.funcname, funcname)) + return False + + debug('crashpoint: matches: %s %s' % (filename, funcname)) + return True + + def triggered(self, filename, funcname): + if self.matches(filename, funcname): + self.called += 1 + return self.called >= self.max_calls + else: + return False + + +crash_conditions = [] + + +def add_crash_condition(filename, funcname, max_calls): + crash_conditions.append(CrashCondition(filename, funcname, max_calls)) + + +def add_crash_conditions(strings): + for s in strings: + words = s.split(':') + if len(words) != 3: # pragma: no cover + logging.error('Ignoring malformed crash condition: %s' % repr(s)) + else: + add_crash_condition(words[0], words[1], int(words[2])) + + +def clear_crash_conditions(): + del crash_conditions[:] + + +def crash_point(frame=None): + if frame is None: + frames = traceback.extract_stack(limit=2) + frame = frames[0] + + filename, lineno, funcname, text = frame + + for condition in crash_conditions: + if condition.triggered(filename, funcname): + logging.critical( + 'Crash triggered from %s:%s:%s' % (filename, lineno, funcname)) + sys.exit(255) + else: + debug( + 'Crash not triggered by %s:%s:%s' % + (filename, lineno, funcname)) + diff --git a/distbuild/crashpoint_tests.py b/distbuild/crashpoint_tests.py new file mode 100644 index 00000000..eb64115e --- /dev/null +++ b/distbuild/crashpoint_tests.py @@ -0,0 +1,109 @@ +# distbuild/crashpoint_tests.py -- unit tests for crashpoint.py +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import unittest + +import crashpoint + + +class CrashConditionTests(unittest.TestCase): + + def setUp(self): + self.c = crashpoint.CrashCondition('bar', 'foofunc', 0) + + def test_matches_exact_filename(self): + self.assertTrue(self.c.matches('bar', 'foofunc')) + + def test_matches_basename(self): + self.assertTrue(self.c.matches('dir/bar', 'foofunc')) + + def test_matches_partial_basename(self): + self.assertTrue(self.c.matches('dir/bar.py', 'foofunc')) + + def test_matches_dirname(self): + self.assertTrue(self.c.matches('bar/something.py', 'foofunc')) + + def test_doesnt_match_wrong_function_name(self): + self.assertFalse(self.c.matches('bar', 'foo')) + + def test_triggered_first_time_with_zero_count(self): + c = crashpoint.CrashCondition('bar', 'foofunc', 0) + self.assertTrue(c.triggered('bar', 'foofunc')) + + def test_triggered_first_time_with_zero_count(self): + c = crashpoint.CrashCondition('bar', 'foofunc', 0) + self.assertTrue(c.triggered('bar', 'foofunc')) + + def test_triggered_second_time_with_zero_count(self): + c = crashpoint.CrashCondition('bar', 'foofunc', 0) + self.assertTrue(c.triggered('bar', 'foofunc')) + self.assertTrue(c.triggered('bar', 'foofunc')) + + def test_triggered_first_time_with_count_of_one(self): + c = crashpoint.CrashCondition('bar', 'foofunc', 1) + self.assertTrue(c.triggered('bar', 'foofunc')) + + def test_triggered_second_time_with_count_of_two(self): + c = crashpoint.CrashCondition('bar', 'foofunc', 2) + self.assertFalse(c.triggered('bar', 'foofunc')) + self.assertTrue(c.triggered('bar', 'foofunc')) + + def test_not_triggered_if_not_matched(self): + c = crashpoint.CrashCondition('bar', 'foofunc', 0) + self.assertFalse(c.triggered('bar', 'otherfunc')) + + +class CrashConditionsListTests(unittest.TestCase): + + def setUp(self): + crashpoint.clear_crash_conditions() + + def test_no_conditions_initially(self): + self.assertEqual(crashpoint.crash_conditions, []) + + def test_adds_condition(self): + crashpoint.add_crash_condition('foo.py', 'bar', 0) + self.assertEqual(len(crashpoint.crash_conditions), 1) + c = crashpoint.crash_conditions[0] + self.assertEqual(c.filename, 'foo.py') + self.assertEqual(c.funcname, 'bar') + self.assertEqual(c.max_calls, 0) + + def test_adds_conditions_from_list_of_strings(self): + crashpoint.add_crash_conditions(['foo.py:bar:0']) + self.assertEqual(len(crashpoint.crash_conditions), 1) + c = crashpoint.crash_conditions[0] + self.assertEqual(c.filename, 'foo.py') + self.assertEqual(c.funcname, 'bar') + self.assertEqual(c.max_calls, 0) + + +class CrashPointTests(unittest.TestCase): + + def setUp(self): + crashpoint.clear_crash_conditions() + crashpoint.add_crash_condition('foo.py', 'bar', 0) + + def test_triggers_crash(self): + self.assertRaises( + SystemExit, + crashpoint.crash_point, frame=('foo.py', 123, 'bar', 'text')) + + def test_does_not_trigger_crash(self): + self.assertEqual(crashpoint.crash_point(), None) + diff --git a/distbuild/distbuild_socket.py b/distbuild/distbuild_socket.py new file mode 100644 index 00000000..ce69f29e --- /dev/null +++ b/distbuild/distbuild_socket.py @@ -0,0 +1,63 @@ +# distbuild/distbuild_socket.py -- wrapper around Python 'socket' module. +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import socket + + +class DistbuildSocket(object): + '''Wraps socket.SocketType with a few helper functions.''' + + def __init__(self, real_socket): + self.real_socket = real_socket + + def __getattr__(self, name): + return getattr(self.real_socket, name) + + def __repr__(self): + return '<DistbuildSocket at 0x%x: %s>' % (id(self), str(self)) + + def __str__(self): + localname = self.localname() or '(closed)' + remotename = self.remotename() + if remotename is None: + return '%s' % self.localname() + else: + return '%s -> %s' % (self.localname(), remotename) + + def accept(self, *args): + result = self.real_socket.accept(*args) + return DistbuildSocket(result[0]), result[1:] + + def localname(self): + '''Get local end of socket connection as a string.''' + try: + return '%s:%s' % self.getsockname() + except socket.error: + # If the socket is in destruction we may get EBADF here. + return None + + def remotename(self): + '''Get remote end of socket connection as a string.''' + try: + return '%s:%s' % self.getpeername() + except socket.error: + return None + + +def create_socket(*args): + return DistbuildSocket(socket.socket(*args)) diff --git a/distbuild/eventsrc.py b/distbuild/eventsrc.py new file mode 100644 index 00000000..560b9b7a --- /dev/null +++ b/distbuild/eventsrc.py @@ -0,0 +1,60 @@ +# mainloop/eventsrc.py -- interface for event sources +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +class EventSource(object): + + '''A source of events for state machines. + + This is a base class. + + An event source watches one file descriptor, and returns events + related to it. The events may vary depending on the file descriptor. + The actual watching is done using select.select. + + ''' + + def get_select_params(self): + '''Return parameters to use for select for this event source. + + Three lists of file descriptors, and a timeout are returned. + The three lists and the timeout are used as arguments to the + select.select function, though they may be manipulated and + combined with return values from other event sources. + + ''' + + return [], [], [], None + + def get_events(self, r, w, x): + '''Return events related to this file descriptor. + + The arguments are the return values of select.select. + + ''' + + return [] + + def is_finished(self): + '''Is this event source finished? + + It's finished if it won't ever return any new events. + + ''' + + return False + diff --git a/distbuild/helper_router.py b/distbuild/helper_router.py new file mode 100644 index 00000000..f7126093 --- /dev/null +++ b/distbuild/helper_router.py @@ -0,0 +1,198 @@ +# distbuild/helper_router.py -- state machine for controller's helper comms +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import logging + +import distbuild + + +class HelperRequest(object): + + def __init__(self, msg): + self.msg = msg + + +class HelperOutput(object): + + def __init__(self, msg): + self.msg = msg + + +class HelperResult(object): + + def __init__(self, msg): + self.msg = msg + + +class HelperRouter(distbuild.StateMachine): + + '''Route JSON messages between helpers and other state machines. + + This state machine relays and schedules access to one distbuild-helper + process. The helper process connects to a socket, which causes an + instance of HelperRouter to be created (one per connection). The + various instances co-ordinate requests automatically amongst + themselves. + + Other state machines in the same mainloop as HelperRouter can + request work from the helper process by emitting an event: + + * event source: the distbuild.HelperProcess class + * event: distbuild.HelperRequest instance + + The HelperRequest event gets a message to be serialised as JSON. + The message must be a Pythondict that the distbuild-helper understands. + + HelperRouter will send the msg to the next available helper process. + When the helper sends back the result, HelperRouter will emit a + HelperResult event, using the same ``request_id`` as the request had. + + For its internal use, HelperRouter sets the ``id`` item in the + request object. + + ''' + + _pending_requests = [] + _running_requests = {} + _pending_helpers = [] + _request_counter = distbuild.IdentifierGenerator('HelperRouter') + _route_map = distbuild.RouteMap() + + def __init__(self, conn): + distbuild.StateMachine.__init__(self, 'idle') + self.conn = conn + + def setup(self): + jm = distbuild.JsonMachine(self.conn) + self.mainloop.add_state_machine(jm) + + spec = [ + # state, source, event_class, new_state, callback + ('idle', HelperRouter, HelperRequest, 'idle', + self._handle_request), + ('idle', jm, distbuild.JsonNewMessage, 'idle', self._helper_msg), + ('idle', jm, distbuild.JsonEof, None, self._close), + ] + self.add_transitions(spec) + + def _handle_request(self, event_source, event): + '''Send request received via mainloop, or put in queue.''' + logging.debug('HelperRouter: received request: %s', repr(event.msg)) + self._enqueue_request(event.msg) + if self._pending_helpers: + self._send_request() + + def _helper_msg(self, event_source, event): + '''Handle message from helper.''' + +# logging.debug('HelperRouter: from helper: %s', repr(event.msg)) + + handlers = { + 'helper-ready': self._handle_helper_ready, + 'exec-output': self._handle_exec_output, + 'exec-response': self._handle_exec_response, + 'http-response': self._handle_http_response, + } + + handler = handlers[event.msg['type']] + handler(event_source, event.msg) + + def _handle_helper_ready(self, event_source, msg): + self._pending_helpers.append(event_source) + if self._pending_requests: + self._send_request() + + def _get_request(self, msg): + request_id = msg['id'] + if request_id in self._running_requests: + request, helper = self._running_requests[request_id] + return request + elif request_id is None: + logging.error( + 'Helper process sent message without "id" field: %s', + repr(event.msg)) + else: + logging.error( + 'Helper process sent message with unknown id: %s', + repr(event.msg)) + + def _new_message(self, msg): + old_id = msg['id'] + new_msg = dict(msg) + new_msg['id'] = self._route_map.get_incoming_id(old_id) + return new_msg + + def _handle_exec_output(self, event_source, msg): + request = self._get_request(msg) + if request is not None: + new_msg = self._new_message(msg) + self.mainloop.queue_event(HelperRouter, HelperOutput(new_msg)) + + def _handle_exec_response(self, event_source, msg): + request = self._get_request(msg) + if request is not None: + new_msg = self._new_message(msg) + self._route_map.remove(msg['id']) + del self._running_requests[msg['id']] + self.mainloop.queue_event(HelperRouter, HelperResult(new_msg)) + + def _handle_http_response(self, event_source, msg): + request = self._get_request(msg) + if request is not None: + new_msg = self._new_message(msg) + self._route_map.remove(msg['id']) + del self._running_requests[msg['id']] + self.mainloop.queue_event(HelperRouter, HelperResult(new_msg)) + + def _close(self, event_source, event): + logging.debug('HelperRouter: closing: %s', repr(event_source)) + event_source.close() + + # Remove from pending helpers. + if event_source in self._pending_helpers: + self._pending_helpers.remove(event_source) + + # Re-queue any requests running on the hlper that just quit. + for request_id in self._running_requests.keys(): + request, helper = self._running_requests[request_id] + if event_source == helper: + del self._running_requests[request_id] + self._enqueue_request(request) + + # Finally, if there are any pending requests and helpers, + # send requests. + while self._pending_requests and self._pending_helpers: + self._send_request() + + def _enqueue_request(self, request): + '''Put request into queue.''' +# logging.debug('HelperRouter: enqueuing request: %s' % repr(request)) + old_id = request['id'] + new_id = self._request_counter.next() + request['id'] = new_id + self._route_map.add(old_id, new_id) + self._pending_requests.append(request) + + def _send_request(self): + '''Pick the first queued request and send it to an available helper.''' + request = self._pending_requests.pop(0) + helper = self._pending_helpers.pop() + self._running_requests[request['id']] = (request, helper) + helper.send(request) +# logging.debug('HelperRouter: sent to helper: %s', repr(request)) + diff --git a/distbuild/idgen.py b/distbuild/idgen.py new file mode 100644 index 00000000..41f2ffcf --- /dev/null +++ b/distbuild/idgen.py @@ -0,0 +1,33 @@ +# distbuild/idgen.py -- generate unique identifiers +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import logging + + +class IdentifierGenerator(object): + + '''Generate unique identifiers.''' + + def __init__(self, series): + self._series = series + self._counter = 0 + + def next(self): + self._counter += 1 + return '%s-%d' % (self._series, self._counter) + diff --git a/distbuild/initiator.py b/distbuild/initiator.py new file mode 100644 index 00000000..b60700fd --- /dev/null +++ b/distbuild/initiator.py @@ -0,0 +1,201 @@ +# distbuild/initiator.py -- state machine for the initiator +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import cliapp +import logging +import random +import sys + +import distbuild + + +class _Finished(object): + + def __init__(self, msg): + self.msg = msg + + +class _Failed(object): + + def __init__(self, msg): + self.msg = msg + + +class Initiator(distbuild.StateMachine): + + def __init__(self, cm, conn, app, repo_name, ref, morphology): + distbuild.StateMachine.__init__(self, 'waiting') + self._cm = cm + self._conn = conn + self._app = app + self._repo_name = repo_name + self._ref = ref + self._morphology = morphology + self._steps = None + self._step_outputs = {} + self.debug_transitions = False + + def setup(self): + distbuild.crash_point() + + self._jm = distbuild.JsonMachine(self._conn) + self.mainloop.add_state_machine(self._jm) + logging.debug('initiator: _jm=%s' % repr(self._jm)) + + spec = [ + # state, source, event_class, new_state, callback + ('waiting', self._jm, distbuild.JsonEof, None, self._terminate), + ('waiting', self._jm, distbuild.JsonNewMessage, 'waiting', + self._handle_json_message), + ('waiting', self, _Finished, None, self._succeed), + ('waiting', self, _Failed, None, self._fail), + ] + self.add_transitions(spec) + + random_id = random.randint(0, 2**32-1) + + self._app.status( + msg='Requesting build of %(repo)s %(ref)s %(morph)s', + repo=self._repo_name, + ref=self._ref, + morph=self._morphology) + msg = distbuild.message('build-request', + id=random_id, + repo=self._repo_name, + ref=self._ref, + morphology=self._morphology + ) + self._jm.send(msg) + logging.debug('Initiator: sent to controller: %s', repr(msg)) + + def _handle_json_message(self, event_source, event): + distbuild.crash_point() + + logging.debug('Initiator: from controller: %s' % repr(event.msg)) + + handlers = { + 'build-finished': self._handle_build_finished_message, + 'build-failed': self._handle_build_failed_message, + 'build-progress': self._handle_build_progress_message, + 'build-steps': self._handle_build_steps_message, + 'step-started': self._handle_step_started_message, + 'step-already-started': self._handle_step_already_started_message, + 'step-output': self._handle_step_output_message, + 'step-finished': self._handle_step_finished_message, + 'step-failed': self._handle_step_failed_message, + } + + handler = handlers[event.msg['type']] + handler(event.msg) + + def _handle_build_finished_message(self, msg): + self.mainloop.queue_event(self, _Finished(msg)) + + def _handle_build_failed_message(self, msg): + self.mainloop.queue_event(self, _Failed(msg)) + + def _handle_build_progress_message(self, msg): + self._app.status(msg='Progress: %(msgtext)s', msgtext=msg['message']) + + def _handle_build_steps_message(self, msg): + self._steps = msg['steps'] + self._app.status( + msg='Build steps in total: %(steps)d', + steps=len(self._steps)) + + def _open_output(self, msg): + assert msg['step_name'] not in self._step_outputs + filename = 'build-step-%s.log' % msg['step_name'] + f = open(filename, 'a') + self._step_outputs[msg['step_name']] = f + + def _close_output(self, msg): + self._step_outputs[msg['step_name']].close() + del self._step_outputs[msg['step_name']] + + def _handle_step_already_started_message(self, msg): + self._app.status( + msg='%s is already building on %s' % (msg['step_name'], + msg['worker_name'])) + self._open_output(msg) + + def _handle_step_started_message(self, msg): + self._app.status( + msg='Started building %(step_name)s on %(worker_name)s', + step_name=msg['step_name'], + worker_name=msg['worker_name']) + self._open_output(msg) + + def _handle_step_output_message(self, msg): + step_name = msg['step_name'] + if step_name in self._step_outputs: + f = self._step_outputs[step_name] + f.write(msg['stdout']) + f.write(msg['stderr']) + f.flush() + else: + logging.warning( + 'Got step-output message for unknown step: %s' % repr(msg)) + + def _handle_step_finished_message(self, msg): + step_name = msg['step_name'] + if step_name in self._step_outputs: + self._app.status( + msg='Finished building %(step_name)s', + step_name=step_name) + self._close_output(msg) + else: + logging.warning( + 'Got step-finished message for unknown step: %s' % repr(msg)) + + def _handle_step_failed_message(self, msg): + step_name = msg['step_name'] + if step_name in self._step_outputs: + self._app.status( + msg='Build failed: %(step_name)s', + step_name=step_name) + self._close_output(msg) + else: + logging.warning( + 'Got step-failed message for unknown step: %s' % repr(msg)) + + def _succeed(self, event_source, event): + self.mainloop.queue_event(self._cm, distbuild.StopConnecting()) + self._jm.close() + logging.info('Build finished OK') + + urls = event.msg['urls'] + if urls: + for url in urls: + self._app.status(msg='Artifact: %(url)s', url=url) + else: + self._app.status( + msg='Controller did not give us any artifact URLs.') + + def _fail(self, event_source, event): + self.mainloop.queue_event(self._cm, distbuild.StopConnecting()) + self._jm.close() + raise cliapp.AppException( + 'Failed to build %s %s %s: %s' % + (self._repo_name, self._ref, self._morphology, + event.msg['reason'])) + + def _terminate(self, event_source, event): + self.mainloop.queue_event(self._cm, distbuild.StopConnecting()) + self._jm.close() + diff --git a/distbuild/initiator_connection.py b/distbuild/initiator_connection.py new file mode 100644 index 00000000..0f009fcc --- /dev/null +++ b/distbuild/initiator_connection.py @@ -0,0 +1,242 @@ +# distbuild/initiator_connection.py -- communicate with initiator +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import logging + +import distbuild + + +class InitiatorDisconnect(object): + + def __init__(self, id): + self.id = id + + +class _Close(object): + + def __init__(self, event_source): + self.event_source = event_source + + +class InitiatorConnection(distbuild.StateMachine): + + '''Communicate with a single initiator. + + When a developer runs 'morph distbuild' and connects to the controller, + the ListenServer object on the controller creates an InitiatorConnection. + + This state machine communicates with the build initiator, relaying and + translating messages from the initiator to the rest of the controller's + state machines, and vice versa. + + ''' + + _idgen = distbuild.IdentifierGenerator('InitiatorConnection') + _route_map = distbuild.RouteMap() + + def __init__(self, conn, artifact_cache_server, morph_instance): + distbuild.StateMachine.__init__(self, 'idle') + self.conn = conn + self.artifact_cache_server = artifact_cache_server + self.morph_instance = morph_instance + self.initiator_name = conn.remotename() + + def __repr__(self): + return '<InitiatorConnection at 0x%x: remote %s>' % (id(self), + self.initiator_name) + + def setup(self): + self.jm = distbuild.JsonMachine(self.conn) + self.mainloop.add_state_machine(self.jm) + + self.our_ids = set() + + spec = [ + # state, source, event_class, new_state, callback + ('idle', self.jm, distbuild.JsonNewMessage, 'idle', + self._handle_msg), + ('idle', self.jm, distbuild.JsonEof, 'closing', self._disconnect), + ('idle', distbuild.BuildController, distbuild.BuildFinished, + 'idle', self._send_build_finished_message), + ('idle', distbuild.BuildController, distbuild.BuildFailed, + 'idle', self._send_build_failed_message), + ('idle', distbuild.BuildController, distbuild.BuildProgress, + 'idle', self._send_build_progress_message), + ('idle', distbuild.BuildController, distbuild.BuildSteps, + 'idle', self._send_build_steps_message), + ('idle', distbuild.BuildController, distbuild.BuildStepStarted, + 'idle', self._send_build_step_started_message), + ('idle', distbuild.BuildController, + distbuild.BuildStepAlreadyStarted, 'idle', + self._send_build_step_already_started_message), + ('idle', distbuild.BuildController, distbuild.BuildOutput, + 'idle', self._send_build_output_message), + ('idle', distbuild.BuildController, distbuild.BuildStepFinished, + 'idle', self._send_build_step_finished_message), + ('idle', distbuild.BuildController, distbuild.BuildStepFailed, + 'idle', self._send_build_step_failed_message), + ('closing', self, _Close, None, self._close), + ] + self.add_transitions(spec) + + def _handle_msg(self, event_source, event): + '''Handle message from initiator.''' + + logging.debug('InitiatorConnection: from %s: %r', self.initiator_name, + event.msg) + + if event.msg['type'] == 'build-request': + new_id = self._idgen.next() + self.our_ids.add(new_id) + self._route_map.add(event.msg['id'], new_id) + event.msg['id'] = new_id + build_controller = distbuild.BuildController( + self, event.msg, self.artifact_cache_server, + self.morph_instance) + self.mainloop.add_state_machine(build_controller) + + def _disconnect(self, event_source, event): + for id in self.our_ids: + logging.debug('InitiatorConnection: %s: InitiatorDisconnect(%s)', + self.initiator_name, str(id)) + self.mainloop.queue_event(InitiatorConnection, + InitiatorDisconnect(id)) + self.mainloop.queue_event(self, _Close(event_source)) + + def _close(self, event_source, event): + logging.debug('InitiatorConnection: %s: closing: %s', + self.initiator_name, repr(event.event_source)) + + event.event_source.close() + + def _handle_result(self, event_source, event): + '''Handle result from helper.''' + + if event.msg['id'] in self.our_ids: + logging.debug( + 'InitiatorConnection: received result: %s', repr(event.msg)) + self.jm.send(event.msg) + + def _log_send(self, msg): + logging.debug( + 'InitiatorConnection: sent to %s: %r', self.initiator_name, msg) + + def _send_build_finished_message(self, event_source, event): + if event.id in self.our_ids: + msg = distbuild.message('build-finished', + id=self._route_map.get_incoming_id(event.id), + urls=event.urls) + self._route_map.remove(event.id) + self.our_ids.remove(event.id) + self.jm.send(msg) + self._log_send(msg) + + def _send_build_failed_message(self, event_source, event): + if event.id in self.our_ids: + msg = distbuild.message('build-failed', + id=self._route_map.get_incoming_id(event.id), + reason=event.reason) + self._route_map.remove(event.id) + self.our_ids.remove(event.id) + self.jm.send(msg) + self._log_send(msg) + + def _send_build_progress_message(self, event_source, event): + if event.id in self.our_ids: + msg = distbuild.message('build-progress', + id=self._route_map.get_incoming_id(event.id), + message=event.message_text) + self.jm.send(msg) + self._log_send(msg) + + def _send_build_steps_message(self, event_source, event): + + def make_step_dict(artifact): + return { + 'name': distbuild.build_step_name(artifact), + 'build-depends': [ + distbuild.build_step_name(x) + for x in artifact.dependencies + ] + } + + if event.id in self.our_ids: + step_names = distbuild.map_build_graph( + event.artifact, make_step_dict) + msg = distbuild.message('build-steps', + id=self._route_map.get_incoming_id(event.id), + steps=step_names) + self.jm.send(msg) + self._log_send(msg) + + def _send_build_step_started_message(self, event_source, event): + logging.debug('InitiatorConnection: build_step_started: ' + 'id=%s step_name=%s worker_name=%s' % + (event.id, event.step_name, event.worker_name)) + if event.id in self.our_ids: + msg = distbuild.message('step-started', + id=self._route_map.get_incoming_id(event.id), + step_name=event.step_name, + worker_name=event.worker_name) + self.jm.send(msg) + self._log_send(msg) + + def _send_build_step_already_started_message(self, event_source, event): + logging.debug('InitiatorConnection: build_step_already_started: ' + 'id=%s step_name=%s worker_name=%s' % (event.id, event.step_name, + event.worker_name)) + + if event.id in self.our_ids: + msg = distbuild.message('step-already-started', + id=self._route_map.get_incoming_id(event.id), + step_name=event.step_name, + worker_name=event.worker_name) + self.jm.send(msg) + self._log_send(msg) + + def _send_build_output_message(self, event_source, event): + logging.debug('InitiatorConnection: build_output: ' + 'id=%s stdout=%s stderr=%s' % + (repr(event.id), repr(event.stdout), repr(event.stderr))) + if event.id in self.our_ids: + msg = distbuild.message('step-output', + id=self._route_map.get_incoming_id(event.id), + step_name=event.step_name, + stdout=event.stdout, + stderr=event.stderr) + self.jm.send(msg) + self._log_send(msg) + + def _send_build_step_finished_message(self, event_source, event): + logging.debug('heard built step finished: event.id: %s our_ids: %s' + % (str(event.id), str(self.our_ids))) + if event.id in self.our_ids: + msg = distbuild.message('step-finished', + id=self._route_map.get_incoming_id(event.id), + step_name=event.step_name) + self.jm.send(msg) + self._log_send(msg) + + def _send_build_step_failed_message(self, event_source, event): + if event.id in self.our_ids: + msg = distbuild.message('step-failed', + id=self._route_map.get_incoming_id(event.id), + step_name=event.step_name) + self.jm.send(msg) + self._log_send(msg) + diff --git a/distbuild/jm.py b/distbuild/jm.py new file mode 100644 index 00000000..513c69fa --- /dev/null +++ b/distbuild/jm.py @@ -0,0 +1,115 @@ +# mainloop/jm.py -- state machine for JSON communication between nodes +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import fcntl +import json +import logging +import os +import socket +import sys + +from sm import StateMachine +from stringbuffer import StringBuffer +from sockbuf import (SocketBuffer, SocketBufferNewData, + SocketBufferEof, SocketError) + + +class JsonNewMessage(object): + + def __init__(self, msg): + self.msg = msg + + +class JsonEof(object): + + pass + + +class _Close2(object): + + pass + + +class JsonMachine(StateMachine): + + '''A state machine for sending/receiving JSON messages across TCP.''' + + max_buffer = 16 * 1024 + + def __init__(self, conn): + StateMachine.__init__(self, 'rw') + self.conn = conn + self.debug_json = False + + def __repr__(self): + return '<JsonMachine at 0x%x: socket %s, max_buffer %s>' % \ + (id(self), self.conn, self.max_buffer) + + def setup(self): + sockbuf = self.sockbuf = SocketBuffer(self.conn, self.max_buffer) + self.mainloop.add_state_machine(sockbuf) + + self._eof = False + self.receive_buf = StringBuffer() + + spec = [ + # state, source, event_class, new_state, callback + ('rw', sockbuf, SocketBufferNewData, 'rw', self._parse), + ('rw', sockbuf, SocketBufferEof, 'w', self._send_eof), + ('rw', self, _Close2, None, self._really_close), + + ('w', self, _Close2, None, self._really_close), + ] + self.add_transitions(spec) + + def send(self, msg): + '''Send a message to the other side.''' + self.sockbuf.write('%s\n' % json.dumps(msg)) + + def close(self): + '''Tell state machine it should shut down. + + The state machine will vanish once it has flushed any pending + writes. + + ''' + + self.mainloop.queue_event(self, _Close2()) + + def _parse(self, event_source, event): + data = event.data + self.receive_buf.add(data) + if self.debug_json: + logging.debug('JsonMachine: Received: %s' % repr(data)) + while True: + line = self.receive_buf.readline() + if line is None: + break + line = line.rstrip() + if self.debug_json: + logging.debug('JsonMachine: line: %s' % repr(line)) + msg = json.loads(line) + self.mainloop.queue_event(self, JsonNewMessage(msg)) + + def _send_eof(self, event_source, event): + self.mainloop.queue_event(self, JsonEof()) + + def _really_close(self, event_source, event): + self.sockbuf.close() + self._send_eof(event_source, event) + diff --git a/distbuild/json_router.py b/distbuild/json_router.py new file mode 100644 index 00000000..8b7b6457 --- /dev/null +++ b/distbuild/json_router.py @@ -0,0 +1,165 @@ +# distbuild/json_router.py -- state machine to route JSON messages +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import logging + +import distbuild + + +class JsonRouter(distbuild.StateMachine): + + '''Route JSON messages between clients and helpers. + + This state machine receives JSON messages from clients and helpers, + and routes messages between them. + + Each incoming request is labeled with a unique identifier, then + sent to the next free helper. The helper's response will retain + the unique id, so that the response can be routed to the right + client. + + ''' + + pending_requests = [] + running_requests = {} + pending_helpers = [] + request_counter = distbuild.IdentifierGenerator('JsonRouter') + route_map = distbuild.RouteMap() + + def __init__(self, conn): + distbuild.StateMachine.__init__(self, 'idle') + self.conn = conn + logging.debug('JsonMachine: connection from %s', conn.getpeername()) + + def setup(self): + jm = distbuild.JsonMachine(self.conn) + jm.debug_json = True + self.mainloop.add_state_machine(jm) + + spec = [ + # state, source, event_class, new_state, callback + ('idle', jm, distbuild.JsonNewMessage, 'idle', self.bloop), + ('idle', jm, distbuild.JsonEof, None, self.close), + ] + self.add_transitions(spec) + + def _lookup_request(self, request_id): + if request_id in self.running_requests: + return self.running_requests[request_id] + else: + return None + + def bloop(self, event_source, event): + logging.debug('JsonRouter: got msg: %s', repr(event.msg)) + handlers = { + 'http-request': self.do_request, + 'http-response': self.do_response, + 'exec-request': self.do_request, + 'exec-cancel': self.do_cancel, + 'exec-output': self.do_exec_output, + 'exec-response': self.do_response, + 'helper-ready': self.do_helper_ready, + } + handler = handlers.get(event.msg['type']) + handler(event_source, event) + + def do_request(self, client, event): + self._enqueue_request(client, event.msg) + if self.pending_helpers: + self._send_request() + + def do_cancel(self, client, event): + for id in self.route_map.get_outgoing_ids(event.msg['id']): + logging.debug('JsonRouter: looking up request for id %s', id) + t = self._lookup_request(id) + if t: + helper = t[2] + new = dict(event.msg) + new['id'] = id + helper.send(new) + logging.debug('JsonRouter: sent to helper: %s', repr(new)) + + def do_response(self, helper, event): + t = self._lookup_request(event.msg['id']) + if t: + client, msg, helper = t + new = dict(event.msg) + new['id'] = self.route_map.get_incoming_id(msg['id']) + client.send(new) + logging.debug('JsonRouter: sent to client: %s', repr(new)) + + def do_helper_ready(self, helper, event): + self.pending_helpers.append(helper) + if self.pending_requests: + self._send_request() + + def do_exec_output(self, helper, event): + t = self._lookup_request(event.msg['id']) + if t: + client, msg, helper = t + new = dict(event.msg) + new['id'] = self.route_map.get_incoming_id(msg['id']) + client.send(new) + logging.debug('JsonRouter: sent to client: %s', repr(new)) + + def close(self, event_source, event): + logging.debug('closing: %s', repr(event_source)) + event_source.close() + + # Remove from pending helpers. + if event_source in self.pending_helpers: + self.pending_helpers.remove(event_source) + + # Remove from running requests, and put the request back in the + # pending requests queue if the helper quit (but not if the + # client quit). + for request_id in self.running_requests.keys(): + client, msg, helper = self.running_requests[request_id] + if event_source == client: + del self.running_requests[request_id] + elif event_source == helper: + del self.running_requests[request_id] + self._enqueue_request(client, msg) + + # Remove from pending requests, if the client quit. + i = 0 + while i < len(self.pending_requests): + client, msg = self.pending_requests[i] + if event_source == client: + del self.pending_requests[i] + else: + i += 1 + + # Finally, if there are any pending requests and helpers, + # send requests. + while self.pending_requests and self.pending_helpers: + self._send_request() + + def _enqueue_request(self, client, msg): + new = dict(msg) + new['id'] = self.request_counter.next() + self.route_map.add(msg['id'], new['id']) + self.pending_requests.append((client, new)) + + def _send_request(self): + client, msg = self.pending_requests.pop(0) + helper = self.pending_helpers.pop() + self.running_requests[msg['id']] = (client, msg, helper) + helper.send(msg) + logging.debug('JsonRouter: sent to helper: %s', repr(msg)) + diff --git a/distbuild/mainloop.py b/distbuild/mainloop.py new file mode 100644 index 00000000..f0e5eebc --- /dev/null +++ b/distbuild/mainloop.py @@ -0,0 +1,129 @@ +# mainloop/mainloop.py -- select-based main loop +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import fcntl +import logging +import os +import select + + +class MainLoop(object): + + '''A select-based main loop. + + The main loop watches a set of file descriptors wrapped in + EventSource objects, and when something happens with them, + asks the EventSource objects to create events, which it then + feeds into user-supplied state machines. The state machines + can create further events, which are processed further. + + When nothing is happening, the main loop sleeps in the + select.select call. + + ''' + + def __init__(self): + self._machines = [] + self._sources = [] + self._events = [] + self.dump_filename = None + + def add_state_machine(self, machine): + logging.debug('MainLoop.add_state_machine: %s' % machine) + machine.mainloop = self + machine.setup() + self._machines.append(machine) + if self.dump_filename: + filename = '%s%s.dot' % (self.dump_filename, + machine.__class__.__name__) + machine.dump_dot(filename) + + def remove_state_machine(self, machine): + logging.debug('MainLoop.remove_state_machine: %s' % machine) + self._machines.remove(machine) + + def add_event_source(self, event_source): + logging.debug('MainLoop.add_event_source: %s' % event_source) + self._sources.append(event_source) + + def remove_event_source(self, event_source): + logging.debug('MainLoop.remove_event_source: %s' % event_source) + self._sources.remove(event_source) + + def _setup_select(self): + r = [] + w = [] + x = [] + timeout = None + + self._sources = [s for s in self._sources if not s.is_finished()] + + for event_source in self._sources: + sr, sw, sx, st = event_source.get_select_params() + r.extend(sr) + w.extend(sw) + x.extend(sx) + if timeout is None: + timeout = st + elif st is not None: + timeout = min(timeout, st) + + return r, w, x, timeout + + def _run_once(self): + r, w, x, timeout = self._setup_select() + assert r or w or x or timeout is not None + r, w, x = select.select(r, w, x, timeout) + + for event_source in self._sources: + if event_source.is_finished(): + self.remove_event_source(event_source) + else: + for event in event_source.get_events(r, w, x): + self.queue_event(event_source, event) + + for event_source, event in self._dequeue_events(): + for machine in self._machines[:]: + for new_event in machine.handle_event(event_source, event): + self.queue_event(event_source, new_event) + if machine.state is None: + self.remove_state_machine(machine) + + def run(self): + '''Run the main loop. + + The main loop terminates when there are no state machines to + run anymore. + + ''' + + logging.debug('MainLoop starts') + while self._machines: + self._run_once() + logging.debug('MainLoop ends') + + def queue_event(self, event_source, event): + '''Add an event to queue of events to be processed.''' + + self._events.append((event_source, event)) + + def _dequeue_events(self): + while self._events: + event_source, event = self._events.pop(0) + + yield event_source, event diff --git a/distbuild/protocol.py b/distbuild/protocol.py new file mode 100644 index 00000000..d5dfe2b7 --- /dev/null +++ b/distbuild/protocol.py @@ -0,0 +1,100 @@ +# distbuild/protocol.py -- abstractions for the JSON messages +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +'''Construct protocol message objects (dicts).''' + + +_types = { + 'build-request': [ + 'id', + 'repo', + 'ref', + 'morphology', + ], + 'build-progress': [ + 'id', + 'message', + ], + 'build-steps': [ + 'id', + 'steps', + ], + 'step-started': [ + 'id', + 'step_name', + 'worker_name', + ], + 'step-already-started': [ + 'id', + 'step_name', + 'worker_name', + ], + 'step-output': [ + 'id', + 'step_name', + 'stdout', + 'stderr', + ], + 'step-finished': [ + 'id', + 'step_name', + ], + 'step-failed': [ + 'id', + 'step_name', + ], + 'build-finished': [ + 'id', + 'urls', + ], + 'build-failed': [ + 'id', + 'reason', + ], + 'exec-request': [ + 'id', + 'argv', + 'stdin_contents', + ], + 'exec-cancel': [ + 'id', + ], + 'http-request': [ + 'id', + 'url', + 'method', + 'headers', + 'body', + ], +} + + +def message(message_type, **kwargs): + assert message_type in _types + required_fields = _types[message_type] + + for name in required_fields: + assert name in kwargs, 'field %s is required' % name + + for name in kwargs: + assert name in required_fields, 'field %s is not allowed' % name + + msg = dict(kwargs) + msg['type'] = message_type + return msg + diff --git a/distbuild/proxy_event_source.py b/distbuild/proxy_event_source.py new file mode 100644 index 00000000..20080800 --- /dev/null +++ b/distbuild/proxy_event_source.py @@ -0,0 +1,47 @@ +# distbuild/proxy_event_source.py -- proxy for temporary event sources +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import errno +import logging +import socket + +import distbuild + + +class ProxyEventSource(object): + + '''Proxy event sources that may come and go.''' + + def __init__(self): + self.event_source = None + + def get_select_params(self): + if self.event_source: + return self.event_source.get_select_params() + else: + return [], [], [], None + + def get_events(self, r, w, x): + if self.event_source: + return self.event_source.get_events(r, w, x) + else: + return [] + + def is_finished(self): + return False + diff --git a/distbuild/route_map.py b/distbuild/route_map.py new file mode 100644 index 00000000..6dd90d78 --- /dev/null +++ b/distbuild/route_map.py @@ -0,0 +1,60 @@ +# distbuild/route_map.py -- map message ids for routing purposes +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +class RouteMap(object): + + '''Map message identifiers for routing purposes. + + Various state machines need to handle requests coming from multiple + sources, and they need to keep track of which responses should be + sent to which requestors. This class provides tools for keeping + track of that. + + Each message is expected to have a unique identifier of some sort. + The incoming request message has one, and all responses to it need + to keep that. An incoming request might be converted into one or more + outgoing requests, each with its own unique id. The responses to all + of those need to be mapped back to the original incoming request. + + For this class, we care about "incoming id" and "outgoing id". + There can be multiple outgoing identifiers for one incoming one. + + ''' + + def __init__(self): + self._routes = {} + + def add(self, incoming_id, outgoing_id): + assert (outgoing_id not in self._routes or + self._routes[outgoing_id] == incoming_id) + self._routes[outgoing_id] = incoming_id + + def get_incoming_id(self, outgoing_id): + '''Get the incoming id corresponding to an outgoing one. + + Raise KeyError if not found. + + ''' + + return self._routes[outgoing_id] + + def get_outgoing_ids(self, incoming_id): + return [o for (o, i) in self._routes.iteritems() if i == incoming_id] + + def remove(self, outgoing_id): + del self._routes[outgoing_id] diff --git a/distbuild/route_map_tests.py b/distbuild/route_map_tests.py new file mode 100644 index 00000000..b5ceca70 --- /dev/null +++ b/distbuild/route_map_tests.py @@ -0,0 +1,56 @@ +# distbuild/route_map_tests.py -- unit tests for message routing +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import unittest + +import distbuild + + +class RouteMapTests(unittest.TestCase): + + def setUp(self): + self.rm = distbuild.RouteMap() + + def test_raises_error_for_unknown_route(self): + self.assertRaises(KeyError, self.rm.get_incoming_id, 'outgoing') + + def test_finds_added_route(self): + self.rm.add('incoming', 'outgoing') + self.assertEqual(self.rm.get_incoming_id('outgoing'), 'incoming') + + def test_finds_outgoing_ids(self): + self.rm.add('incoming', 'outgoing') + self.assertEqual(self.rm.get_outgoing_ids('incoming'), ['outgoing']) + + def test_removes_added_route(self): + self.rm.add('incoming', 'outgoing') + self.rm.remove('outgoing') + self.assertRaises(KeyError, self.rm.get_incoming_id, 'outgoing') + + def test_raises_error_if_forgetting_unknown_route(self): + self.assertRaises(KeyError, self.rm.remove, 'outgoing') + + def test_silently_ignores_adding_existing_route(self): + self.rm.add('incoming', 'outgoing') + self.rm.add('incoming', 'outgoing') + self.assertEqual(self.rm.get_incoming_id('outgoing'), 'incoming') + + def test_raises_assert_if_adding_conflicting_route(self): + self.rm.add('incoming', 'outgoing') + self.assertRaises(AssertionError, self.rm.add, 'different', 'outgoing') + diff --git a/distbuild/serialise.py b/distbuild/serialise.py new file mode 100644 index 00000000..0a60b0c2 --- /dev/null +++ b/distbuild/serialise.py @@ -0,0 +1,191 @@ +# distbuild/serialise.py -- (de)serialise Artifact object graphs +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import json + +import morphlib +import logging + + +def serialise_artifact(artifact): + '''Serialise an Artifact object and its dependencies into string form.''' + + def encode_morphology(morphology): + result = {} + for key in morphology.keys(): + result[key] = morphology[key] + return result + + def encode_source(source): + source_dic = { + 'name': source.name, + 'repo': None, + 'repo_name': source.repo_name, + 'original_ref': source.original_ref, + 'sha1': source.sha1, + 'tree': source.tree, + 'morphology': str(id(source.morphology)), + 'filename': source.filename, + + # dict keys are converted to strings by json + # so we encode the artifact ids as strings + 'artifact_ids': [str(id(artifact)) for (_, artifact) + in source.artifacts.iteritems()], + 'cache_id': source.cache_id, + 'cache_key': source.cache_key, + 'dependencies': [str(id(d)) + for d in source.dependencies], + } + + if source.morphology['kind'] == 'chunk': + source_dic['build_mode'] = source.build_mode + source_dic['prefix'] = source.prefix + return source_dic + + def encode_artifact(a): + if artifact.source.morphology['kind'] == 'system': # pragma: no cover + arch = artifact.source.morphology['arch'] + else: + arch = artifact.arch + + return { + 'source_id': id(a.source), + 'name': a.name, + 'arch': arch + } + + encoded_artifacts = {} + encoded_sources = {} + encoded_morphologies = {} + + for a in artifact.walk(): + if id(a.source) not in encoded_sources: + for (_, sa) in a.source.artifacts.iteritems(): + if id(sa) not in encoded_artifacts: + encoded_artifacts[id(sa)] = encode_artifact(sa) + encoded_morphologies[id(a.source.morphology)] = encode_morphology(a.source.morphology) + encoded_sources[id(a.source)] = encode_source(a.source) + + if id(a) not in encoded_artifacts: # pragma: no cover + encoded_artifacts[id(a)] = encode_artifact(a) + + return json.dumps({'sources': encoded_sources, + 'artifacts': encoded_artifacts, + 'morphologies': encoded_morphologies, + 'root_artifact': str(id(artifact)), + 'default_split_rules': { + 'chunk': morphlib.artifactsplitrule.DEFAULT_CHUNK_RULES, + 'stratum': morphlib.artifactsplitrule.DEFAULT_STRATUM_RULES, + }, + }) + + +def deserialise_artifact(encoded): + '''Re-construct the Artifact object (and dependencies). + + The argument should be a string returned by ``serialise_artifact``. + The reconstructed Artifact objects will be sufficiently like the + originals that they can be used as a build graph, and other such + purposes, by Morph. + + ''' + + def decode_morphology(le_dict): + '''Convert a dict into something that kinda acts like a Morphology. + + As it happens, we don't need the full Morphology so we cheat. + Cheating is good. + + ''' + + return morphlib.morphology.Morphology(le_dict) + + def decode_source(le_dict, morphology, split_rules): + '''Convert a dict into a Source object.''' + + source = morphlib.source.Source(le_dict['name'], + le_dict['repo_name'], + le_dict['original_ref'], + le_dict['sha1'], + le_dict['tree'], + morphology, + le_dict['filename'], + split_rules) + + if morphology['kind'] == 'chunk': + source.build_mode = le_dict['build_mode'] + source.prefix = le_dict['prefix'] + source.cache_id = le_dict['cache_id'] + source.cache_key = le_dict['cache_key'] + return source + + def decode_artifact(artifact_dict, source): + '''Convert dict into an Artifact object. + + Do not set dependencies, that will be dealt with later. + + ''' + + artifact = morphlib.artifact.Artifact(source, artifact_dict['name']) + artifact.arch = artifact_dict['arch'] + artifact.source = source + + return artifact + + le_dicts = json.loads(encoded) + artifacts_dict = le_dicts['artifacts'] + sources_dict = le_dicts['sources'] + morphologies_dict = le_dicts['morphologies'] + root_artifact = le_dicts['root_artifact'] + + artifact_ids = ([root_artifact] + artifacts_dict.keys()) + + artifacts = {} + sources = {} + morphologies = {id: decode_morphology(d) + for (id, d) in morphologies_dict.iteritems()} + + for source_id, source_dict in sources_dict.iteritems(): + morphology = morphologies[source_dict['morphology']] + kind = morphology['kind'] + ruler = getattr(morphlib.artifactsplitrule, 'unify_%s_matches' % kind) + rules = ruler(morphology, le_dicts['default_split_rules'][kind]) + sources[source_id] = decode_source(source_dict, morphology, rules) + + # clear the source artifacts that get automatically generated + # we want to add the ones that were sent to us + sources[source_id].artifacts = {} + source_artifacts = source_dict['artifact_ids'] + + for artifact_id in source_artifacts: + if artifact_id not in artifacts: + artifact_dict = artifacts_dict[artifact_id] + artifact = decode_artifact(artifact_dict, sources[source_id]) + + artifacts[artifact_id] = artifact + + key = artifacts[artifact_id].name + sources[source_id].artifacts[key] = artifacts[artifact_id] + + # now add the dependencies + for source_id, source_dict in sources_dict.iteritems(): + source = sources[source_id] + source.dependencies = [artifacts[aid] + for aid in source_dict['dependencies']] + + return artifacts[root_artifact] diff --git a/distbuild/serialise_tests.py b/distbuild/serialise_tests.py new file mode 100644 index 00000000..70973346 --- /dev/null +++ b/distbuild/serialise_tests.py @@ -0,0 +1,173 @@ +# distbuild/serialise_tests.py -- unit tests for Artifact serialisation +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import unittest + +import distbuild + + +class MockMorphology(object): + + def __init__(self, name, kind): + self.dict = { + 'name': '%s.morphology.name' % name, + 'kind': kind, + 'chunks': [], + 'products': [ + { + 'artifact': name, + 'include': [r'.*'], + }, + ], + } + + @property + def needs_artifact_metadata_cached(self): + return self.dict['kind'] == 'stratum' + + def keys(self): + return self.dict.keys() + + def __getitem__(self, key): + return self.dict[key] + + +class MockSource(object): + + build_mode = 'staging' + prefix = '/usr' + def __init__(self, name, kind): + self.name = name + self.repo = None + self.repo_name = '%s.source.repo_name' % name + self.original_ref = '%s.source.original_ref' % name + self.sha1 = '%s.source.sha1' % name + self.tree = '%s.source.tree' % name + self.morphology = MockMorphology(name, kind) + self.filename = '%s.source.filename' % name + self.dependencies = [] + self.cache_id = { + 'blip': '%s.blip' % name, + 'integer': 42, + } + self.cache_key = '%s.cache_key' % name + self.artifacts = {} + + +class MockArtifact(object): + + arch = 'testarch' + + def __init__(self, name, kind): + self.source = MockSource(name, kind) + self.source.artifacts = {name: self} + self.name = name + + def walk(self): # pragma: no cover + done = set() + + def depth_first(a): + if a not in done: + done.add(a) + for dep in a.source.dependencies: + for ret in depth_first(dep): + yield ret + yield a + + return list(depth_first(self)) + + +class SerialisationTests(unittest.TestCase): + + def setUp(self): + self.art1 = MockArtifact('name1', 'stratum') + self.art2 = MockArtifact('name2', 'chunk') + self.art3 = MockArtifact('name3', 'chunk') + self.art4 = MockArtifact('name4', 'chunk') + + def assertEqualMorphologies(self, a, b): + self.assertEqual(sorted(a.keys()), sorted(b.keys())) + keys = sorted(a.keys()) + a_values = [a[k] for k in keys] + b_values = [b[k] for k in keys] + self.assertEqual(a_values, b_values) + self.assertEqual(a.needs_artifact_metadata_cached, + b.needs_artifact_metadata_cached) + + def assertEqualSources(self, a, b): + self.assertEqual(a.repo, b.repo) + self.assertEqual(a.repo_name, b.repo_name) + self.assertEqual(a.original_ref, b.original_ref) + self.assertEqual(a.sha1, b.sha1) + self.assertEqual(a.tree, b.tree) + self.assertEqualMorphologies(a.morphology, b.morphology) + self.assertEqual(a.filename, b.filename) + + def assertEqualArtifacts(self, a, b): + self.assertEqualSources(a.source, b.source) + self.assertEqual(a.name, b.name) + self.assertEqual(a.source.cache_id, b.source.cache_id) + self.assertEqual(a.source.cache_key, b.source.cache_key) + self.assertEqual(len(a.source.dependencies), + len(b.source.dependencies)) + for i in range(len(a.source.dependencies)): + self.assertEqualArtifacts(a.source.dependencies[i], + b.source.dependencies[i]) + + def verify_round_trip(self, artifact): + encoded = distbuild.serialise_artifact(artifact) + decoded = distbuild.deserialise_artifact(encoded) + self.assertEqualArtifacts(artifact, decoded) + + objs = {} + queue = [decoded] + while queue: + obj = queue.pop() + k = obj.source.cache_key + if k in objs: + self.assertTrue(obj is objs[k]) + else: + objs[k] = obj + queue.extend(obj.source.dependencies) + + def test_returns_string(self): + encoded = distbuild.serialise_artifact(self.art1) + self.assertEqual(type(encoded), str) + + def test_works_without_dependencies(self): + self.verify_round_trip(self.art1) + + def test_works_with_single_dependency(self): + self.art1.source.dependencies = [self.art2] + self.verify_round_trip(self.art1) + + def test_works_with_two_dependencies(self): + self.art1.source.dependencies = [self.art2, self.art3] + self.verify_round_trip(self.art1) + + def test_works_with_two_levels_of_dependencies(self): + self.art2.source.dependencies = [self.art4] + self.art1.source.dependencies = [self.art2, self.art3] + self.verify_round_trip(self.art1) + + def test_works_with_dag(self): + self.art2.source.dependencies = [self.art4] + self.art3.source.dependencies = [self.art4] + self.art1.source.dependencies = [self.art2, self.art3] + self.verify_round_trip(self.art1) + diff --git a/distbuild/sm.py b/distbuild/sm.py new file mode 100644 index 00000000..e773962b --- /dev/null +++ b/distbuild/sm.py @@ -0,0 +1,151 @@ +# mainloop/sm.py -- state machine abstraction +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import logging +import re + + +classnamepat = re.compile(r"<class '(?P<name>.*)'>") + + +class StateMachine(object): + + '''A state machine abstraction. + + The caller may specify call backs for events coming from specific + event sources. An event source might, for example, be a socket + file descriptor, and the event might be incoming data from the + socket. The callback would then process the data, perhaps by + collecting it into a buffer and parsing out messages from it. + + A callback gets the event source and event as arguments. It returns + the new state, and a list of new events to + + A callback may return or yield new events, which will be handled + eventually. They may or may not be handled in order. + + There can only be one callback for one state, source, and event + class combination. + + States are represented by unique objects, e.g., strings containing + the names of the states. When a machine wants to stop, it sets its + state to None. + + ''' + + def __init__(self, initial_state): + self._transitions = {} + self.state = self._initial_state = initial_state + self.debug_transitions = False + + def setup(self): + '''Set up machine for execution. + + This is called when the machine is added to the main loop. + + ''' + + def _key(self, state, event_source, event_class): + return (state, event_source, event_class) + + def add_transition(self, state, source, event_class, new_state, callback): + '''Add a transition to the state machine. + + When the state machine is in the given state, and an event of + a given type comes from a given source, move the state machine + to the new state and call the callback function. + + ''' + + key = self._key(state, source, event_class) + assert key not in self._transitions, \ + 'Transition %s already registered' % str(key) + self._transitions[key] = (new_state, callback) + + def add_transitions(self, specification): + '''Add many transitions. + + The specification is a list of transitions. + Each transition is a tuple of the arguments given to + ``add_transition``. + + ''' + + for t in specification: + self.add_transition(*t) + + def handle_event(self, event_source, event): + '''Handle a given event. + + Return list of new events to handle. + + ''' + + key = self._key(self.state, event_source, event.__class__) + if key not in self._transitions: + if self.debug_transitions: # pragma: no cover + prefix = '%s: handle_event: ' % self.__class__.__name__ + logging.debug(prefix + 'not relevant for us: %s' % repr(event)) + logging.debug(prefix + 'key: %s', repr(key)) + logging.debug(prefix + 'state: %s', repr(self.state)) + return [] + + new_state, callback = self._transitions[key] + if self.debug_transitions: # pragma: no cover + logging.debug( + '%s: state change %s -> %s callback=%s' % + (self.__class__.__name__, self.state, new_state, + str(callback))) + self.state = new_state + if callback is not None: + ret = callback(event_source, event) + if ret is None: + return [] + else: + return list(ret) + else: + return [] + + def dump_dot(self, filename): # pragma: no cover + '''Write a Graphviz DOT file for the state machine.''' + + with open(filename, 'w') as f: + f.write('digraph %s {\n' % self._classname(self.__class__)) + first = True + for key in self._transitions: + state, src, event_class = key + if first: + f.write('"START" -> "%s" [label=""];\n' % + self._initial_state) + first = False + + new_state, callback = self._transitions[key] + if new_state is None: + new_state = 'END' + f.write('"%s" -> "%s" [label="%s"];\n' % + (state, new_state, self._classname(event_class))) + f.write('}\n') + + def _classname(self, klass): # pragma: no cover + s = str(klass) + m = classnamepat.match(s) + if m: + return m.group('name').split('.')[-1] + else: + return s + diff --git a/distbuild/sm_tests.py b/distbuild/sm_tests.py new file mode 100644 index 00000000..59b9c023 --- /dev/null +++ b/distbuild/sm_tests.py @@ -0,0 +1,98 @@ +# distbuild/sm_tests.py -- unit tests for state machine abstraction +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import unittest + +import distbuild + + +class DummyEventSource(object): + + pass + + +class DummyEvent(object): + + pass + + +class StateMachineTests(unittest.TestCase): + + def setUp(self): + self.sm = distbuild.StateMachine('init') + self.sm.distbuild = None + self.sm.setup() + self.event_source = DummyEventSource() + self.event = DummyEvent() + self.event_sources = [] + self.events = [] + self.callback_result = None + + def callback(self, event_source, event): + self.event_sources.append(event_source) + self.events.append(event) + return self.callback_result + + def test_ignores_event_when_there_are_no_transitions(self): + new_events = self.sm.handle_event(self.event_source, self.event) + self.assertEqual(new_events, []) + self.assertEqual(self.event_sources, []) + self.assertEqual(self.events, []) + + def test_ignores_event_when_no_transition_matches(self): + spec = [ + ('init', self.event_source, str, 'init', self.callback), + ] + self.sm.add_transitions(spec) + new_events = self.sm.handle_event(self.event_source, self.event) + self.assertEqual(new_events, []) + self.assertEqual(self.event_sources, []) + self.assertEqual(self.events, []) + + def test_handles_lack_of_callback_ok(self): + spec = [ + ('init', self.event_source, DummyEvent, 'init', None), + ] + self.sm.add_transitions(spec) + new_events = self.sm.handle_event(self.event_source, self.event) + self.assertEqual(new_events, []) + self.assertEqual(self.event_sources, []) + self.assertEqual(self.events, []) + + def test_calls_registered_callback_for_right_event(self): + spec = [ + ('init', self.event_source, DummyEvent, 'init', self.callback), + ] + self.sm.add_transitions(spec) + new_events = self.sm.handle_event(self.event_source, self.event) + self.assertEqual(new_events, []) + self.assertEqual(self.event_sources, [self.event_source]) + self.assertEqual(self.events, [self.event]) + + def test_handle_converts_nonlist_to_list(self): + self.callback_result = ('foo', 'bar') + + spec = [ + ('init', self.event_source, DummyEvent, 'init', self.callback), + ] + self.sm.add_transitions(spec) + new_events = self.sm.handle_event(self.event_source, self.event) + self.assertEqual(new_events, ['foo', 'bar']) + self.assertEqual(self.event_sources, [self.event_source]) + self.assertEqual(self.events, [self.event]) + diff --git a/distbuild/sockbuf.py b/distbuild/sockbuf.py new file mode 100644 index 00000000..fc0315b0 --- /dev/null +++ b/distbuild/sockbuf.py @@ -0,0 +1,180 @@ +# mainloop/sockbuf.py -- a buffering, non-blocking socket I/O state machine +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import logging + + +'''A buffering, non-blocking I/O state machine for sockets. + +The state machine is given an open socket. It reads from the socket, +and writes to it, when it can do so without blocking. A maximum size +for the read buffer can be set: the state machine will stop reading +if the buffer becomes full. This avoids the problem of an excessively +large buffer. + +The state machine generates events to indicate that the buffer contains +data or that the end of the file for reading has been reached. An event +is also generated if there is an error while doing I/O with the socket. + +* SocketError: an error has occurred +* SocketBufferNewData: socket buffer has received new data; the data + is available as the ``data`` attribute +* SocketBufferEof: socket buffer has reached EOF for reading, but + still writes anything in the write buffer (or anything that gets added + to the write buffer) +* SocketBufferClosed: socket is now closed + +The state machine starts shutting down when ``close`` method is called, +but continues to operate in write-only mode until the write buffer has +been emptied. + +''' + + +from socketsrc import (SocketError, SocketReadable, SocketWriteable, + SocketEventSource) +from sm import StateMachine +from stringbuffer import StringBuffer + + +class SocketBufferNewData(object): + + '''Socket buffer has received new data.''' + + def __init__(self, data): + self.data = data + + +class SocketBufferEof(object): + + '''Socket buffer has reached end of file when reading. + + Note that the socket buffer may still be available for writing. + However, no more new data will be read. + + ''' + + +class SocketBufferClosed(object): + + '''Socket buffer has closed its socket.''' + + +class _Close(object): pass +class _WriteBufferIsEmpty(object): pass +class _WriteBufferNotEmpty(object): pass + + + +class SocketBuffer(StateMachine): + + def __init__(self, sock, max_buffer): + StateMachine.__init__(self, 'reading') + + self._sock = sock + self._max_buffer = max_buffer + + def __repr__(self): + return '<SocketBuffer at 0x%x: socket %s max_buffer %i>' % ( + id(self), self._sock, self._max_buffer) + + def setup(self): + src = self._src = SocketEventSource(self._sock) + src.stop_writing() # We'll start writing when we need to. + self.mainloop.add_event_source(src) + + self._wbuf = StringBuffer() + + spec = [ + # state, source, event_class, new_state, callback + ('reading', src, SocketReadable, 'reading', self._fill), + ('reading', self, _WriteBufferNotEmpty, 'rw', + self._start_writing), + ('reading', self, SocketBufferEof, 'idle', None), + ('reading', self, _Close, None, self._really_close), + + ('rw', src, SocketReadable, 'rw', self._fill), + ('rw', src, SocketWriteable, 'rw', self._flush), + ('rw', self, _WriteBufferIsEmpty, 'reading', self._stop_writing), + ('rw', self, SocketBufferEof, 'w', None), + ('rw', self, _Close, 'wc', None), + + ('idle', self, _WriteBufferNotEmpty, 'w', self._start_writing), + ('idle', self, _Close, None, self._really_close), + + ('w', src, SocketWriteable, 'w', self._flush), + ('w', self, _WriteBufferIsEmpty, 'idle', self._stop_writing), + + ('wc', src, SocketWriteable, 'wc', self._flush), + ('wc', self, _WriteBufferIsEmpty, None, self._really_close), + ] + self.add_transitions(spec) + + def write(self, data): + '''Put data into write queue.''' + + was_empty = len(self._wbuf) == 0 + self._wbuf.add(data) + if was_empty and len(self._wbuf) > 0: + self._start_writing(None, None) + self.mainloop.queue_event(self, _WriteBufferNotEmpty()) + + def close(self): + '''Tell state machine to terminate.''' + self.mainloop.queue_event(self, _Close()) + + def _report_error(self, event_source, event): + logging.error(str(event)) + + def _fill(self, event_source, event): + try: + data = event.sock.read(self._max_buffer) + except (IOError, OSError), e: + logging.debug( + '%s: _fill(): Exception %s from sock.read()', self, e) + return [SocketError(event.sock, e)] + + if data: + self.mainloop.queue_event(self, SocketBufferNewData(data)) + else: + event_source.stop_reading() + self.mainloop.queue_event(self, SocketBufferEof()) + + def _really_close(self, event_source, event): + self._src.close() + self.mainloop.queue_event(self, SocketBufferClosed()) + + def _flush(self, event_source, event): + max_write = 1024**2 + data = self._wbuf.read(max_write) + try: + n = event.sock.write(data) + except (IOError, OSError), e: + logging.debug( + '%s: _flush(): Exception %s from sock.write()', self, e) + return [SocketError(event.sock, e)] + self._wbuf.remove(n) + if len(self._wbuf) == 0: + self.mainloop.queue_event(self, _WriteBufferIsEmpty()) + + def _start_writing(self, event_source, event): + self._src.start_writing() + + def _stop_writing(self, event_source, event): + self._src.stop_writing() + diff --git a/distbuild/socketsrc.py b/distbuild/socketsrc.py new file mode 100644 index 00000000..15283140 --- /dev/null +++ b/distbuild/socketsrc.py @@ -0,0 +1,184 @@ +# mainloop/socketsrc.py -- events and event sources for sockets +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import fcntl +import logging +import os +import socket + +import distbuild + +from eventsrc import EventSource + + +def set_nonblocking(handle): + '''Make a socket, file descriptor, or other such thing be non-blocking.''' + + if type(handle) is int: + fd = handle + else: + fd = handle.fileno() + + flags = fcntl.fcntl(fd, fcntl.F_GETFL, 0) + flags = flags | os.O_NONBLOCK + fcntl.fcntl(fd, fcntl.F_SETFL, flags) + + +class SocketError(object): + + '''An error has occured with a socket.''' + + def __init__(self, sock, exception): + self.sock = sock + self.exception = exception + + +class NewConnection(object): + + '''A new client connection.''' + + def __init__(self, connection, addr): + self.connection = connection + self.addr = addr + + +class ListeningSocketEventSource(EventSource): + + '''An event source for a socket that listens for connections.''' + + def __init__(self, addr, port): + self.sock = distbuild.create_socket() + self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + logging.info('Binding socket to %s', addr) + self.sock.bind((addr, port)) + self.sock.listen(5) + self._accepting = True + logging.info('Listening at %s' % self.sock.remotename()) + + def get_select_params(self): + r = [self.sock.fileno()] if self._accepting else [] + return r, [], [], None + + def get_events(self, r, w, x): + if self._accepting and self.sock.fileno() in r: + try: + conn, addr = self.sock.accept() + except socket.error, e: + return [SocketError(self.sock, e)] + else: + logging.info( + 'New connection to %s from %s' % + (conn.getsockname(), addr)) + return [NewConnection(conn, addr)] + + return [] + + def start_accepting(self): + self._accepting = True + + def stop_accepting(self): + self._accepting = False + + +class SocketReadable(object): + + '''A socket is readable.''' + + def __init__(self, sock): + self.sock = sock + + +class SocketWriteable(object): + + '''A socket is writeable.''' + + def __init__(self, sock): + self.sock = sock + + +class SocketEventSource(EventSource): + + '''Event source for normal sockets (for I/O). + + This generates events for indicating the socket is readable or + writeable. It does not actually do any I/O itself, that's for the + handler of the events. There are, however, methods for doing the + reading/writing, and for closing the socket. + + The event source can be told to stop checking for readability + or writeability, so that the user may, for example, stop those + events from being triggered while a buffer is full. + + ''' + + def __init__(self, sock): + self.sock = sock + self._reading = True + self._writing = True + + set_nonblocking(sock) + + def __repr__(self): + return '<SocketEventSource at %x: socket %s>' % (id(self), self.sock) + + def get_select_params(self): + r = [self.sock.fileno()] if self._reading else [] + w = [self.sock.fileno()] if self._writing else [] + return r, w, [], None + + def get_events(self, r, w, x): + events = [] + fd = self.sock.fileno() + + if self._reading and fd in r: + events.append(SocketReadable(self)) + + if self._writing and fd in w: + events.append(SocketWriteable(self)) + + return events + + def start_reading(self): + self._reading = True + + def stop_reading(self): + self._reading = False + + def start_writing(self): + self._writing = True + + def stop_writing(self): + self._writing = False + + def read(self, max_bytes): + fd = self.sock.fileno() + return os.read(fd, max_bytes) + + def write(self, data): + fd = self.sock.fileno() + return os.write(fd, data) + + def close(self): + self.stop_reading() + self.stop_writing() + self.sock.close() + self.sock = None + + def is_finished(self): + return self.sock is None + diff --git a/distbuild/sockserv.py b/distbuild/sockserv.py new file mode 100644 index 00000000..68991a93 --- /dev/null +++ b/distbuild/sockserv.py @@ -0,0 +1,63 @@ +# mainloop/sockserv.py -- socket server state machines +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import logging + +from sm import StateMachine +from socketsrc import NewConnection, SocketError, ListeningSocketEventSource + + +class ListenServer(StateMachine): + + '''Listen for new connections on a port, send events for them.''' + + def __init__(self, addr, port, machine, extra_args=None, port_file=''): + StateMachine.__init__(self, 'listening') + self._addr = addr + self._port = port + self._machine = machine + self._extra_args = extra_args or [] + self._port_file = port_file + + def setup(self): + src = ListeningSocketEventSource(self._addr, self._port) + if self._port_file: + host, port = src.sock.getsockname() + with open(self._port_file, 'w') as f: + f.write(port) + self.mainloop.add_event_source(src) + + spec = [ + # state, source, event_class, new_state, callback + ('listening', src, NewConnection, 'listening', self.new_conn), + ('listening', src, SocketError, None, self.report_error), + ] + self.add_transitions(spec) + + def new_conn(self, event_source, event): + logging.debug( + 'ListenServer: Creating new %s using %s and %s' % + (self._machine, + repr(event.connection), + repr(self._extra_args))) + m = self._machine(event.connection, *self._extra_args) + self.mainloop.add_state_machine(m) + + def report_error(self, event_source, event): + logging.error(str(event)) + diff --git a/distbuild/stringbuffer.py b/distbuild/stringbuffer.py new file mode 100644 index 00000000..2b94dd19 --- /dev/null +++ b/distbuild/stringbuffer.py @@ -0,0 +1,102 @@ +# mainloop/stringbuffer.py -- efficient buffering of strings as a queue +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +class StringBuffer(object): + + '''Buffer data for a file descriptor. + + The data may arrive in small pieces, and it is buffered in a way that + avoids excessive string catenation or splitting. + + ''' + + def __init__(self): + self.strings = [] + self.len = 0 + + def add(self, data): + '''Add data to buffer.''' + self.strings.append(data) + self.len += len(data) + + def remove(self, num_bytes): + '''Remove specified number of bytes from buffer.''' + while num_bytes > 0 and self.strings: + first = self.strings[0] + if len(first) <= num_bytes: + num_bytes -= len(first) + del self.strings[0] + self.len -= len(first) + else: + self.strings[0] = first[num_bytes:] + self.len -= num_bytes + num_bytes = 0 + + def peek(self): + '''Return contents of buffer as one string.''' + + if len(self.strings) == 0: + return '' + elif len(self.strings) == 1: + return self.strings[0] + else: + self.strings = [''.join(self.strings)] + return self.strings[0] + + def read(self, max_bytes): + '''Return up to max_bytes from the buffer. + + Less is returned if the buffer does not contain at least max_bytes. + The returned data will remain in the buffer; use remove to remove + it. + + ''' + + use = [] + size = 0 + for s in self.strings: + n = max_bytes - size + if len(s) <= n: + use.append(s) + size += len(s) + else: + use.append(s[:n]) + size += n + break + return ''.join(use) + + def readline(self): + '''Return a complete line (ends with '\n') or None.''' + + for i, s in enumerate(self.strings): + newline = s.find('\n') + if newline != -1: + if newline+1 == len(s): + use = self.strings[:i+1] + del self.strings[:i+1] + else: + pre = s[:newline+1] + use = self.strings[:i] + [pre] + del self.strings[:i] + self.strings[0] = s[newline+1:] + return ''.join(use) + return None + + def __len__(self): + return self.len + diff --git a/distbuild/stringbuffer_tests.py b/distbuild/stringbuffer_tests.py new file mode 100644 index 00000000..da324f20 --- /dev/null +++ b/distbuild/stringbuffer_tests.py @@ -0,0 +1,152 @@ +# distbuild/stringbuffer_tests.py -- unit tests +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import unittest + +import distbuild + + +class StringBufferTests(unittest.TestCase): + + def setUp(self): + self.buf = distbuild.StringBuffer() + + def test_is_empty_initially(self): + self.assertEqual(self.buf.peek(), '') + self.assertEqual(len(self.buf), 0) + + def test_adds_a_string(self): + s = 'foo' + self.buf.add(s) + self.assertEqual(self.buf.peek(), s) + self.assertEqual(len(self.buf), len(s)) + + def test_adds_a_second_string(self): + s = 'foo' + t = 'bar' + self.buf.add(s) + self.buf.add(t) + self.assertEqual(self.buf.peek(), s + t) + self.assertEqual(len(self.buf), len(s + t)) + + +class StringBufferRemoveTests(unittest.TestCase): + + def setUp(self): + self.buf = distbuild.StringBuffer() + self.first = 'foo' + self.second = 'bar' + self.all = self.first + self.second + self.buf.add(self.first) + self.buf.add(self.second) + + def test_removes_part_of_first_string(self): + self.assertTrue(len(self.first) > 1) + self.buf.remove(1) + self.assertEqual(self.buf.peek(), self.all[1:]) + self.assertEqual(len(self.buf), len(self.all) - 1) + + def test_removes_all_of_first_string(self): + self.buf.remove(len(self.first)) + self.assertEqual(self.buf.peek(), self.second) + self.assertEqual(len(self.buf), len(self.second)) + + def test_removes_more_than_first_string(self): + self.assertTrue(len(self.first) > 1) + self.assertTrue(len(self.second) > 1) + self.buf.remove(len(self.first) + 1) + self.assertEqual(self.buf.peek(), self.second[1:]) + self.assertEqual(len(self.buf), len(self.second) - 1) + + def test_removes_all_strings(self): + self.buf.remove(len(self.all)) + self.assertEqual(self.buf.peek(), '') + self.assertEqual(len(self.buf), 0) + + def test_removes_more_than_all_strings(self): + self.buf.remove(len(self.all) + 1) + self.assertEqual(self.buf.peek(), '') + self.assertEqual(len(self.buf), 0) + + +class StringBufferReadTests(unittest.TestCase): + + def setUp(self): + self.buf = distbuild.StringBuffer() + + def test_returns_empty_string_for_empty_buffer(self): + self.assertEqual(self.buf.read(100), '') + self.assertEqual(self.buf.peek(), '') + + def test_returns_partial_string_for_short_buffer(self): + self.buf.add('foo') + self.assertEqual(self.buf.read(100), 'foo') + self.assertEqual(self.buf.peek(), 'foo') + + def test_returns_catenated_strings(self): + self.buf.add('foo') + self.buf.add('bar') + self.assertEqual(self.buf.read(100), 'foobar') + self.assertEqual(self.buf.peek(), 'foobar') + + def test_returns_requested_amount_when_available(self): + self.buf.add('foo') + self.buf.add('bar') + self.assertEqual(self.buf.read(4), 'foob') + self.assertEqual(self.buf.peek(), 'foobar') + + +class StringBufferReadlineTests(unittest.TestCase): + + def setUp(self): + self.buf = distbuild.StringBuffer() + + def test_returns_None_on_empty_buffer(self): + self.assertEqual(self.buf.readline(), None) + + def test_returns_None_on_incomplete_line_in_buffer(self): + self.buf.add('foo') + self.assertEqual(self.buf.readline(), None) + + def test_extracts_complete_line(self): + self.buf.add('foo\n') + self.assertEqual(self.buf.readline(), 'foo\n') + self.assertEqual(self.buf.peek(), '') + + def test_extracts_only_the_initial_line_and_leaves_rest_of_buffer(self): + self.buf.add('foo\nbar\n') + self.assertEqual(self.buf.readline(), 'foo\n') + self.assertEqual(self.buf.peek(), 'bar\n') + + def test_extracts_only_the_initial_line_and_leaves_partial_line(self): + self.buf.add('foo\nbar') + self.assertEqual(self.buf.readline(), 'foo\n') + self.assertEqual(self.buf.peek(), 'bar') + + def test_extracts_only_the_initial_line_from_multiple_pieces(self): + self.buf.add('foo\n') + self.buf.add('bar\n') + self.assertEqual(self.buf.readline(), 'foo\n') + self.assertEqual(self.buf.peek(), 'bar\n') + + def test_extracts_only_the_initial_line_from_multiple_pieces_incomp(self): + self.buf.add('foo\n') + self.buf.add('bar') + self.assertEqual(self.buf.readline(), 'foo\n') + self.assertEqual(self.buf.peek(), 'bar') + diff --git a/distbuild/timer_event_source.py b/distbuild/timer_event_source.py new file mode 100644 index 00000000..4a2e81b7 --- /dev/null +++ b/distbuild/timer_event_source.py @@ -0,0 +1,59 @@ +# distbuild/timer_event_source.py -- event source for timer events +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import time + + +class Timer(object): + + pass + + +class TimerEventSource(object): + + def __init__(self, interval): + self.interval = interval + self.last_event = time.time() + self.enabled = False + + def start(self): + self.enabled = True + self.last_event = time.time() + + def stop(self): + self.enabled = False + + def get_select_params(self): + if self.enabled: + next_event = self.last_event + self.interval + timeout = next_event - time.time() + return [], [], [], max(0, timeout) + else: + return [], [], [], None + + def get_events(self, r, w, x): + if self.enabled: + now = time.time() + if now >= self.last_event + self.interval: + self.last_event = now + return [Timer()] + return [] + + def is_finished(self): + return False + diff --git a/distbuild/worker_build_scheduler.py b/distbuild/worker_build_scheduler.py new file mode 100644 index 00000000..6cda5972 --- /dev/null +++ b/distbuild/worker_build_scheduler.py @@ -0,0 +1,620 @@ +# distbuild/worker_build_scheduler.py -- schedule worker-builds on workers +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import collections +import httplib +import logging +import socket +import urllib +import urlparse + +import distbuild + + +class WorkerBuildRequest(object): + + def __init__(self, artifact, initiator_id): + self.artifact = artifact + self.initiator_id = initiator_id + +class WorkerCancelPending(object): + + def __init__(self, initiator_id): + self.initiator_id = initiator_id + +class WorkerBuildStepStarted(object): + + def __init__(self, initiators, cache_key, worker_name): + self.initiators = initiators + self.artifact_cache_key = cache_key + self.worker_name = worker_name + +class WorkerBuildStepAlreadyStarted(object): + + def __init__(self, initiator_id, cache_key, worker_name): + self.initiator_id = initiator_id + self.artifact_cache_key = cache_key + self.worker_name = worker_name + +class WorkerBuildWaiting(object): + + def __init__(self, initiator_id, cache_key): + self.initiator_id = initiator_id + self.artifact_cache_key = cache_key + +class WorkerBuildOutput(object): + + def __init__(self, msg, cache_key): + self.msg = msg + self.artifact_cache_key = cache_key + +class WorkerBuildCaching(object): + + def __init__(self, initiators, cache_key): + self.initiators = initiators + self.artifact_cache_key = cache_key + +class WorkerBuildFinished(object): + + def __init__(self, msg, cache_key): + self.msg = msg + self.artifact_cache_key = cache_key + +class WorkerBuildFailed(object): + + def __init__(self, msg, cache_key): + self.msg = msg + self.artifact_cache_key = cache_key + + +class _NeedJob(object): + + def __init__(self, who): + self.who = who + + +class _HaveAJob(object): + + def __init__(self, job): + self.job = job + +class Job(object): + + def __init__(self, job_id, artifact, initiator_id): + self.id = job_id + self.artifact = artifact + self.initiators = [initiator_id] + self.who = None # we don't know who's going to do this yet + self.running = False + self.failed = False + + +class Jobs(object): + + def __init__(self, idgen): + self._idgen = idgen + self._jobs = {} + + def get(self, artifact_basename): + return (self._jobs[artifact_basename] + if artifact_basename in self._jobs else None) + + def create(self, artifact, initiator_id): + job = Job(self._idgen.next(), artifact, initiator_id) + self._jobs[job.artifact.basename()] = job + return job + + def remove(self, job): + if job.artifact.basename() in self._jobs: + del self._jobs[job.artifact.basename()] + else: + logging.warning("Tried to remove a job that doesn't exist " + "(%s)", job.artifact.basename()) + + def get_jobs(self): + return self._jobs + + def remove_jobs(self, jobs): + for job in jobs: + self.remove(job) + + def exists(self, artifact_basename): + return artifact_basename in self._jobs + + def get_next_job(self): + # for now just return the first thing we find that's not being built + waiting = [job for (_, job) in + self._jobs.iteritems() if job.who == None] + + return waiting.pop() if len(waiting) > 0 else None + + def __repr__(self): + return str([job.artifact.basename() + for (_, job) in self._jobs.iteritems()]) + + +class _BuildFinished(object): + + pass + + +class _BuildFailed(object): + + pass + + +class _BuildCancelled(object): + + pass + + +class _Cached(object): + + pass + + +class _JobStarted(object): + + def __init__(self, job): + self.job = job + + +class _JobFinished(object): + + def __init__(self, job): + self.job = job + + +class _JobFailed(object): + + def __init__(self, job): + self.job = job + +class WorkerBuildQueuer(distbuild.StateMachine): + + '''Maintain queue of outstanding worker-build requests. + + This state machine captures WorkerBuildRequest events, and puts them + into a queue. It also catches _NeedJob events, from a + WorkerConnection, and responds to them with _HaveAJob events, + when it has an outstanding request. + + ''' + + def __init__(self): + distbuild.StateMachine.__init__(self, 'idle') + + def setup(self): + distbuild.crash_point() + + logging.debug('WBQ: Setting up %s' % self) + self._available_workers = [] + self._jobs = Jobs( + distbuild.IdentifierGenerator('WorkerBuildQueuerJob')) + + spec = [ + # state, source, event_class, new_state, callback + ('idle', WorkerBuildQueuer, WorkerBuildRequest, 'idle', + self._handle_request), + ('idle', WorkerBuildQueuer, WorkerCancelPending, 'idle', + self._handle_cancel), + + ('idle', WorkerConnection, _NeedJob, 'idle', self._handle_worker), + ('idle', WorkerConnection, _JobStarted, 'idle', + self._set_job_started), + ('idle', WorkerConnection, _JobFinished, 'idle', + self._set_job_finished), + ('idle', WorkerConnection, _JobFailed, 'idle', + self._set_job_failed) + ] + self.add_transitions(spec) + + def _set_job_started(self, event_source, event): + logging.debug('Setting job state for job %s with id %s: ' + 'Job is running', + event.job.artifact.basename(), event.job.id) + + event.job.running = True + + def _set_job_finished(self, event_source, event): + logging.debug('Setting job state for job %s with id %s: ' + 'Job is NOT running', + event.job.artifact.basename(), event.job.id) + + event.job.running = False + + def _set_job_failed(self, event_source, event): + logging.debug('Job %s with id %s failed', + event.job.artifact.basename(), event.job.id) + event.job.failed = True + + def _handle_request(self, event_source, event): + distbuild.crash_point() + + logging.debug('Handling build request for %s' % event.initiator_id) + logging.debug('Current jobs: %s' % self._jobs) + logging.debug('Workers available: %d' % len(self._available_workers)) + + # Have we already made a job for this thing? + # If so, add our initiator id to the existing job + # If not, create a job + + if self._jobs.exists(event.artifact.basename()): + job = self._jobs.get(event.artifact.basename()) + job.initiators.append(event.initiator_id) + + if job.running: + logging.debug('Worker build step already started: %s' % + event.artifact.basename()) + progress = WorkerBuildStepAlreadyStarted(event.initiator_id, + event.artifact.cache_key, job.who.name()) + else: + logging.debug('Job created but not building yet ' + '(waiting for a worker to become available): %s' % + event.artifact.basename()) + progress = WorkerBuildWaiting(event.initiator_id, + event.artifact.cache_key) + + self.mainloop.queue_event(WorkerConnection, progress) + else: + logging.debug('WBQ: Creating job for: %s' % event.artifact.name) + job = self._jobs.create(event.artifact, event.initiator_id) + + if self._available_workers: + self._give_job(job) + else: + progress = WorkerBuildWaiting(event.initiator_id, + event.artifact.cache_key) + self.mainloop.queue_event(WorkerConnection, progress) + + def _handle_cancel(self, event_source, event): + + def cancel_this(job): + if event.initiator_id not in job.initiators: + return False # not for us + + name = job.artifact.basename() + job_id = job.id + + logging.debug('Checking whether to remove job %s with job id %s', + name, job_id) + + if len(job.initiators) == 1: + if job.running or job.failed: + logging.debug('NOT removing running job %s with job id %s ' + '(WorkerConnection will cancel job)', + name, job_id) + else: + logging.debug('Removing job %s with job id %s', + name, job_id) + return True + else: + # Don't cancel, but still remove this initiator from + # the list of initiators + logging.debug('NOT removing job %s with job id %s ' + 'other initiators want it: %s', name, job_id, + [i for i in job.initiators + if i != event.initiator_id]) + + job.initiators.remove(event.initiator_id) + + return False + + self._jobs.remove_jobs( + [job for (_, job) in self._jobs.get_jobs().iteritems() + if cancel_this(job)]) + + def _handle_worker(self, event_source, event): + distbuild.crash_point() + + who = event.who + last_job = who.job() # the job this worker's just completed + + if last_job: + logging.debug('%s wants new job, just did %s', + who.name(), last_job.artifact.basename()) + + logging.debug('Removing job %s with job id %s', + last_job.artifact.basename(), last_job.id) + self._jobs.remove(last_job) + else: + logging.debug('%s wants its first job', who.name()) + + logging.debug('WBQ: Adding worker to queue: %s', event.who.name()) + self._available_workers.append(event) + + logging.debug('Current jobs: %s', self._jobs) + logging.debug('Workers available: %d', len(self._available_workers)) + + job = self._jobs.get_next_job() + + if job: + self._give_job(job) + + def _give_job(self, job): + worker = self._available_workers.pop(0) + job.who = worker.who + + logging.debug( + 'WBQ: Giving %s to %s' % + (job.artifact.name, worker.who.name())) + + self.mainloop.queue_event(worker.who, _HaveAJob(job)) + + +class WorkerConnection(distbuild.StateMachine): + + '''Communicate with a single worker.''' + + _request_ids = distbuild.IdentifierGenerator('WorkerConnection') + _initiator_request_map = collections.defaultdict(set) + + def __init__(self, cm, conn, writeable_cache_server, + worker_cache_server_port, morph_instance): + distbuild.StateMachine.__init__(self, 'idle') + self._cm = cm + self._conn = conn + self._writeable_cache_server = writeable_cache_server + self._worker_cache_server_port = worker_cache_server_port + self._morph_instance = morph_instance + self._helper_id = None + self._job = None + self._exec_response_msg = None + self._debug_json = False + + addr, port = self._conn.getpeername() + name = socket.getfqdn(addr) + self._worker_name = '%s:%s' % (name, port) + + def name(self): + return self._worker_name + + def job(self): + return self._job + + def setup(self): + distbuild.crash_point() + + logging.debug('WC: Setting up instance %s' % repr(self)) + + self._jm = distbuild.JsonMachine(self._conn) + self.mainloop.add_state_machine(self._jm) + + spec = [ + # state, source, event_class, new_state, callback + ('idle', self._jm, distbuild.JsonEof, None, self._reconnect), + ('idle', self, _HaveAJob, 'building', self._start_build), + + ('building', distbuild.BuildController, + distbuild.BuildCancel, 'building', + self._maybe_cancel), + + ('building', self._jm, distbuild.JsonEof, None, self._reconnect), + ('building', self._jm, distbuild.JsonNewMessage, 'building', + self._handle_json_message), + ('building', self, _BuildFailed, 'idle', self._request_job), + ('building', self, _BuildCancelled, 'idle', self._request_job), + ('building', self, _BuildFinished, 'caching', + self._request_caching), + + ('caching', distbuild.HelperRouter, distbuild.HelperResult, + 'caching', self._maybe_handle_helper_result), + ('caching', self, _Cached, 'idle', self._request_job), + ('caching', self, _BuildFailed, 'idle', self._request_job), + ] + self.add_transitions(spec) + + self._request_job(None, None) + + def _maybe_cancel(self, event_source, build_cancel): + + if build_cancel.id not in self._job.initiators: + return # event not relevant + + logging.debug('WC: BuildController %r requested a cancel', + event_source) + + if (len(self._job.initiators) == 1): + logging.debug('WC: Cancelling running job %s ' + 'with job id %s running on %s', + self._job.artifact.basename(), + self._job.id, + self.name()) + + msg = distbuild.message('exec-cancel', id=self._job.id) + self._jm.send(msg) + self.mainloop.queue_event(self, _BuildCancelled()) + else: + logging.debug('WC: Not cancelling running job %s with job id %s, ' + 'other initiators want it done: %s', + self._job.artifact.basename(), + self._job.id, + [i for i in self._job.initiators + if i != build_cancel.id]) + + self._job.initiators.remove(build_cancel.id) + + def _reconnect(self, event_source, event): + distbuild.crash_point() + + logging.debug('WC: Triggering reconnect') + self.mainloop.queue_event(self._cm, distbuild.Reconnect()) + + def _start_build(self, event_source, event): + distbuild.crash_point() + + self._job = event.job + self._helper_id = None + self._exec_response_msg = None + + logging.debug('WC: starting build: %s for %s' % + (self._job.artifact.name, self._job.initiators)) + + argv = [ + self._morph_instance, + 'worker-build', + '--build-log-on-stdout', + self._job.artifact.name, + ] + msg = distbuild.message('exec-request', + id=self._job.id, + argv=argv, + stdin_contents=distbuild.serialise_artifact(self._job.artifact), + ) + self._jm.send(msg) + + if self._debug_json: + logging.debug('WC: sent to worker %s: %r' + % (self._worker_name, msg)) + + started = WorkerBuildStepStarted(self._job.initiators, + self._job.artifact.cache_key, self.name()) + + self.mainloop.queue_event(WorkerConnection, _JobStarted(self._job)) + self.mainloop.queue_event(WorkerConnection, started) + + def _handle_json_message(self, event_source, event): + '''Handle JSON messages from the worker.''' + + distbuild.crash_point() + + logging.debug( + 'WC: from worker %s: %r' % (self._worker_name, event.msg)) + + handlers = { + 'exec-output': self._handle_exec_output, + 'exec-response': self._handle_exec_response, + } + + handler = handlers[event.msg['type']] + handler(event.msg) + + def _handle_exec_output(self, msg): + new = dict(msg) + new['ids'] = self._job.initiators + logging.debug('WC: emitting: %s', repr(new)) + self.mainloop.queue_event( + WorkerConnection, + WorkerBuildOutput(new, self._job.artifact.cache_key)) + + def _handle_exec_response(self, msg): + logging.debug('WC: finished building: %s' % self._job.artifact.name) + logging.debug('initiators that need to know: %s' + % self._job.initiators) + + new = dict(msg) + new['ids'] = self._job.initiators + + if new['exit'] != 0: + # Build failed. + new_event = WorkerBuildFailed(new, self._job.artifact.cache_key) + self.mainloop.queue_event(WorkerConnection, new_event) + self.mainloop.queue_event(WorkerConnection, _JobFailed(self._job)) + self.mainloop.queue_event(self, _BuildFailed()) + else: + # Build succeeded. We have more work to do: caching the result. + self.mainloop.queue_event(self, _BuildFinished()) + self._exec_response_msg = new + + def _request_job(self, event_source, event): + distbuild.crash_point() + self.mainloop.queue_event(WorkerConnection, _NeedJob(self)) + + def _request_caching(self, event_source, event): + # This code should be moved into the morphlib.remoteartifactcache + # module. It would be good to share it with morphlib.buildcommand, + # which also wants to fetch artifacts from a remote cache. + distbuild.crash_point() + + logging.debug('Requesting shared artifact cache to get artifacts') + + kind = self._job.artifact.source.morphology['kind'] + + if kind == 'chunk': + source_artifacts = self._job.artifact.source.artifacts + + suffixes = ['%s.%s' % (kind, name) for name in source_artifacts] + suffixes.append('build-log') + else: + filename = '%s.%s' % (kind, self._job.artifact.name) + suffixes = [filename] + + if kind == 'stratum': + suffixes.append(filename + '.meta') + elif kind == 'system': + # FIXME: This is a really ugly hack. + if filename.endswith('-rootfs'): + suffixes.append(filename[:-len('-rootfs')] + '-kernel') + + suffixes = [urllib.quote(x) for x in suffixes] + suffixes = ','.join(suffixes) + + worker_host = self._conn.getpeername()[0] + + url = urlparse.urljoin( + self._writeable_cache_server, + '/1.0/fetch?host=%s:%d&cacheid=%s&artifacts=%s' % + (urllib.quote(worker_host), + self._worker_cache_server_port, + urllib.quote(self._job.artifact.cache_key), + suffixes)) + + msg = distbuild.message( + 'http-request', id=self._request_ids.next(), url=url, + method='GET', body=None, headers=None) + self._helper_id = msg['id'] + req = distbuild.HelperRequest(msg) + self.mainloop.queue_event(distbuild.HelperRouter, req) + + progress = WorkerBuildCaching(self._job.initiators, + self._job.artifact.cache_key) + self.mainloop.queue_event(WorkerConnection, progress) + + def _maybe_handle_helper_result(self, event_source, event): + if event.msg['id'] == self._helper_id: + distbuild.crash_point() + + logging.debug('caching: event.msg: %s' % repr(event.msg)) + if event.msg['status'] == httplib.OK: + logging.debug('Shared artifact cache population done') + + new_event = WorkerBuildFinished( + self._exec_response_msg, self._job.artifact.cache_key) + self.mainloop.queue_event(WorkerConnection, new_event) + self.mainloop.queue_event(self, _Cached()) + else: + logging.error( + 'Failed to populate artifact cache: %s %s' % + (event.msg['status'], event.msg['body'])) + + # We will attempt to remove this job twice + # unless we mark it as failed before the BuildController + # processes the WorkerBuildFailed event. + # + # The BuildController will not try to cancel jobs that have + # been marked as failed. + self.mainloop.queue_event(WorkerConnection, + _JobFailed(self._job)) + + new_event = WorkerBuildFailed( + self._exec_response_msg, self._job.artifact.cache_key) + self.mainloop.queue_event(WorkerConnection, new_event) + + self.mainloop.queue_event(self, _BuildFailed()) + + self.mainloop.queue_event(WorkerConnection, _JobFinished(self._job)) diff --git a/doc/branching-merging-systems.mdwn b/doc/branching-merging-systems.mdwn new file mode 100644 index 00000000..3bc19aab --- /dev/null +++ b/doc/branching-merging-systems.mdwn @@ -0,0 +1,316 @@ +Branching and merging at the system level in Baserock +===================================================== + +NOTE: This is a spec. The code does not yet match it. + +As I write this, Baserock consists of just under 70 upstream projects, +each of which we keep in their own git repository. We need a way to +manage changes to them in a sensible manner, particularly when things +touch more than one repository. What we need is a way to do branch +and merge the whole system, across all our git repositories, with +similar ease and efficiency as what git provides for an individual +project. Where possible we need to allow the use of raw git so that +we do not constrain our developers unnecessarily. + +There are other things we will want to do across all the Baserock git +repositories, but that is outside the scope of this document, and will +be dealt with later. + +A couple of use cases: + +* I have a problem on a particular device, and want to make changes to + analyze and fix it. I need to branch the specific version of everything + that was using the system image version that the device was running. + I then want to be able to make changes to selected components and build + the system with those changes. Everything I do not explicitly touch should + stay at the same version. +* I am developing Baserock and I want to fix something, or add a new + feature, or other such change. I want to take the current newest + version of everything (the mainline development branch, whatever it + might be named), and make changes to some components, and build and + test those changes. While I'm doing that, I don't want to get random + other changes by other people, except when I explicitly ask for them + (e.g. "git pull" on an individual repository.), to avoid unnecessary + conflicts and building changes that don't affect my changes. + +In both users cases, when I'm done, I want to get my changes into the +relevant branches. This might happen by merging my changes directly, +by generating a pull request on a git server, or by generating a patch +series for each affected repository to be mailed to people who can do +the merging. + +Overview +-------- + +We want a clear, convenient, and efficient way of working with multiple +repositories and multiple projects at the same time. To manage this, +we introduce the following concepts (FIXME: naming needs attention): + +* **git repository** is exactly the same as usually with git, as are + all other concepts related to git +* **system branch** is a collection of branches in individual git + repositories that together form a particular line of development of + the whole system; in other words, given all the git repositories + that are part of Baserock, system branch `foo` consists of branch + `foo` in each git repository that has a branch with that name +* **system branch directory** contains git repositories relevant to + a system branch; it need not contain all the repositories, just the + ones that are being worked on by the user, or that the user for + other reasons have checked out +* **morph workspace** is where all Morph keeps global + state and shared caches, so that creating a system branch directory + is a fairly cheap operation; all the system branch directories are + inside the morph workspace directory + +As a picture: + + /home/liw/ -- user's home directory + baserock/ -- morph workspace + .morph/ -- morph shared state, cache, etc + unstable/ -- system branch directory: mainline devel + morphs/ -- git repository for system, stratum morphs + magnetic-frobbles/ -- system branch directory: new feature + morphs/ -- system branch specific changes to morphs + linux/ -- ditto for linux + +To use the system branching and merging, you do the following (which we'll +cover in more detail later): + +1. Initialize the morph workspace. This creates the `.morph` directory and + populates it with some initial stuff. You can have as many workspaces as + you want, but you don't have to have more than one, and you only + need to initialize it once. +2. Branch the system from some version of it (e.g., `master`) to work + on a new feature or bug fix. + This creates a system branch directory under the workspace directory. + The system branch directory initially contains a clone of the `morphs` + git repository, with some changes specific to this system branch. + (See petrification, below.) +3. Edit one or more components (chunks) in the project. This typically + requires adding more clones of git repositories inside the system + branch directory. +4. Build, test, and fix, repeating as necessary. This requires using + git directly (not via morph) in the git repositories inside the + system branch directory. +5. Merge the changes to relevant target branches. Depending on what the + change was, it may be necessary ot merge into many branches, e.g., + for each stable release branch. + +Walkthrough +----------- + +Let's walk through what happens, making things more concrete. This is +still fairly high level, and implementation details will come later. + + morph init ~/baserock + +This creates the `~/baserock` directory if it does not exist, and then +initializes it as a "morph workspace" directory, by creating a `.morph` +subdirectory. `.morph` will contain the Morph cache directory, and +other shared state between the various branches. As part of the cache, +any git repositories that Morph clones get put into the cache first, +and cloned into the system branch directories from there (probably +using hard-linking for speed), so that if there's a need for another +clone of the repository, it does not need to be cloned from a server +a second time. + + cd ~/baserock + morph branch liw/foo + morph branch liw/foo baserock/stable-1.0 + morph branch liw/foo --branch-off-system=/home/liw/system.img + +Create a new system branch, and the corresponding system branch +directory. The three versions shown differ in the starting point +of the branch: the first one uses the `master` branch in `morphs`, +the second one uses the named branch instead, and the third one +gets the SHA-1s from a system image file. + +Also, clone the `morphs` git repository inside the system branch +directory. + + cd ~/baserock/liw/foo/morphs + edit base-system.morph devel-system.morph + git commit -a + +Modify the specified morphologies (or the stratum morphologies they +refer to) to nail down the references to chunk repositories to use SHA-1s +instead of branch names or whatever. The changes need to be committed +to git manually, so that the user has a chance to give a good commit +message. + +Petrification is useful to prevent the set of changes including changes +by other team members. When a chunk is edited it will be made to refer +to that ref instead of the SHA-1 that it is petrified to. + +Petrification can be done by resolving the chunk references against +the current state of the git repositories, or it can be done by getting +the SHA-1s directly from a system image, or a data file. + + cd ~/baserock/liw/foo + morph edit linux + +Tell Morph that you want to edit a particular component (chunk). +This will clone the repository into the system branch directory, +at the point in history indicated by the morphologies in the +local version of `morphs`. + + cd ~/baserock/liw/foo + morph git -- log -p master..HEAD + +This allows running a git command in each git repository in a +system branch directory. Morph may offer short forms ("morph status") +as well, for things that are needed very commonly. + + cd ~/baserock/baserock/mainline + morph merge liw/foo + +This merges the changes made in the `liw/foo` branch into the +`baserock/mainline` branch. The petrification changes are automatically +undone, since they're not going to be wanted in the merge. + + cd ~/baserock + morph mass-merge liw/foo baserock/stable* + +Do the merge from `liw/foo` to every branch matching `baserock/stable*` +(as expanded by the shell). This is a wrapper around the simpler +`morph merge` command to make it easier to push a change into many +branches (e.g., a security fix to all stable branches). + + +Implementation: `morph init` +-------------- + +Usage: + + morph init [DIR] + +DIR defaults to the current working directory. If DIR is given, +but does not exist, it is created. + +* Create `DIR/.morph`. + + +Implementation: `morph branch` +-------------- + +Usage: + + morph branch BRANCH [COMMIT] + +This needs to be run in the morph workspace directory (the one initialized +with `morph init`). + +* If `./BRANCH` as a directory exists, abort. +* Create `./BRANCH` directory. +* Clone the `morphs` repository to `BRANCH/morphs`. +* Create a new branch called `BRANCH` in morphs, based either the tip of + `master` or from `COMMIT` if given. Store the SHA-1 of the branch origin + in some way so we get at it later. + + +Implementation: `morph checkout` +-------------- + +Usage: + + morph checkout BRANCH + +This needs to be run in the morph workspace directory. It works like +`morph branch`, except it does not create the new branch and requires +it to exist instead. + +* If `./BRANCH` as a directory exists, abort. +* Create `./BRANCH` directory. +* Clone the `morphs` repository to `BRANCH/morphs`. +* Run `git checkout BRANCH` in the `morphs` repository. + + +Implementation: `morph edit` +-------------- + +Usage: + + morph edit REPO MORPH... + +where `REPO` is a chunk repository (absolute URL or one relative to one of +the `git-base-url` values). The command must be run in the `morphs` +directory of the system branch. + +* `git clone REPOURL` where the URL is constructed with `git-base-url` + if necessary. +* `git branch BRANCH REF` where `BRANCH` is the branch name given to + `morph branch` and `REF` is the reference to the chunk we want to edit, + as specified in morphologies. +* Modify the affected morphologies to refer to the repository using + the `BRANCH` name, and commit those changes. + +If the specified morphology is not a stratum morphology (that is, it is +a system one), we check all the stratum morphologies mentioned and find +the one that refers to the specified repository. + +Multiple morphologies can be specified. They must have the same original +reference to the repository. However, they will all be modified. + + +Implementation: `morph git` +-------------- + +Usage: + + morph git -- log -p master..HEAD + +This is to be run in the morph workspace. It runs git with the arguments on +the command line in each local git repository in the workspace. (The `--` is +only necessary if the git arguments are to contain options.) + + +Implementation: `morph merge` +-------------- + +Usage: + + morph merge BRANCH + +This needs to be run inside a system branch directory's `morphs` +repository, and `BRANCH` must be another system branch checked out +in the morph workspace. + +* In each git repository modified by the `BRANCH` system branch, + run `git merge --no-commit BRANCH`, then undo any changes to + stratum morphologies made by `morph edit`, and finally commit + the changes. + + +Implementation: `morph mass-merge` +-------------- + +Usage: + + morph mass-merge BRANCH [TARGET]... + +To be run in the morph workspace directory. + +This just runs `morph merge BRANCH` in each `TARGET` system branch. + + +Implementation: `morph cherry-pick` +-------------- + +Usage: + + morph cherry-pick BRANCH [COMMIT]... + morph cherry-pick BRANCH --since-branch-point + +To be run in the system branch directory. + +In the first form: + +* For each git repository modified by the `BRANCH` system branch, + run `git cherry-pick COMMIT` for each `COMMIT`. + +In the second form: + +* For each git repository modified by the `BRANCH` system branch, + run `git cherry-pick` giving it a list of all commits made after + the system branch was created. + @@ -0,0 +1,21 @@ +#!/usr/bin/python +# +# Copyright (C) 2011-2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import morphlib + +morphlib.app.Morph(version=morphlib.__version__).run() diff --git a/morph.1.in b/morph.1.in new file mode 100644 index 00000000..232ae396 --- /dev/null +++ b/morph.1.in @@ -0,0 +1,117 @@ +.\" Copyright (C) 2012 Codethink Limited +.\" +.\" This program is free software; you can redistribute it and/or modify +.\" it under the terms of the GNU General Public License as published by +.\" the Free Software Foundation; version 2 of the License. +.\" +.\" This program is distributed in the hope that it will be useful, +.\" but WITHOUT ANY WARRANTY; without even the implied warranty of +.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +.\" GNU General Public License for more details. +.\" +.\" You should have received a copy of the GNU General Public License along +.\" with this program; if not, write to the Free Software Foundation, Inc., +.\" 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +.\" +.TH MORPH 1 +.SH NAME +morph \- Baserock development workflow tool +.SH SYNOPSIS +.SH DESCRIPTION +Baserock is an embedded Linux system. +.B morph +is its workflow tool. +It manages building binaries, +and branching and merging of the entire system. +Morph is designed to turn collections of git repositories into system images +using morphology files to define their dependencies. +.PP +A +.B system +image is defined as a group of +.B strata +describing subsystems, +each of which comprises a series of +.BR chunks , +each of which in turn corresponds +to an individual upstream project. For example, there might be a 'generic +developer system' system morphology, containing a stratum for the basic +bootable system and another for developer tools; the latter would then have +individual chunks for make, gcc, binutils and so forth. +.PP +A chunk is a git repository based on an individual upstream project's revision +control system, converted into git if upstream does not already use it. +The build is controlled by a +.B something.morph +configuration file +defining how to build the chunk and any other changes required to get the +repository to build with the rest of Baserock. +.PP +Morph is also capable of branching the whole system (that is branching +all constituent git repositories of a system simultaneously) in order +to allow system-wide changes that cross the boundaries of individual +git repositories, and of generating commits to the group of git +repositories that have been modified in such a branch. +.PP +For more details, please see the Baserock wiki at http://wiki.baserock.org. +.SH OPTIONS +.SH ENVIRONMENT +.B morph +cleans out the environment when it runs builds, +so that builds are not affected by random enviroment variables set by the user. +However, a few environment variables do affect either +.B morph +itself, or the builds it runs. +.PP +.TP +.B PATH +.B morph +supports building chunks in +.B bootstrap +mode, which exposes the host's tools for building rather than using a +controlled chroot. The +.B PATH +variable is significant for chunks built in this mode. +.TP +.BR DISTCC_HOSTS ", " TMPDIR ", " LD_PRELOAD ", " LD_LIBRARY_PATH ", " \ +FAKEROOTKEY ", " FAKED_MODE ", " FAKEROOT_FD_BASE +.B morph +keeps these environment variable, if set. +.TP +.B MORPH_ARCH +The system morphology defines the architecture it should be built for, and +.B morph +sets this variable in the build environment accordingly. Only a small set of +predefined values can be used, and it is expected that morphologies can change +the configuration of the chunk they are building based on this value. +.TP +.B TARGET +This value is set to the GNU machine triplet for the machine +.B MORPH_ARCH +defines. +.TP +.B TARGET_STAGE1 +The same as +.B TARGET +but with the vendor field replaced with +.BR bootstrap +.TP +.B MORPH_PLUGIN_PATH +.B morph +looks for additional plugins in the directories given in this variable. +Syntax is same as for +.B PATH +(i.e., colon delimited pathnames). +.PP +The +.BR cliapp (5) +manual page has some more variables that affect +.B morph +itself. +.SH "SEE ALSO" +.BR cliapp (5). +.PP +http://wiki.baserock.org/ +.br +http://www.baserock.com/ + diff --git a/morphlib/__init__.py b/morphlib/__init__.py new file mode 100644 index 00000000..f98c11aa --- /dev/null +++ b/morphlib/__init__.py @@ -0,0 +1,93 @@ +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''Baserock library.''' + + +# Import yaml if available. This can go away once Baserock has made a +# release that includes yaml (also in its staging filler). +try: + import yaml +except ImportError: + got_yaml = False + class YAMLError(Exception): + pass +else: + got_yaml = True + YAMLError = yaml.YAMLError + + +import cliapp + +import gitversion + +__version__ = gitversion.version + + +# List of architectures that Morph supports +valid_archs = ['armv7l', 'armv7lhf', 'armv7b', 'testarch', + 'x86_32', 'x86_64', 'ppc64'] + +class Error(cliapp.AppException): + + '''Base for all morph exceptions that cause user-visible messages.''' + + +import artifact +import artifactcachereference +import artifactresolver +import artifactsplitrule +import branchmanager +import bins +import buildbranch +import buildcommand +import buildenvironment +import buildsystem +import builder2 +import cachedrepo +import cachekeycomputer +import extensions +import extractedtarball +import fsutils +import git +import gitdir +import gitindex +import localartifactcache +import localrepocache +import mountableimage +import morphologyfactory +import morphologyfinder +import morphology +import morphloader +import morphset +import remoteartifactcache +import remoterepocache +import repoaliasresolver +import savefile +import source +import sourcepool +import stagingarea +import stopwatch +import sysbranchdir +import systemmetadatadir +import util +import workspace + +import yamlparse + +import writeexts + +import app # this needs to be last diff --git a/morphlib/app.py b/morphlib/app.py new file mode 100644 index 00000000..9ab102b3 --- /dev/null +++ b/morphlib/app.py @@ -0,0 +1,563 @@ +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import collections +import logging +import os +import sys +import time +import urlparse +import warnings +import extensions + +import morphlib + +class InvalidUrlError(cliapp.AppException): + + def __init__(self, parameter, url): + cliapp.AppException.__init__( + self, 'Value %s for argument %s is not a url' % + (url, parameter)) + +defaults = { + 'trove-host': 'git.baserock.org', + 'trove-id': [], + 'repo-alias': [ + ('freedesktop=' + 'git://anongit.freedesktop.org/#' + 'ssh://git.freedesktop.org/'), + ('gnome=' + 'git://git.gnome.org/%s#' + 'ssh://git.gnome.org/git/%s'), + ('github=' + 'git://github.com/%s#' + 'ssh://git@github.com/%s'), + ], + 'cachedir': os.path.expanduser('~/.cache/morph'), + 'max-jobs': morphlib.util.make_concurrency() +} + + +class Morph(cliapp.Application): + + def add_settings(self): + self.settings.boolean(['verbose', 'v'], + 'show what is happening in much detail') + self.settings.boolean(['quiet', 'q'], + 'show no output unless there is an error') + + self.settings.boolean(['help', 'h'], + 'show this help message and exit') + self.settings.boolean(['help-all'], + 'show help message including hidden subcommands') + + self.settings.string(['build-ref-prefix'], + 'Prefix to use for temporary build refs', + metavar='PREFIX', + default=None) + self.settings.string(['trove-host'], + 'hostname of Trove instance', + metavar='TROVEHOST', + default=defaults['trove-host']) + self.settings.string_list(['trove-id', 'trove-prefix'], + 'list of URL prefixes that should be ' + 'resolved to Trove', + metavar='PREFIX, ...', + default=defaults['trove-id']) + + group_advanced = 'Advanced Options' + self.settings.boolean(['no-git-update'], + 'do not update the cached git repositories ' + 'automatically', + group=group_advanced) + self.settings.boolean(['build-log-on-stdout'], + 'write build log on stdout', + group=group_advanced) + self.settings.string_list(['repo-alias'], + 'list of URL prefix definitions, in the ' + 'form: example=git://git.example.com/%s' + '#git@git.example.com/%s', + metavar='ALIAS=PREFIX#PULL#PUSH', + default=defaults['repo-alias'], + group=group_advanced) + self.settings.string(['cache-server'], + 'HTTP URL of the morph cache server to use. ' + 'If not provided, defaults to ' + 'http://TROVEHOST:8080/', + metavar='URL', + default=None, + group=group_advanced) + self.settings.string( + ['artifact-cache-server'], + 'HTTP URL for the artifact cache server; ' + 'if not set, then the cache-server setting is used instead', + metavar='URL', + default=None, + group=group_advanced) + self.settings.string( + ['git-resolve-cache-server'], + 'HTTP URL for the git ref resolving cache server; ' + 'if not set, then the cache-server setting is used instead', + metavar='URL', + default=None, + group=group_advanced) + self.settings.string(['tarball-server'], + 'base URL to download tarballs. ' + 'If not provided, defaults to ' + 'http://TROVEHOST/tarballs/', + metavar='URL', + default=None, + group=group_advanced) + + group_build = 'Build Options' + self.settings.integer(['max-jobs'], + 'run at most N parallel jobs with make (default ' + 'is to a value based on the number of CPUs ' + 'in the machine running morph', + metavar='N', + default=defaults['max-jobs'], + group=group_build) + self.settings.boolean(['no-ccache'], 'do not use ccache', + group=group_build) + self.settings.boolean(['no-distcc'], + 'do not use distcc (default: true)', + group=group_build, default=True) + self.settings.boolean(['push-build-branches'], + 'always push temporary build branches to the ' + 'remote repository', + group=group_build) + + group_storage = 'Storage Options' + self.settings.string(['tempdir'], + 'temporary directory to use for builds ' + '(this is separate from just setting $TMPDIR ' + 'or /tmp because those are used internally ' + 'by things that cannot be on NFS, but ' + 'this setting can point at a directory in ' + 'NFS)', + metavar='DIR', + default=None, + group=group_storage) + self.settings.string(['cachedir'], + 'cache git repositories and build results in DIR', + metavar='DIR', + group=group_storage, + default=defaults['cachedir']) + self.settings.string(['compiler-cache-dir'], + 'cache compiled objects in DIR/REPO. If not ' + 'provided, defaults to CACHEDIR/ccache/', + metavar='DIR', + group=group_storage, + default=None) + # The tempdir default size of 4G comes from the staging area needing to + # be the size of the largest known system, plus the largest repository, + # plus the largest working directory. + # The largest system is 2G, linux is the largest git repository at + # 700M, the checkout of this is 600M. This is rounded up to 4G because + # there are likely to be file-system overheads. + self.settings.bytesize(['tempdir-min-space'], + 'Immediately fail to build if the directory ' + 'specified by tempdir has less space remaining ' + 'than SIZE bytes (default: %default)', + metavar='SIZE', + group=group_storage, + default='4G') + # The cachedir default size of 4G comes from twice the size of the + # largest system artifact. + # It's twice the size because it needs space for all the chunks that + # make up the system artifact as well. + # The git cache and ccache are also kept in cachedir, but it's hard to + # estimate size needed for the git cache, and it tends to not grow + # too quickly once everything is checked out. + # ccache is self-managing so does not need much extra attention + self.settings.bytesize(['cachedir-min-space'], + 'Immediately fail to build if the directory ' + 'specified by cachedir has less space ' + 'remaining than SIZE bytes (default: %default)', + metavar='SIZE', + group=group_storage, + default='4G') + + def check_time(self): + # Check that the current time is not far in the past. + if time.localtime(time.time()).tm_year < 2012: + raise morphlib.Error( + 'System time is far in the past, please set your system clock') + + def setup(self): + self.status_prefix = '' + + self.add_subcommand('help-extensions', self.help_extensions) + + def log_config(self): + with morphlib.util.hide_password_environment_variables(os.environ): + cliapp.Application.log_config(self) + + def process_args(self, args): + self.check_time() + + if self.settings['help']: + self.help(args) + sys.exit(0) + + if self.settings['help-all']: + self.help_all(args) + sys.exit(0) + + if self.settings['build-ref-prefix'] is None: + if self.settings['trove-id']: + self.settings['build-ref-prefix'] = os.path.join( + self.settings['trove-id'][0], 'builds') + else: + self.settings['build-ref-prefix'] = "baserock/builds" + + # Combine the aliases into repo-alias before passing on to normal + # command processing. This means everything from here on down can + # treat settings['repo-alias'] as the sole source of prefixes for git + # URL expansion. + self.settings['repo-alias'] = morphlib.util.combine_aliases(self) + if self.settings['cache-server'] is None: + self.settings['cache-server'] = 'http://%s:8080/' % ( + self.settings['trove-host']) + if self.settings['tarball-server'] is None: + self.settings['tarball-server'] = 'http://%s/tarballs/' % ( + self.settings['trove-host']) + if self.settings['compiler-cache-dir'] is None: + self.settings['compiler-cache-dir'] = os.path.join( + self.settings['cachedir'], 'ccache') + if self.settings['tempdir'] is None: + tmpdir_base = os.environ.get('TMPDIR', '/tmp') + tmpdir = os.path.join(tmpdir_base, 'morph_tmp') + self.settings['tempdir'] = tmpdir + + if self.settings['tarball-server']: + url_split = urlparse.urlparse(self.settings['tarball-server']) + if not (url_split.netloc and + url_split.scheme in ('http', 'https', 'file')): + raise InvalidUrlError('tarball-server', + self.settings['tarball-server']) + + if 'MORPH_DUMP_PROCESSED_CONFIG' in os.environ: + self.settings.dump_config(sys.stdout) + sys.exit(0) + + tmpdir = self.settings['tempdir'] + for required_dir in (os.path.join(tmpdir, 'chunks'), + os.path.join(tmpdir, 'staging'), + os.path.join(tmpdir, 'failed'), + os.path.join(tmpdir, 'deployments'), + self.settings['cachedir']): + if not os.path.exists(required_dir): + os.makedirs(required_dir) + + cliapp.Application.process_args(self, args) + + def setup_plugin_manager(self): + cliapp.Application.setup_plugin_manager(self) + + self.pluginmgr.locations += os.path.join( + os.path.dirname(morphlib.__file__), 'plugins') + + s = os.environ.get('MORPH_PLUGIN_PATH', '') + self.pluginmgr.locations += s.split(':') + + self.hookmgr = cliapp.HookManager() + self.hookmgr.new('new-build-command', cliapp.FilterHook()) + + def itertriplets(self, args): + '''Generate repo, ref, filename triples from args.''' + + if (len(args) % 3) != 0: + raise cliapp.AppException('Argument list must have full triplets') + + while args: + assert len(args) >= 2, args + yield (args[0], args[1], + morphlib.util.sanitise_morphology_path(args[2])) + args = args[3:] + + def create_source_pool(self, lrc, rrc, repo, ref, filename): + pool = morphlib.sourcepool.SourcePool() + + def add_to_pool(reponame, ref, filename, absref, tree, morphology): + sources = morphlib.source.make_sources(reponame, ref, + filename, absref, + tree, morphology) + for source in sources: + pool.add(source) + + self.traverse_morphs(repo, ref, [filename], lrc, rrc, + update=not self.settings['no-git-update'], + visit=add_to_pool) + return pool + + def resolve_ref(self, lrc, rrc, reponame, ref, update=True): + '''Resolves commit and tree sha1s of the ref in a repo and returns it. + + If update is True then this has the side-effect of updating + or cloning the repository into the local repo cache. + ''' + absref = None + + if lrc.has_repo(reponame): + repo = lrc.get_repo(reponame) + if update and repo.requires_update_for_ref(ref): + self.status(msg='Updating cached git repository %(reponame)s ' + 'for ref %(ref)s', reponame=reponame, ref=ref) + repo.update() + # If the user passed --no-git-update, and the ref is a SHA1 not + # available locally, this call will raise an exception. + absref, tree = repo.resolve_ref(ref) + elif rrc is not None: + try: + absref, tree = rrc.resolve_ref(reponame, ref) + if absref is not None: + self.status(msg='Resolved %(reponame)s %(ref)s via remote ' + 'repo cache', + reponame=reponame, + ref=ref, + chatty=True) + except BaseException, e: + logging.warning('Caught (and ignored) exception: %s' % str(e)) + if absref is None: + if update: + self.status(msg='Caching git repository %(reponame)s', + reponame=reponame) + repo = lrc.cache_repo(reponame) + repo.update() + else: + repo = lrc.get_repo(reponame) + absref, tree = repo.resolve_ref(ref) + return absref, tree + + def traverse_morphs(self, definitions_repo, definitions_ref, + system_filenames, lrc, rrc, update=True, + visit=lambda rn, rf, fn, arf, m: None): + morph_factory = morphlib.morphologyfactory.MorphologyFactory(lrc, rrc, + self) + definitions_queue = collections.deque(system_filenames) + chunk_in_definitions_repo_queue = [] + chunk_in_source_repo_queue = [] + resolved_refs = {} + resolved_morphologies = {} + + # Resolve the (repo, ref) pair for the definitions repo, cache result. + definitions_absref, definitions_tree = self.resolve_ref( + lrc, rrc, definitions_repo, definitions_ref, update) + + while definitions_queue: + filename = definitions_queue.popleft() + + key = (definitions_repo, definitions_absref, filename) + if not key in resolved_morphologies: + resolved_morphologies[key] = morph_factory.get_morphology(*key) + morphology = resolved_morphologies[key] + + visit(definitions_repo, definitions_ref, filename, + definitions_absref, definitions_tree, morphology) + if morphology['kind'] == 'cluster': + raise cliapp.AppException( + "Cannot build a morphology of type 'cluster'.") + elif morphology['kind'] == 'system': + definitions_queue.extend( + morphlib.util.sanitise_morphology_path(s['morph']) + for s in morphology['strata']) + elif morphology['kind'] == 'stratum': + if morphology['build-depends']: + definitions_queue.extend( + morphlib.util.sanitise_morphology_path(s['morph']) + for s in morphology['build-depends']) + for c in morphology['chunks']: + if 'morph' not in c: + path = morphlib.util.sanitise_morphology_path( + c.get('morph', c['name'])) + chunk_in_source_repo_queue.append( + (c['repo'], c['ref'], path)) + continue + chunk_in_definitions_repo_queue.append( + (c['repo'], c['ref'], c['morph'])) + + for repo, ref, filename in chunk_in_definitions_repo_queue: + if (repo, ref) not in resolved_refs: + resolved_refs[repo, ref] = self.resolve_ref( + lrc, rrc, repo, ref, update) + absref, tree = resolved_refs[repo, ref] + key = (definitions_repo, definitions_absref, filename) + if not key in resolved_morphologies: + resolved_morphologies[key] = morph_factory.get_morphology(*key) + morphology = resolved_morphologies[key] + visit(repo, ref, filename, absref, tree, morphology) + + for repo, ref, filename in chunk_in_source_repo_queue: + if (repo, ref) not in resolved_refs: + resolved_refs[repo, ref] = self.resolve_ref( + lrc, rrc, repo, ref, update) + absref, tree = resolved_refs[repo, ref] + key = (repo, absref, filename) + if key not in resolved_morphologies: + resolved_morphologies[key] = morph_factory.get_morphology(*key) + morphology = resolved_morphologies[key] + visit(repo, ref, filename, absref, tree, morphology) + + def cache_repo_and_submodules(self, cache, url, ref, done): + subs_to_process = set() + subs_to_process.add((url, ref)) + while subs_to_process: + url, ref = subs_to_process.pop() + done.add((url, ref)) + cached_repo = cache.cache_repo(url) + cached_repo.update() + + try: + submodules = morphlib.git.Submodules(self, cached_repo.path, + ref) + submodules.load() + except morphlib.git.NoModulesFileError: + pass + else: + for submod in submodules: + if (submod.url, submod.commit) not in done: + subs_to_process.add((submod.url, submod.commit)) + + def status(self, **kwargs): + '''Show user a status update. + + The keyword arguments are formatted and presented to the user in + a pleasing manner. Some keywords are special: + + * ``msg`` is the message text; it can use ``%(foo)s`` to embed the + value of keyword argument ``foo`` + * ``chatty`` should be true when the message is only informative, + and only useful for users who want to know everything (--verbose) + * ``error`` should be true when it is an error message + + All other keywords are ignored unless embedded in ``msg``. + + The ``self.status_prefix`` string is prepended to the output. + It is set to the empty string by default. + + ''' + + assert 'msg' in kwargs + text = self.status_prefix + (kwargs['msg'] % kwargs) + + error = kwargs.get('error', False) + chatty = kwargs.get('chatty', False) + quiet = self.settings['quiet'] + verbose = self.settings['verbose'] + + if error: + logging.error(text) + elif chatty: + logging.debug(text) + else: + logging.info(text) + + ok = verbose or error or (not quiet and not chatty) + if ok: + timestamp = time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime()) + self.output.write('%s %s\n' % (timestamp, text)) + self.output.flush() + + def runcmd(self, argv, *args, **kwargs): + if 'env' not in kwargs: + kwargs['env'] = dict(os.environ) + + if 'print_command' in kwargs: + print_command = kwargs['print_command'] + del kwargs['print_command'] + else: + print_command = True + + # convert the command line arguments into a string + commands = [argv] + list(args) + for command in commands: + if isinstance(command, list): + for i in xrange(0, len(command)): + command[i] = str(command[i]) + commands = [' '.join(command) for command in commands] + + # print the command line + if print_command: + self.status(msg='# %(cmdline)s', + cmdline=' | '.join(commands), + chatty=True) + + # Log the environment. + prev = getattr(self, 'prev_env', {}) + morphlib.util.log_environment_changes(self, kwargs['env'], prev) + self.prev_env = kwargs['env'] + + # run the command line + return cliapp.Application.runcmd(self, argv, *args, **kwargs) + + def parse_args(self, args, configs_only=False): + return self.settings.parse_args(args, + configs_only=configs_only, + arg_synopsis=self.arg_synopsis, + cmd_synopsis=self.cmd_synopsis, + compute_setting_values=self.compute_setting_values, + add_help_option=False) + + def _help(self, show_all): + pp = self.settings.build_parser( + configs_only=True, + arg_synopsis=self.arg_synopsis, + cmd_synopsis=self.cmd_synopsis, + all_options=show_all, + add_help_option=False) + text = pp.format_help() + self.output.write(text) + + def _help_topic(self, topic): + if topic in self.subcommands: + usage = self._format_usage_for(topic) + description = self._format_subcommand_help(topic) + text = '%s\n\n%s' % (usage, description) + self.output.write(text) + elif topic in extensions.list_extensions(): + name, kind = os.path.splitext(topic) + try: + with extensions.get_extension_filename( + name, + kind + '.help', executable=False) as fname: + with open(fname, 'r') as f: + help_data = morphlib.yamlparse.load(f.read()) + print help_data['help'] + except extensions.ExtensionError: + raise cliapp.AppException( + 'Help not available for extension %s' % topic) + else: + raise cliapp.AppException( + 'Unknown subcommand or extension %s' % topic) + + def help(self, args): # pragma: no cover + '''Print help.''' + if args: + self._help_topic(args[0]) + else: + self._help(False) + + def help_all(self, args): # pragma: no cover + '''Print help, including hidden subcommands.''' + self._help(True) + + def help_extensions(self, args): + exts = extensions.list_extensions(self.settings['build-ref-prefix']) + template = "Extensions:\n %s\n" + ext_string = '\n '.join(exts) + self.output.write(template % (ext_string)) diff --git a/morphlib/artifact.py b/morphlib/artifact.py new file mode 100644 index 00000000..8b4ce65e --- /dev/null +++ b/morphlib/artifact.py @@ -0,0 +1,68 @@ +# Copyright (C) 2012, 2013, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +class Artifact(object): + + '''Represent a build result generated from a source. + + Has the following properties: + + * ``source`` -- the source from which the artifact is built + * ``name`` -- the name of the artifact + * ``dependents`` -- list of Sources that need this Artifact to be built + + The ``dependencies`` and ``dependents`` lists MUST be modified by + the ``add_dependencies`` and ``add_dependent`` methods only. + + ''' + + def __init__(self, source, name): + self.source = source + self.name = name + self.dependents = [] + + def basename(self): # pragma: no cover + return '%s.%s' % (self.source.basename(), str(self.name)) + + def metadata_basename(self, metadata_name): # pragma: no cover + return '%s.%s' % (self.basename(), metadata_name) + + def __str__(self): # pragma: no cover + return '%s|%s' % (self.source, self.name) + + def __repr__(self): # pragma: no cover + return 'Artifact(%s)' % str(self) + + + def walk(self): # pragma: no cover + '''Return list of an artifact and its build dependencies. + + The artifacts are returned in depth-first order: an artifact + is returned only after all of its dependencies. + + ''' + + done = set() + + def depth_first(a): + if a not in done: + done.add(a) + for dep in a.source.dependencies: + for ret in depth_first(dep): + yield ret + yield a + + return list(depth_first(self)) diff --git a/morphlib/artifact_tests.py b/morphlib/artifact_tests.py new file mode 100644 index 00000000..abd8767e --- /dev/null +++ b/morphlib/artifact_tests.py @@ -0,0 +1,60 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import copy +import unittest + +import morphlib + + +class ArtifactTests(unittest.TestCase): + + def setUp(self): + loader = morphlib.morphloader.MorphologyLoader() + morph = loader.load_from_string( + ''' + name: chunk + kind: chunk + products: + - artifact: chunk-runtime + include: + - usr/bin + - usr/sbin + - usr/lib + - usr/libexec + - artifact: chunk-devel + include: + - usr/include + ''') + self.source, = morphlib.source.make_sources('repo', 'ref', + 'chunk.morph', 'sha1', + 'tree', morph) + self.artifact_name = 'chunk-runtime' + self.artifact = self.source.artifacts[self.artifact_name] + self.other_source, = morphlib.source.make_sources('repo', 'ref', + 'chunk.morph', + 'sha1', 'tree', + morph) + self.other = self.other_source.artifacts[self.artifact_name] + + def test_constructor_sets_source(self): + self.assertEqual(self.artifact.source, self.source) + + def test_constructor_sets_name(self): + self.assertEqual(self.artifact.name, self.artifact_name) + + def test_sets_dependents_to_empty(self): + self.assertEqual(self.artifact.dependents, []) diff --git a/morphlib/artifactcachereference.py b/morphlib/artifactcachereference.py new file mode 100644 index 00000000..8211f6b5 --- /dev/null +++ b/morphlib/artifactcachereference.py @@ -0,0 +1,38 @@ +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +class ArtifactCacheReference(object): + '''Represent the information needed to retrieve an artifact + + The artifact cache doesn't need to know the dependencies or the + morphology of an artifact, it just needs to know the basename + + The basename could be generated, from the name, cache_key and kind, + but if the algorithm changes then morph wouldn't be able to find + old artifacts with a saved ArtifactCacheReference. + + Conversely if it generated the basename then old strata wouldn't be + able to refer to new chunks, but strata change more often than the chunks. + ''' + + def __init__(self, basename): + self._basename = basename + + def basename(self): + return self._basename + + def metadata_basename(self, metadata_name): + return '%s.%s' % (self._basename, metadata_name) diff --git a/morphlib/artifactresolver.py b/morphlib/artifactresolver.py new file mode 100644 index 00000000..5deb25b7 --- /dev/null +++ b/morphlib/artifactresolver.py @@ -0,0 +1,243 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import collections +import logging + +import morphlib + + +class MutualDependencyError(cliapp.AppException): + + def __init__(self, a, b): + cliapp.AppException.__init__( + self, 'Cyclic dependency between %s and %s detected' % (a, b)) + + +class DependencyOrderError(cliapp.AppException): + + def __init__(self, stratum_source, chunk, dependency_name): + cliapp.AppException.__init__( + self, 'In stratum %s, chunk %s references its dependency %s ' + 'before it is defined' % + (stratum_source, chunk, dependency_name)) + + +class ArtifactResolver(object): + + '''Resolves sources into artifacts that would be build from the sources. + + This class takes a CacheKeyComputer and a SourcePool, analyses the + sources and their dependencies and creates a list of artifacts + (represented by Artifact objects) that are involved in building the + sources in the pool. + + ''' + + def __init__(self): + self._added_artifacts = None + self._source_pool = None + + def resolve_artifacts(self, source_pool): + self._source_pool = source_pool + self._added_artifacts = set() + + artifacts = self._resolve_artifacts_recursively() + # TODO perform cycle detection, e.g. based on: + # http://stackoverflow.com/questions/546655/finding-all-cycles-in-graph + return artifacts + + def _resolve_artifacts_recursively(self): + artifacts = [] + + queue = self._create_initial_queue() + while queue: + source = queue.popleft() + + if source.morphology['kind'] == 'system': # pragma: no cover + systems = [source.artifacts[name] + for name in source.split_rules.artifacts] + + for system in (s for s in systems + if s not in self._added_artifacts): + artifacts.append(system) + self._added_artifacts.add(system) + + resolved_artifacts = self._resolve_system_dependencies( + systems, source, queue) + + for artifact in resolved_artifacts: + if not artifact in self._added_artifacts: + artifacts.append(artifact) + self._added_artifacts.add(artifact) + elif source.morphology['kind'] == 'stratum': + # Iterate split_rules.artifacts, rather than + # artifacts.values() to preserve ordering + strata = [source.artifacts[name] + for name in source.split_rules.artifacts + if name in source.artifacts] + + # If we were not given systems, return the strata here, + # rather than have the systems return them. + if not any(s.morphology['kind'] == 'system' + for s in self._source_pool): + for stratum in (s for s in strata + if s not in self._added_artifacts): + artifacts.append(stratum) + self._added_artifacts.add(stratum) + + resolved_artifacts = self._resolve_stratum_dependencies( + strata, source, queue) + + for artifact in resolved_artifacts: + if not artifact in self._added_artifacts: + artifacts.append(artifact) + self._added_artifacts.add(artifact) + elif source.morphology['kind'] == 'chunk': + chunks = [source.artifacts[name] + for name in source.split_rules.artifacts] + # If we were only given chunks, return them here, rather than + # have the strata return them. + if not any(s.morphology['kind'] == 'stratum' + for s in self._source_pool): + for chunk in (c for c in chunks + if c not in self._added_artifacts): + artifacts.append(chunk) + self._added_artifacts.add(chunk) + + return artifacts + + def _create_initial_queue(self): + if all([x.morphology['kind'] == 'chunk' for x in self._source_pool]): + return collections.deque(self._source_pool) + else: + sources = [x for x in self._source_pool + if x.morphology['kind'] != 'chunk'] + return collections.deque(sources) + + def _resolve_system_dependencies(self, systems, + source, queue): # pragma: no cover + artifacts = [] + + for info in source.morphology['strata']: + for stratum_source in self._source_pool.lookup( + info.get('repo') or source.repo_name, + info.get('ref') or source.original_ref, + morphlib.util.sanitise_morphology_path(info['morph'])): + + stratum_morph_name = stratum_source.morphology['name'] + + matches, overlaps, unmatched = source.split_rules.partition( + ((stratum_morph_name, sta_name) for sta_name + in stratum_source.split_rules.artifacts)) + for system in systems: + for (stratum_name, sta_name) in matches[system.name]: + if sta_name in stratum_source.artifacts: + stratum_artifact = \ + stratum_source.artifacts[sta_name] + source.add_dependency(stratum_artifact) + artifacts.append(stratum_artifact) + + queue.append(stratum_source) + + return artifacts + + def _resolve_stratum_dependencies(self, strata, source, queue): + artifacts = [] + + stratum_build_depends = [] + + for stratum_info in source.morphology.get('build-depends') or []: + for other_source in self._source_pool.lookup( + stratum_info.get('repo') or source.repo_name, + stratum_info.get('ref') or source.original_ref, + morphlib.util.sanitise_morphology_path(stratum_info['morph'])): + + # Make every stratum artifact this stratum source produces + # depend on every stratum artifact the other stratum source + # produces. + for sta_name in other_source.split_rules.artifacts: + # Strata have split rules for artifacts they don't build, + # since they need to know to yield a match to its sibling + if sta_name not in other_source.artifacts: + continue + other_stratum = other_source.artifacts[sta_name] + + stratum_build_depends.append(other_stratum) + + artifacts.append(other_stratum) + + for stratum in strata: + if other_source.depends_on(stratum): + raise MutualDependencyError(stratum, other_stratum) + + source.add_dependency(other_stratum) + + queue.append(other_source) + + # 'name' here is the chunk artifact name + name_to_processed_artifacts = {} + + for info in source.morphology['chunks']: + filename = morphlib.util.sanitise_morphology_path( + info.get('morph', info['name'])) + chunk_source = self._source_pool.lookup( + info['repo'], + info['ref'], + filename)[0] + + chunk_name = chunk_source.name + + # Resolve now to avoid a search for the parent morphology later + chunk_source.build_mode = info['build-mode'] + chunk_source.prefix = info['prefix'] + + build_depends = info.get('build-depends', None) + + # Add our stratum's build depends as dependencies of this chunk + for other_stratum in stratum_build_depends: + chunk_source.add_dependency(other_stratum) + + # Add dependencies between chunks mentioned in this stratum + for name in build_depends: # pragma: no cover + if name not in name_to_processed_artifacts: + raise DependencyOrderError( + source, info['name'], name) + other_artifacts = name_to_processed_artifacts[name] + for other_artifact in other_artifacts: + chunk_source.add_dependency(other_artifact) + + # Add build dependencies between our stratum's artifacts + # and the chunk artifacts produced by this stratum. + matches, overlaps, unmatched = source.split_rules.partition( + ((chunk_name, ca_name) for ca_name + in chunk_source.split_rules.artifacts)) + for (chunk_name, ca_name) in matches[source.name]: + chunk_artifact = chunk_source.artifacts[ca_name] + source.add_dependency(chunk_artifact) + # Only return chunks required to build strata we need + if chunk_artifact not in artifacts: + artifacts.append(chunk_artifact) + + + # Add these chunks to the processed artifacts, so other + # chunks may refer to them. + name_to_processed_artifacts[info['name']] = \ + [chunk_source.artifacts[n] for n + in chunk_source.split_rules.artifacts] + + return artifacts diff --git a/morphlib/artifactresolver_tests.py b/morphlib/artifactresolver_tests.py new file mode 100644 index 00000000..89f30010 --- /dev/null +++ b/morphlib/artifactresolver_tests.py @@ -0,0 +1,329 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import itertools +import unittest +import yaml + +import morphlib + + +def get_chunk_morphology(name, artifact_names=[]): + assert(isinstance(artifact_names, list)) + + if artifact_names: + # fake a list of artifacts + artifacts = [] + for artifact_name in artifact_names: + artifacts.append({'artifact': artifact_name, + 'include': [artifact_name]}) + text = yaml.dump({"name": name, + "kind": "chunk", + "products": artifacts}, default_flow_style=False) + else: + text = yaml.dump({'name': name, + 'kind': 'chunk'}, default_flow_style=False) + + loader = morphlib.morphloader.MorphologyLoader() + morph = loader.load_from_string(text) + return morph + +def get_stratum_morphology(name, chunks=[], build_depends=[]): + assert(isinstance(chunks, list)) + assert(isinstance(build_depends, list)) + + chunks_list = [] + for source_name, morph, repo, ref in chunks: + chunks_list.append({ + 'name': source_name, + 'morph': morph, + 'repo': repo, + 'ref': ref, + 'build-depends': [], + }) + build_depends_list = [] + for morph in build_depends: + build_depends_list.append({ + 'morph': morph, + }) + if chunks_list: + text = yaml.dump({"name": name, + "kind": "stratum", + "build-depends": build_depends_list, + "chunks": chunks_list,}, default_flow_style=False) + else: + text = yaml.dump({"name": name, + "kind": "stratum", + "build-depends": build_depends_list}, + default_flow_style=False) + + loader = morphlib.morphloader.MorphologyLoader() + morph = loader.load_from_string(text) + return morph + + +class ArtifactResolverTests(unittest.TestCase): + + def setUp(self): + self.resolver = morphlib.artifactresolver.ArtifactResolver() + + def test_resolve_artifacts_using_an_empty_pool(self): + pool = morphlib.sourcepool.SourcePool() + artifacts = self.resolver.resolve_artifacts(pool) + self.assertEqual(len(artifacts), 0) + + def test_resolve_single_chunk_with_no_subartifacts(self): + pool = morphlib.sourcepool.SourcePool() + + morph = get_chunk_morphology('chunk') + sources = morphlib.source.make_sources('repo', 'ref', + 'chunk.morph', 'sha1', + 'tree', morph) + for source in sources: + pool.add(source) + + artifacts = self.resolver.resolve_artifacts(pool) + + self.assertEqual(len(artifacts), + sum(len(s.split_rules.artifacts) for s in pool)) + + for artifact in artifacts: + self.assertEqual(artifact.source, source) + self.assertTrue(artifact.name.startswith('chunk')) + self.assertEqual(source.dependencies, []) + self.assertEqual(artifact.dependents, []) + + def test_resolve_single_chunk_with_one_new_artifact(self): + pool = morphlib.sourcepool.SourcePool() + + morph = get_chunk_morphology('chunk', ['chunk-foobar']) + sources = morphlib.source.make_sources('repo', 'ref', + 'chunk.morph', 'sha1', + 'tree', morph) + for source in sources: + pool.add(source) + + artifacts = self.resolver.resolve_artifacts(pool) + + self.assertEqual(len(artifacts), + sum(len(s.split_rules.artifacts) for s in pool)) + + foobartifact, = (a for a in artifacts if a.name == 'chunk-foobar') + self.assertEqual(foobartifact.source, source) + self.assertEqual(foobartifact.source.dependencies, []) + self.assertEqual(foobartifact.dependents, []) + + def test_resolve_single_chunk_with_two_new_artifacts(self): + pool = morphlib.sourcepool.SourcePool() + + morph = get_chunk_morphology('chunk', ['chunk-baz', 'chunk-qux']) + sources = morphlib.source.make_sources('repo', 'ref', + 'chunk.morph', 'sha1', + 'tree', morph) + for source in sources: + pool.add(source) + + artifacts = self.resolver.resolve_artifacts(pool) + artifacts.sort(key=lambda a: a.name) + + self.assertEqual(len(artifacts), + sum(len(s.split_rules.artifacts) for s in pool)) + + for name in ('chunk-baz', 'chunk-qux'): + artifact, = (a for a in artifacts if a.name == name) + self.assertEqual(artifact.source, source) + self.assertEqual(artifact.source.dependencies, []) + self.assertEqual(artifact.dependents, []) + + def test_resolve_stratum_and_chunk(self): + pool = morphlib.sourcepool.SourcePool() + + morph = get_chunk_morphology('chunk') + sources = morphlib.source.make_sources('repo', 'ref', + 'chunk.morph', 'sha1', + 'tree', morph) + for chunk in sources: + pool.add(chunk) + + morph = get_stratum_morphology( + 'stratum', chunks=[('chunk', 'chunk', 'repo', 'ref')]) + stratum_sources = set(morphlib.source.make_sources('repo', 'ref', + 'stratum.morph', + 'sha1', 'tree', + morph)) + for stratum in stratum_sources: + pool.add(stratum) + + artifacts = self.resolver.resolve_artifacts(pool) + + all_artifacts = set() + for s in pool: all_artifacts.update(s.split_rules.artifacts) + + self.assertEqual(set(a.name for a in artifacts), all_artifacts) + self.assertEqual(len(artifacts), + len(all_artifacts)) + + + stratum_artifacts = set(a for a in artifacts + if a.source in stratum_sources) + chunk_artifacts = set(a for a in artifacts if a.source == chunk) + + for stratum_artifact in stratum_artifacts: + self.assertTrue(stratum_artifact.name.startswith('stratum')) + self.assertEqual(stratum_artifact.dependents, []) + self.assertTrue( + any(dep in chunk_artifacts + for dep in stratum_artifact.source.dependencies)) + + for chunk_artifact in chunk_artifacts: + self.assertTrue(chunk_artifact.name.startswith('chunk')) + self.assertEqual(chunk_artifact.source.dependencies, []) + self.assertTrue(any(dep in stratum_sources + for dep in chunk_artifact.dependents)) + + def test_resolve_stratum_and_chunk_with_two_new_artifacts(self): + pool = morphlib.sourcepool.SourcePool() + + morph = get_chunk_morphology('chunk', ['chunk-foo', 'chunk-bar']) + sources = morphlib.source.make_sources('repo', 'ref', + 'chunk.morph', 'sha1', + 'tree', morph) + for chunk in sources: + pool.add(chunk) + + morph = get_stratum_morphology( + 'stratum', + chunks=[ + ('chunk', 'chunk', 'repo', 'ref'), + ]) + stratum_sources = set(morphlib.source.make_sources('repo', 'ref', + 'stratum.morph', + 'sha1', 'tree', + morph)) + for stratum in stratum_sources: + pool.add(stratum) + + artifacts = self.resolver.resolve_artifacts(pool) + + self.assertEqual( + set(artifacts), + set(itertools.chain.from_iterable( + s.artifacts.itervalues() + for s in pool))) + + stratum_artifacts = set(a for a in artifacts + if a.source in stratum_sources) + chunk_artifacts = set(a for a in artifacts if a.source == chunk) + + for stratum_artifact in stratum_artifacts: + self.assertTrue(stratum_artifact.name.startswith('stratum')) + self.assertEqual(stratum_artifact.dependents, []) + self.assertTrue( + any(dep in chunk_artifacts + for dep in stratum_artifact.source.dependencies)) + + for chunk_artifact in chunk_artifacts: + self.assertTrue(chunk_artifact.name.startswith('chunk')) + self.assertEqual(chunk_artifact.source.dependencies, []) + self.assertTrue(any(dep in stratum_sources + for dep in chunk_artifact.dependents)) + + def test_detection_of_mutual_dependency_between_two_strata(self): + loader = morphlib.morphloader.MorphologyLoader() + pool = morphlib.sourcepool.SourcePool() + + chunk = get_chunk_morphology('chunk1') + chunk1, = morphlib.source.make_sources( + 'repo', 'original/ref', 'chunk1.morph', 'sha1', 'tree', chunk) + pool.add(chunk1) + + morph = get_stratum_morphology( + 'stratum1', + chunks=[(loader.save_to_string(chunk), 'chunk1.morph', + 'repo', 'original/ref')], + build_depends=['stratum2']) + sources = morphlib.source.make_sources('repo', 'original/ref', + 'stratum1.morph', 'sha1', + 'tree', morph) + for stratum1 in sources: + pool.add(stratum1) + + chunk = get_chunk_morphology('chunk2') + chunk2, = morphlib.source.make_sources( + 'repo', 'original/ref', 'chunk2.morph', 'sha1', 'tree', chunk) + pool.add(chunk2) + + morph = get_stratum_morphology( + 'stratum2', + chunks=[(loader.save_to_string(chunk), 'chunk2.morph', + 'repo', 'original/ref')], + build_depends=['stratum1']) + sources = morphlib.source.make_sources('repo', 'original/ref', + 'stratum2.morph', 'sha1', + 'tree', morph) + for stratum2 in sources: + pool.add(stratum2) + + self.assertRaises(morphlib.artifactresolver.MutualDependencyError, + self.resolver.resolve_artifacts, pool) + + def test_detection_of_chunk_dependencies_in_invalid_order(self): + pool = morphlib.sourcepool.SourcePool() + + loader = morphlib.morphloader.MorphologyLoader() + morph = loader.load_from_string( + ''' + name: stratum + kind: stratum + build-depends: [] + chunks: + - name: chunk1 + repo: repo + ref: original/ref + build-depends: + - chunk2 + - name: chunk2 + repo: repo + ref: original/ref + build-depends: [] + ''') + sources = morphlib.source.make_sources('repo', 'original/ref', + 'stratum.morph', 'sha1', + 'tree', morph) + for stratum in sources: + pool.add(stratum) + + morph = get_chunk_morphology('chunk1') + sources = morphlib.source.make_sources('repo', 'original/ref', + 'chunk1.morph', 'sha1', + 'tree', morph) + for chunk1 in sources: + pool.add(chunk1) + + morph = get_chunk_morphology('chunk2') + sources = morphlib.source.make_sources('repo', 'original/ref', + 'chunk2.morph', 'sha1', + 'tree', morph) + for chunk2 in sources: + pool.add(chunk2) + + self.assertRaises(morphlib.artifactresolver.DependencyOrderError, + self.resolver.resolve_artifacts, pool) + + +# TODO: Expand test suite to include better dependency checking, many +# tests were removed due to the fundamental change in how artifacts +# and dependencies are constructed diff --git a/morphlib/artifactsplitrule.py b/morphlib/artifactsplitrule.py new file mode 100644 index 00000000..1511d694 --- /dev/null +++ b/morphlib/artifactsplitrule.py @@ -0,0 +1,324 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import collections +import itertools +import re + +import morphlib + + +class Rule(object): + '''Rule base class. + + Rules are passed an object and are expected to determine whether + it matches. It's roughly the same machinery for matching files + as artifacts, it's just that Files are given just the path, while + Artifact matches are given the artifact name and the name of the + source it came from. + + ''' + + def match(self, *args): + return True + + +class FileMatch(Rule): + '''Match a file path against a list of regular expressions. + + If the path matches any of the regular expressions, then the file + is counted as a valid match. + + ''' + + def __init__(self, regexes): + # Possible optimisation: compile regexes as one pattern + self._regexes = [re.compile(r) for r in regexes] + + def match(self, path): + return any(r.match(path) for r in self._regexes) + + def __repr__(self): + return 'FileMatch(%s)' % '|'.join(r.pattern for r in self._regexes) + + +class ArtifactMatch(Rule): + '''Match an artifact's name against a list of regular expressions. + ''' + + def __init__(self, regexes): + # Possible optimisation: compile regexes as one pattern + self._regexes = [re.compile(r) for r in regexes] + + def match(self, (source_name, artifact_name)): + return any(r.match(artifact_name) for r in self._regexes) + + def __repr__(self): + return 'ArtifactMatch(%s)' % '|'.join(r.pattern for r in self._regexes) + + +class ArtifactAssign(Rule): + '''Match only artifacts with the specified source and artifact names. + + This is a valid match if the source and artifact names exactly match. + This is used for explicit artifact assignment e.g. chunk artifact + foo-bins which comes from chunk source foo goes into stratum + bar-runtime. + + ''' + + def __init__(self, source_name, artifact_name): + self._key = (source_name, artifact_name) + + def match(self, (source_name, artifact_name)): + return (source_name, artifact_name) == self._key + + def __repr__(self): + return 'ArtifactAssign(%s, %s)' % self._key + + +class SourceAssign(Rule): + '''Match only artifacts which come from the specified source. + + This is a valid match only if the artifact comes from the specified + source. e.g. all artifacts produced by source bar-runtime go into + system baz + + ''' + + def __init__(self, source_name): + self._source = source_name + + def match(self, (source_name, artifact_name)): + return source_name == self._source + + def __repr__(self): + return 'SourceAssign(%s, *)' % self._source + + +class SplitRules(collections.Iterable): + '''Rules engine for splitting a source's artifacts. + + Rules are added with the .add(artifact, rule) method, though another + SplitRules may be created by passing a SplitRules to the constructor. + + .match(path|(source, artifact)) and .partition(iterable) are used + to determine if an artifact matches the rules. Rules are processed + in order, so more specific matches first can be followed by more + generic catch-all matches. + + ''' + + def __init__(self, *args): + self._rules = list(*args) + + def __iter__(self): + return iter(self._rules) + + def add(self, artifact, rule): + self._rules.append((artifact, rule)) + + @property + def artifacts(self): + '''Get names of all artifacts in the rule set. + + Returns artifact names in the order they were added to the rules, + and not repeating the artifact. + + ''' + + seen = set() + result = [] + for artifact_name, rule in self._rules: + if artifact_name not in seen: + seen.add(artifact_name) + result.append(artifact_name) + return result + + def match(self, *args): + '''Return all artifact names the given argument matches. + + It's returned in match order as a list, so it's possible to + detect overlapping matches, even though most of the time, the + only used entry will be the first. + + ''' + + return [a for a, r in self._rules if r.match(*args)] + + def partition(self, iterable): + '''Match many files or artifacts. + + This function takes an iterable of file names, and groups them + using the rules that have been added to this object. + + This is a convenience function that uses the match() method internally. + + ''' + + matches = collections.defaultdict(list) + overlaps = collections.defaultdict(set) + unmatched = set() + + for arg in iterable: + matched = self.match(arg) + if len(matched) == 0: + unmatched.add(arg) + continue + if len(matched) != 1: + overlaps[arg].update(matched) + matches[matched[0]].append(arg) + + return matches, overlaps, unmatched + + def __repr__(self): + return 'SplitRules(%s)' % ', '.join( + '%s=%s' % (artifact, rule) + for artifact, rule in self._rules) + + +# TODO: Work out a good way to feed new defaults in. This is good for +# the usual Linux userspace, but we may find issues and need a +# migration path to a more useful set, or develop a system with +# a different layout, like Android. +DEFAULT_CHUNK_RULES = [ + ('-bins', [ r"(usr/)?s?bin/.*" ]), + ('-libs', [ + r"(usr/)?lib(32|64)?/lib[^/]*\.so(\.\d+)*", + r"(usr/)libexec/.*"]), + ('-devel', [ + r"(usr/)?include/.*", + r"(usr/)?lib(32|64)?/lib.*\.a", + r"(usr/)?lib(32|64)?/lib.*\.la", + r"(usr/)?(lib(32|64)?|share)/pkgconfig/.*\.pc"]), + ('-doc', [ + r"(usr/)?share/doc/.*", + r"(usr/)?share/man/.*", + r"(usr/)?share/info/.*"]), + ('-locale', [ + r"(usr/)?share/locale/.*", + r"(usr/)?share/i18n/.*", + r"(usr/)?share/zoneinfo/.*"]), + ('-misc', [ r".*" ]), +] + + +DEFAULT_STRATUM_RULES = [ + ('-devel', [ + r'.*-devel', + r'.*-debug', + r'.*-doc']), + ('-runtime', [ + r'.*-bins', + r'.*-libs', + r'.*-locale', + r'.*-misc', + r'.*']), +] + + +def unify_chunk_matches(morphology, default_rules=DEFAULT_CHUNK_RULES): + '''Create split rules including defaults and per-chunk rules. + + With rules specified in the morphology's 'products' field and the + default rules for chunks, generate rules to match the files produced + by building the chunk to the chunk artifact they should be put in. + + ''' + + split_rules = SplitRules() + + for ca_name, patterns in ((d['artifact'], d['include']) + for d in morphology['products']): + split_rules.add(ca_name, FileMatch(patterns)) + + name = morphology['name'] + for suffix, patterns in default_rules: + ca_name = name + suffix + # Explicit rules override the default rules. This is an all-or-nothing + # override: there is no way to extend the default split rules right now + # without duplicating them in the chunk morphology. + if ca_name not in split_rules.artifacts: + split_rules.add(ca_name, FileMatch(patterns)) + + return split_rules + + +def unify_stratum_matches(morphology, default_rules=DEFAULT_STRATUM_RULES): + '''Create split rules including defaults and per-stratum rules. + + With rules specified in the chunk spec's 'artifacts' fields, the + stratum's 'products' field and the default rules for strata, generate + rules to match the artifacts produced by building the chunks in the + strata to the stratum artifact they should be put in. + + ''' + + assignment_split_rules = SplitRules() + for spec in morphology['chunks']: + source_name = spec['name'] + for ca_name, sta_name in sorted(spec.get('artifacts', {}).iteritems()): + assignment_split_rules.add(sta_name, + ArtifactAssign(source_name, ca_name)) + + # Construct match rules separately, so we can use the SplitRules object's + # own knowledge of which rules already exist to determine whether + # to include the default rule. + # Rather than use the existing SplitRules, use a new one, since + # match rules suppliment assignment rules, rather than replace. + match_split_rules = SplitRules() + for sta_name, patterns in ((d['artifact'], d['include']) + for d in morphology.get('products', {})): + match_split_rules.add(sta_name, ArtifactMatch(patterns)) + + for suffix, patterns in default_rules: + sta_name = morphology['name'] + suffix + # Explicit rules override the default rules. This is an all-or-nothing + # override: there is no way to extend the default split rules right now + # without duplicating them in the chunk morphology. + if sta_name not in match_split_rules.artifacts: + match_split_rules.add(sta_name, ArtifactMatch(patterns)) + + # Construct a new SplitRules with the assignments before matches + return SplitRules(itertools.chain(assignment_split_rules, + match_split_rules)) + + +def unify_system_matches(morphology): + '''Create split rules including defaults and per-chunk rules. + + With rules specified in the morphology's 'products' field and the + default rules for chunks, generate rules to match the files produced + by building the chunk to the chunk artifact they should be put in. + + ''' + + name = morphology['name'] + '-rootfs' + split_rules = SplitRules() + + for spec in morphology['strata']: + source_name = spec.get('name', spec['morph']) + if spec.get('artifacts', None) is None: + split_rules.add(name, SourceAssign(source_name)) + continue + for sta_name in spec['artifacts']: + split_rules.add(name, ArtifactAssign(source_name, sta_name)) + + return split_rules + + +def unify_cluster_matches(_): + return SplitRules() diff --git a/morphlib/bins.py b/morphlib/bins.py new file mode 100644 index 00000000..560e68bb --- /dev/null +++ b/morphlib/bins.py @@ -0,0 +1,236 @@ +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''Functions for dealing with Baserock binaries. + +Binaries are chunks, strata, and system images. + +''' + + +import cliapp +import logging +import os +import sys +import re +import errno +import stat +import shutil +import tarfile + +import morphlib + +from morphlib.extractedtarball import ExtractedTarball +from morphlib.mountableimage import MountableImage + +# Work around http://bugs.python.org/issue16477 +if sys.version_info < (2, 7, 4): # pragma: no cover + def safe_makefile(self, tarinfo, targetpath): + '''Create a file, closing correctly in case of exception''' + + source = self.extractfile(tarinfo) + try: + with open(targetpath, "wb") as target: + shutil.copyfileobj(source, target) + finally: + source.close() + tarfile.TarFile.makefile = safe_makefile + +# Work around http://bugs.python.org/issue12841 +if sys.version_info < (2, 7, 3): # pragma: no cover + try: + import grp, pwd + except ImportError: + grp = pwd = None + + def fixed_chown(self, tarinfo, targetpath): + '''Set owner of targetpath according to tarinfo.''' + + if pwd and hasattr(os, "geteuid") and os.geteuid() == 0: + # We have to be root to do so. + try: + g = grp.getgrnam(tarinfo.gname)[2] + except KeyError: + g = tarinfo.gid + try: + u = pwd.getpwnam(tarinfo.uname)[2] + except KeyError: + u = tarinfo.uid + try: + if tarinfo.issym() and hasattr(os, "lchown"): + os.lchown(targetpath, u, g) + else: + if sys.platform != "os2emx": + os.chown(targetpath, u, g) + except EnvironmentError, e: + raise ExtractError("could not change owner") + tarfile.TarFile.chown = fixed_chown + +def create_chunk(rootdir, f, include, dump_memory_profile=None): + '''Create a chunk from the contents of a directory. + + ``f`` is an open file handle, to which the tar file is written. + + ''' + + dump_memory_profile = dump_memory_profile or (lambda msg: None) + + # This timestamp is used to normalize the mtime for every file in + # chunk artifact. This is useful to avoid problems from smallish + # clock skew. It needs to be recent enough, however, that GNU tar + # does not complain about an implausibly old timestamp. + normalized_timestamp = 683074800 + + dump_memory_profile('at beginning of create_chunk') + + path_pairs = [(relname, os.path.join(rootdir, relname)) + for relname in include] + tar = tarfile.open(fileobj=f, mode='w') + for relname, filename in path_pairs: + # Normalize mtime for everything. + tarinfo = tar.gettarinfo(filename, + arcname=relname) + tarinfo.ctime = normalized_timestamp + tarinfo.mtime = normalized_timestamp + if tarinfo.isreg(): + with open(filename, 'rb') as f: + tar.addfile(tarinfo, fileobj=f) + else: + tar.addfile(tarinfo) + tar.close() + + for relname, filename in reversed(path_pairs): + if os.path.isdir(filename) and not os.path.islink(filename): + continue + else: + os.remove(filename) + dump_memory_profile('after removing in create_chunks') + + +def unpack_binary_from_file(f, dirname): # pragma: no cover + '''Unpack a binary into a directory. + + The directory must exist already. + + ''' + + # This is evil, but necessary. For some reason Python's system + # call wrappers (os.mknod and such) do not (always?) set the + # filename attribute of the OSError exception they raise. We + # fix that by monkey patching the tf instance with wrappers + # for the relevant methods to add things. The wrapper further + # ignores EEXIST errors, since we do not (currently!) care about + # overwriting files. + + def follow_symlink(path): # pragma: no cover + try: + return os.stat(path) + except OSError: + return None + + def prepare_extract(tarinfo, targetpath): # pragma: no cover + '''Prepare to extract a tar file member onto targetpath? + + If the target already exist, and we can live with it or + remove it, we do so. Otherwise, raise an error. + + It's OK to extract if: + + * the target does not exist + * the member is a directory a directory and the + target is a directory or a symlink to a directory + (just extract, no need to remove) + * the member is not a directory, and the target is not a directory + or a symlink to a directory (remove target, then extract) + + ''' + + try: + existing = os.lstat(targetpath) + except OSError: + return True # target does not exist + + if tarinfo.isdir(): + if stat.S_ISDIR(existing.st_mode): + return True + elif stat.S_ISLNK(existing.st_mode): + st = follow_symlink(targetpath) + return st and stat.S_ISDIR(st.st_mode) + else: + if stat.S_ISDIR(existing.st_mode): + return False + elif stat.S_ISLNK(existing.st_mode): + st = follow_symlink(targetpath) + if st and not stat.S_ISDIR(st.st_mode): + os.remove(targetpath) + return True + else: + os.remove(targetpath) + return True + return False + + def monkey_patcher(real): + def make_something(tarinfo, targetpath): # pragma: no cover + prepare_extract(tarinfo, targetpath) + try: + ret = real(tarinfo, targetpath) + except (IOError, OSError), e: + if e.errno != errno.EEXIST: + if e.filename is None: + e.filename = targetpath + raise + else: + return ret + return make_something + + tf = tarfile.open(fileobj=f, errorlevel=2) + tf.makedir = monkey_patcher(tf.makedir) + tf.makefile = monkey_patcher(tf.makefile) + tf.makeunknown = monkey_patcher(tf.makeunknown) + tf.makefifo = monkey_patcher(tf.makefifo) + tf.makedev = monkey_patcher(tf.makedev) + tf.makelink = monkey_patcher(tf.makelink) + + try: + tf.extractall(path=dirname) + finally: + tf.close() + + +def unpack_binary(filename, dirname): + with open(filename, "rb") as f: + unpack_binary_from_file(f, dirname) + + +class ArtifactNotMountableError(cliapp.AppException): # pragma: no cover + + def __init__(self, filename): + cliapp.AppException.__init__( + self, 'Artifact %s cannot be extracted or mounted' % filename) + + +def call_in_artifact_directory(app, filename, callback): # pragma: no cover + '''Call a function in a directory the artifact is extracted/mounted in.''' + + try: + with ExtractedTarball(app, filename) as dirname: + callback(dirname) + except tarfile.TarError: + try: + with MountableImage(app, filename) as dirname: + callback(dirname) + except (IOError, OSError): + raise ArtifactNotMountableError(filename) diff --git a/morphlib/bins_tests.py b/morphlib/bins_tests.py new file mode 100644 index 00000000..60361ece --- /dev/null +++ b/morphlib/bins_tests.py @@ -0,0 +1,217 @@ +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import gzip +import os +import shutil +import stat +import tempfile +import tarfile +import unittest +import StringIO + +import morphlib + + +class BinsTest(unittest.TestCase): + + def recursive_lstat(self, root): + '''Return a list of (pathname, stat) pairs for everything in root. + + Pathnames are relative to root. Directories are recursed into. + The stat result is selective, not all fields are used. + + ''' + + def remove_root(pathname): + self.assertTrue(pathname.startswith(root)) + if pathname == root: + return '.' + else: + return pathname[(len(root) + 1):] + + def lstat(filename): + st = os.lstat(filename) + + # For directories, the size is dependent on the contents, and + # possibly on things that have been deleted already. An unpacked + # directory can be identical even if the size field is different. + # So we ignore it for directories. + # + # Similarly, the mtime for a directory will change when we remove + # files in the directory, and a different mtime is not necessarily + # a sign of a bad unpack. It's possible for the tests to arrange + # for everything to be correct as far as directory mtimes are + # concerned, but it's not worth it, so we fudge the mtime too. + if stat.S_ISDIR(st.st_mode): + return (st.st_mode, 0, 0) + else: + return (st.st_mode, st.st_size, 0) + + result = [] + + for dirname, subdirs, basenames in os.walk(root): + result.append((remove_root(dirname), lstat(dirname))) + for basename in sorted(basenames): + filename = os.path.join(dirname, basename) + result.append((remove_root(filename), lstat(filename))) + subdirs.sort() + + return result + + +class ChunkTests(BinsTest): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.instdir = os.path.join(self.tempdir, 'inst') + self.chunk_file = os.path.join(self.tempdir, 'chunk') + self.chunk_f = open(self.chunk_file, 'wb') + self.unpacked = os.path.join(self.tempdir, 'unpacked') + + def tearDown(self): + self.chunk_f.close() + shutil.rmtree(self.tempdir) + + def populate_instdir(self): + timestamp = 12765 + + os.mkdir(self.instdir) + + bindir = os.path.join(self.instdir, 'bin') + os.mkdir(bindir) + filename = os.path.join(bindir, 'foo') + with open(filename, 'w'): + pass + os.utime(filename, (timestamp, timestamp)) + + libdir = os.path.join(self.instdir, 'lib') + os.mkdir(libdir) + filename = os.path.join(libdir, 'libfoo.so') + with open(filename, 'w'): + pass + os.utime(filename, (timestamp, timestamp)) + + self.instdir_orig_files = self.recursive_lstat(self.instdir) + + def create_chunk(self, includes): + self.populate_instdir() + morphlib.bins.create_chunk(self.instdir, self.chunk_f, includes) + self.chunk_f.flush() + + def unpack_chunk(self): + os.mkdir(self.unpacked) + morphlib.bins.unpack_binary(self.chunk_file, self.unpacked) + + def test_empties_files(self): + self.create_chunk(['bin/foo', 'lib/libfoo.so']) + self.assertEqual([x for x, y in self.recursive_lstat(self.instdir)], + ['.', 'bin', 'lib']) + + def test_creates_and_unpacks_chunk_exactly(self): + self.create_chunk(['bin', 'bin/foo', 'lib', 'lib/libfoo.so']) + self.unpack_chunk() + self.assertEqual(self.instdir_orig_files, + self.recursive_lstat(self.unpacked)) + + def test_uses_only_matching_names(self): + self.create_chunk(['bin/foo']) + self.unpack_chunk() + self.assertEqual([x for x, y in self.recursive_lstat(self.unpacked)], + ['.', 'bin', 'bin/foo']) + self.assertEqual([x for x, y in self.recursive_lstat(self.instdir)], + ['.', 'bin', 'lib', 'lib/libfoo.so']) + + def test_does_not_compress_artifact(self): + self.create_chunk(['bin']) + f = gzip.open(self.chunk_file) + self.assertRaises(IOError, f.read) + f.close() + + +class ExtractTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.instdir = os.path.join(self.tempdir, 'inst') + self.unpacked = os.path.join(self.tempdir, 'unpacked') + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def create_chunk(self, callback): + fh = StringIO.StringIO() + os.mkdir(self.instdir) + patterns = callback(self.instdir) + morphlib.bins.create_chunk(self.instdir, fh, patterns) + shutil.rmtree(self.instdir) + fh.flush() + fh.seek(0) + return fh + + def test_extracted_files_replace_links(self): + def make_linkfile(basedir): + with open(os.path.join(basedir, 'babar'), 'w') as f: + pass + os.symlink('babar', os.path.join(basedir, 'bar')) + return ['babar'] + linktar = self.create_chunk(make_linkfile) + + def make_file(basedir): + with open(os.path.join(basedir, 'bar'), 'w') as f: + pass + return ['bar'] + filetar = self.create_chunk(make_file) + + os.mkdir(self.unpacked) + morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) + morphlib.bins.unpack_binary_from_file(filetar, self.unpacked) + mode = os.lstat(os.path.join(self.unpacked, 'bar')).st_mode + self.assertTrue(stat.S_ISREG(mode)) + + def test_extracted_dirs_keep_links(self): + def make_usrlink(basedir): + os.symlink('.', os.path.join(basedir, 'usr')) + return ['usr'] + linktar = self.create_chunk(make_usrlink) + + def make_usrdir(basedir): + os.mkdir(os.path.join(basedir, 'usr')) + return ['usr'] + dirtar = self.create_chunk(make_usrdir) + + morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) + morphlib.bins.unpack_binary_from_file(dirtar, self.unpacked) + mode = os.lstat(os.path.join(self.unpacked, 'usr')).st_mode + self.assertTrue(stat.S_ISLNK(mode)) + + def test_extracted_files_follow_links(self): + def make_usrlink(basedir): + os.symlink('.', os.path.join(basedir, 'usr')) + return ['usr'] + linktar = self.create_chunk(make_usrlink) + + def make_usrdir(basedir): + os.mkdir(os.path.join(basedir, 'usr')) + with open(os.path.join(basedir, 'usr', 'foo'), 'w') as f: + pass + return ['usr', 'usr/foo'] + dirtar = self.create_chunk(make_usrdir) + + morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) + morphlib.bins.unpack_binary_from_file(dirtar, self.unpacked) + mode = os.lstat(os.path.join(self.unpacked, 'foo')).st_mode + self.assertTrue(stat.S_ISREG(mode)) diff --git a/morphlib/branchmanager.py b/morphlib/branchmanager.py new file mode 100644 index 00000000..a33b4ccb --- /dev/null +++ b/morphlib/branchmanager.py @@ -0,0 +1,224 @@ +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import collections + +import morphlib + + +class RefCleanupError(cliapp.AppException): + def __init__(self, primary_exception, exceptions): + self.exceptions = exceptions + self.ex_nr = ex_nr = len(exceptions) + self.primary_exception = primary_exception + cliapp.AppException.__init__( + self, '%(ex_nr)d exceptions caught when cleaning up '\ + 'after exception: %(primary_exception)r: '\ + '%(exceptions)r' % locals()) + + +class LocalRefManager(object): + '''Provide atomic update over a set of refs in a set of repositories. + + When used in a with statement, if an exception is raised in the + body, then any ref changes are reverted, so deletes get replaced, + new branches get deleted and ref changes are changed back to the + value before the LocalRefManager was created. + + By default, changes are kept after the with statement ends. This can + be overridden to revert after the manager exits by passing True to + the construcor. + + with LocalRefManager(True) as lrm: + # Update refs with lrm.update, lrm.add or lrm.delete + # Use changed refs + # refs are back to their previous value + + There is also an explicit .close() method to clean up after the + context has exited like so: + + with LocalRefManager() as lrm: + # update refs + # Do something with altered refs + lrm.close() # Explicitly clean up + + The name .close() was chosen for the cleanup method, so the + LocalRefManager object may also be used again in a second with + statement using contextlib.closing(). + + with LocalRefManager() as lrm: + # update refs + with contextlib.closing(lrm) as lrm: + # Do something with pushed refs and clean up if there is an + # exception + + This is also useful if the LocalRefManager is nested in another + object, since the .close() method can be called in that object's + cleanup method. + + ''' + + def __init__(self, cleanup_on_success=False): + self._cleanup_on_success = cleanup_on_success + self._cleanup = collections.deque() + + def __enter__(self): + return self + + def __exit__(self, etype, evalue, estack): + # No exception was raised, so no cleanup is required + if not self._cleanup_on_success and evalue is None: + return + self.close(evalue) + + def close(self, primary=None): + exceptions = [] + d = self._cleanup + while d: + op, args = d.pop() + try: + op(*args) + except Exception, e: + exceptions.append((op, args, e)) + if exceptions: + raise RefCleanupError(primary, exceptions) + + def update(self, gd, ref, commit, old_commit, message=None): + '''Update a git repository's ref, reverting it on failure. + + Use gd and the other parameters to update a ref to a new value, + and if an execption is raised in the body of the with statement + the LocalRefManager is used in, revert the update back to its + old value. + + See morphlib.gitdir.update_ref for more information. + + ''' + + gd.update_ref(ref, commit, old_commit, message) + # Register a cleanup callback of setting the ref back to its old value + self._cleanup.append((type(gd).update_ref, + (gd, ref, old_commit, commit, + message and 'Revert ' + message))) + + def add(self, gd, ref, commit, message=None): + '''Add ref to a git repository, removing it on failure. + + Use gd and the other parameters to add a new ref to the repository, + and if an execption is raised in the body of the with statement + the LocalRefManager is used in, delete the ref. + + See morphlib.gitdir.add_ref for more information. + + ''' + + gd.add_ref(ref, commit, message) + # Register a cleanup callback of deleting the newly added ref. + self._cleanup.append((type(gd).delete_ref, (gd, ref, commit, + message and 'Revert ' + message))) + + def delete(self, gd, ref, old_commit, message=None): + '''Delete ref from a git repository, reinstating it on failure. + + Use gd and the other parameters to delete an existing ref from + the repository, and if an execption is raised in the body of the + with statement the LocalRefManager is used in, re-create the ref. + + See morphlib.gitdir.add_ref for more information. + + ''' + + gd.delete_ref(ref, old_commit, message) + # Register a cleanup callback of replacing the deleted ref. + self._cleanup.append((type(gd).add_ref, (gd, ref, old_commit, + message and 'Revert ' + message))) + + +class RemoteRefManager(object): + '''Provide temporary pushes to remote repositories. + + When used in a with statement, if an exception is raised in the body, + then any pushed refs are reverted, so deletes get replaced and new + branches get deleted. + + By default it will also undo pushed refs when an exception is not + raised, this can be overridden by passing False to the constructor. + + There is also an explicit .close() method to clean up after the + context has exited like so: + + with RemoteRefManager(False) as rrm: + # push refs with rrm.push(...) + # Do something with pushed refs + rrm.close() # Explicitly clean up + + The name .close() was chosen for the cleanup method, so the + RemoteRefManager object may also be used again in a second with + statement using contextlib.closing(). + + with RemoteRefManager(False) as rrm: + rrm.push(...) + with contextlib.closing(rrm) as rrm: + # Do something with pushed refs and clean up if there is an + # exception + + This is also useful if the RemoteRefManager is nested in another + object, since the .close() method can be called in that object's + cleanup method. + + ''' + + def __init__(self, cleanup_on_success=True): + self._cleanup_on_success = cleanup_on_success + self._cleanup = collections.deque() + + def __enter__(self): + return self + + def __exit__(self, etype, evalue, estack): + if not self._cleanup_on_success and evalue is None: + return + self.close(evalue) + + def close(self, primary=None): + exceptions = [] + d = self._cleanup + while d: + remote, refspecs = d.pop() + try: + remote.push(*refspecs) + except Exception, e: + exceptions.append((remote, refspecs, e)) + if exceptions: + raise RefCleanupError(primary, exceptions) + + def push(self, remote, *refspecs): + '''Push refspecs to remote and revert on failure. + + Push the specified refspecs to the remote and reverse the change + after the end of the block the with statement the RemoteRefManager + is used in. + + ''' + + # Calculate the refspecs required to undo the pushed changes. + delete_specs = tuple(rs.revert() for rs in refspecs) + result = remote.push(*refspecs) + # Register cleanup after pushing, so that if this push fails, + # we don't try to undo it. + self._cleanup.append((remote, delete_specs)) + return result diff --git a/morphlib/branchmanager_tests.py b/morphlib/branchmanager_tests.py new file mode 100644 index 00000000..cf3be73c --- /dev/null +++ b/morphlib/branchmanager_tests.py @@ -0,0 +1,432 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import os +import shutil +import tempfile +import unittest + +import morphlib + + +class LocalRefManagerTests(unittest.TestCase): + + REPO_COUNT = 3 + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.repos = [] + for i in xrange(self.REPO_COUNT): + dirname = os.path.join(self.tempdir, 'repo%d' % i) + os.mkdir(dirname) + gd = morphlib.gitdir.init(dirname) + with open(os.path.join(dirname, 'foo'), 'w') as f: + f.write('dummy text\n') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Initial commit') + morphlib.git.gitcmd(gd._runcmd, 'checkout', '-b', 'dev-branch') + with open(os.path.join(dirname, 'foo'), 'w') as f: + f.write('updated text\n') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Second commit') + self.repos.append(gd) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + @staticmethod + def lrm(*args, **kwargs): + return morphlib.branchmanager.LocalRefManager(*args, **kwargs) + + def test_refs_added(self): + refinfo = [] + with self.lrm() as lrm: + for i, gd in enumerate(self.repos): + commit = gd.resolve_ref_to_commit('refs/heads/master') + refinfo.append(commit) + lrm.add(gd, 'refs/heads/create%d' % i, commit) + for i, gd in enumerate(self.repos): + self.assertEqual(gd.resolve_ref_to_commit( + 'refs/heads/create%d' % i), + refinfo[i]) + + def test_add_rollback(self): + with self.assertRaises(Exception): + with self.lrm() as lrm: + for i, gd in enumerate(self.repos): + commit = gd.resolve_ref_to_commit('refs/heads/master') + lrm.add(gd, 'refs/heads/create%d' % i, commit) + raise Exception() + for i, gd in enumerate(self.repos): + with self.assertRaises(morphlib.gitdir.InvalidRefError): + gd.resolve_ref_to_commit('refs/heads/create%d' % i) + + def test_add_rollback_on_success(self): + with self.lrm(True) as lrm: + for i, gd in enumerate(self.repos): + commit = gd.resolve_ref_to_commit('refs/heads/master') + lrm.add(gd, 'refs/heads/create%d' % i, commit) + for i, gd in enumerate(self.repos): + with self.assertRaises(morphlib.gitdir.InvalidRefError): + gd.resolve_ref_to_commit('refs/heads/create%d' % i) + + def test_add_rollback_deferred(self): + with self.lrm(False) as lrm: + for i, gd in enumerate(self.repos): + commit = gd.resolve_ref_to_commit('refs/heads/master') + lrm.add(gd, 'refs/heads/create%d' % i, commit) + lrm.close() + for i, gd in enumerate(self.repos): + with self.assertRaises(morphlib.gitdir.InvalidRefError): + gd.resolve_ref_to_commit('refs/heads/create%d' % i) + + def test_add_rollback_failure(self): + failure_exception = Exception() + with self.assertRaises(morphlib.branchmanager.RefCleanupError) as cm: + with self.lrm() as lrm: + for i, gd in enumerate(self.repos): + ref = 'refs/heads/create%d' % i + commit = gd.resolve_ref_to_commit('refs/heads/master') + lrm.add(gd, ref, commit) + # Make changes independent of LRM, so that rollback fails + new_commit = gd.resolve_ref_to_commit( + 'refs/heads/dev-branch') + gd.update_ref(ref, new_commit, commit) + raise failure_exception + self.assertEqual(cm.exception.primary_exception, failure_exception) + self.assertEqual([e.__class__ for _, _, e in cm.exception.exceptions], + [morphlib.gitdir.RefDeleteError] * self.REPO_COUNT) + + def test_refs_updated(self): + refinfo = [] + with self.lrm() as lrm: + for i, gd in enumerate(self.repos): + old_master = gd.resolve_ref_to_commit('refs/heads/master') + commit = gd.resolve_ref_to_commit('refs/heads/dev-branch') + refinfo.append(commit) + lrm.update(gd, 'refs/heads/master', commit, old_master) + for i, gd in enumerate(self.repos): + self.assertEqual(gd.resolve_ref_to_commit('refs/heads/master'), + refinfo[i]) + + def test_update_rollback(self): + refinfo = [] + with self.assertRaises(Exception): + with self.lrm() as lrm: + for i, gd in enumerate(self.repos): + old_master = gd.resolve_ref_to_commit('refs/heads/master') + commit = gd.resolve_ref_to_commit('refs/heads/dev-branch') + refinfo.append(old_master) + lrm.update(gd, 'refs/heads/master', commit, old_master) + raise Exception() + for i, gd in enumerate(self.repos): + self.assertEqual(gd.resolve_ref_to_commit('refs/heads/master'), + refinfo[i]) + + def test_update_rollback_on_success(self): + refinfo = [] + with self.lrm(True) as lrm: + for i, gd in enumerate(self.repos): + old_master = gd.resolve_ref_to_commit('refs/heads/master') + commit = gd.resolve_ref_to_commit('refs/heads/dev-branch') + refinfo.append(old_master) + lrm.update(gd, 'refs/heads/master', commit, old_master) + for i, gd in enumerate(self.repos): + self.assertEqual(gd.resolve_ref_to_commit('refs/heads/master'), + refinfo[i]) + + def test_update_rollback_deferred(self): + refinfo = [] + with self.lrm(False) as lrm: + for i, gd in enumerate(self.repos): + old_master = gd.resolve_ref_to_commit('refs/heads/master') + commit = gd.resolve_ref_to_commit('refs/heads/dev-branch') + refinfo.append(old_master) + lrm.update(gd, 'refs/heads/master', commit, old_master) + lrm.close() + for i, gd in enumerate(self.repos): + self.assertEqual(gd.resolve_ref_to_commit('refs/heads/master'), + refinfo[i]) + + def test_update_rollback_failure(self): + failure_exception = Exception() + with self.assertRaises(morphlib.branchmanager.RefCleanupError) as cm: + with self.lrm() as lrm: + for i, gd in enumerate(self.repos): + old_master = gd.resolve_ref_to_commit('refs/heads/master') + commit = gd.resolve_ref_to_commit('refs/heads/dev-branch') + lrm.update(gd, 'refs/heads/master', commit, old_master) + # Delete the ref, so rollback fails + gd.delete_ref('refs/heads/master', commit) + raise failure_exception + self.assertEqual(cm.exception.primary_exception, failure_exception) + self.assertEqual([e.__class__ for _, _, e in cm.exception.exceptions], + [morphlib.gitdir.RefUpdateError] * self.REPO_COUNT) + + def test_refs_deleted(self): + with self.lrm() as lrm: + for i, gd in enumerate(self.repos): + commit = gd.resolve_ref_to_commit('refs/heads/master') + lrm.delete(gd, 'refs/heads/master', commit) + for i, gd in enumerate(self.repos): + self.assertRaises(morphlib.gitdir.InvalidRefError, + gd.resolve_ref_to_commit, 'refs/heads/master') + + def test_delete_rollback(self): + refinfo = [] + with self.assertRaises(Exception): + with self.lrm() as lrm: + for i, gd in enumerate(self.repos): + commit = gd.resolve_ref_to_commit('refs/heads/master') + refinfo.append(commit) + lrm.delete(gd, 'refs/heads/master', commit) + raise Exception() + for i, gd in enumerate(self.repos): + self.assertEqual(gd.resolve_ref_to_commit('refs/heads/master'), + refinfo[i]) + + def test_delete_rollback_on_success(self): + refinfo = [] + with self.lrm(True) as lrm: + for i, gd in enumerate(self.repos): + commit = gd.resolve_ref_to_commit('refs/heads/master') + refinfo.append(commit) + lrm.delete(gd, 'refs/heads/master', commit) + for i, gd in enumerate(self.repos): + self.assertEqual(gd.resolve_ref_to_commit('refs/heads/master'), + refinfo[i]) + + def test_delete_rollback_deferred(self): + refinfo = [] + with self.lrm(False) as lrm: + for i, gd in enumerate(self.repos): + commit = gd.resolve_ref_to_commit('refs/heads/master') + refinfo.append(commit) + lrm.delete(gd, 'refs/heads/master', commit) + lrm.close() + for i, gd in enumerate(self.repos): + self.assertEqual(gd.resolve_ref_to_commit('refs/heads/master'), + refinfo[i]) + + def test_delete_rollback_failure(self): + failure_exception = Exception() + with self.assertRaises(morphlib.branchmanager.RefCleanupError) as cm: + with self.lrm() as lrm: + for gd in self.repos: + commit = gd.resolve_ref_to_commit('refs/heads/master') + lrm.delete(gd, 'refs/heads/master', commit) + gd.add_ref('refs/heads/master', commit) + raise failure_exception + self.assertEqual(cm.exception.primary_exception, failure_exception) + self.assertEqual([e.__class__ for _, _, e in cm.exception.exceptions], + [morphlib.gitdir.RefAddError] * self.REPO_COUNT) + + +class RemoteRefManagerTests(unittest.TestCase): + + TARGET_COUNT = 2 + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.source = os.path.join(self.tempdir, 'source') + os.mkdir(self.source) + self.sgd = morphlib.gitdir.init(self.source) + with open(os.path.join(self.source, 'foo'), 'w') as f: + f.write('dummy text\n') + morphlib.git.gitcmd(self.sgd._runcmd, 'add', '.') + morphlib.git.gitcmd(self.sgd._runcmd, 'commit', '-m', 'Initial commit') + morphlib.git.gitcmd(self.sgd._runcmd, 'checkout', '-b', 'dev-branch') + with open(os.path.join(self.source, 'foo'), 'w') as f: + f.write('updated text\n') + morphlib.git.gitcmd(self.sgd._runcmd, 'add', '.') + morphlib.git.gitcmd(self.sgd._runcmd, 'commit', '-m', 'Second commit') + morphlib.git.gitcmd(self.sgd._runcmd, 'checkout', '--orphan', 'no-ff') + with open(os.path.join(self.source, 'foo'), 'w') as f: + f.write('parallel dimension text\n') + morphlib.git.gitcmd(self.sgd._runcmd, 'add', '.') + morphlib.git.gitcmd(self.sgd._runcmd, 'commit', '-m', + 'Non-fast-forward commit') + + self.remotes = [] + for i in xrange(self.TARGET_COUNT): + name = 'remote-%d' % i + dirname = os.path.join(self.tempdir, name) + + # Allow deleting HEAD + morphlib.git.gitcmd(cliapp.runcmd, 'init', '--bare', dirname) + gd = morphlib.gitdir.GitDirectory(dirname) + gd.set_config('receive.denyDeleteCurrent', 'warn') + + morphlib.git.gitcmd(self.sgd._runcmd, 'remote', 'add', + name, dirname) + self.remotes.append((name, dirname, gd)) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + @staticmethod + def list_refs(gd): + out = morphlib.git.gitcmd(gd._runcmd, 'for-each-ref', + '--format=%(refname)%00%(objectname)%00') + return dict(line.split('\0') for line in + out.strip('\0\n').split('\0\n') if line) + + def push_creates(self, rrm): + for name, dirname, gd in self.remotes: + rrm.push(self.sgd.get_remote(name), + morphlib.gitdir.RefSpec('refs/heads/master'), + morphlib.gitdir.RefSpec('refs/heads/dev-branch')) + + def push_deletes(self, rrm): + null_commit = '0' * 40 + master_commit = self.sgd.resolve_ref_to_commit('refs/heads/master') + dev_commit = self.sgd.resolve_ref_to_commit('refs/heads/dev-branch') + for name, dirname, gd in self.remotes: + rrm.push(self.sgd.get_remote(name), + morphlib.gitdir.RefSpec( + source=null_commit, + target='refs/heads/master', + require=master_commit), + morphlib.gitdir.RefSpec( + source=null_commit, + target='refs/heads/dev-branch', + require=dev_commit)) + + def assert_no_remote_branches(self): + for name, dirname, gd in self.remotes: + self.assertEqual(self.list_refs(gd), {}) + + def assert_remote_branches(self): + for name, dirname, gd in self.remotes: + for name, sha1 in self.list_refs(gd).iteritems(): + self.assertEqual(self.sgd.resolve_ref_to_commit(name), sha1) + + def test_rollback_after_create_success(self): + with morphlib.branchmanager.RemoteRefManager() as rrm: + self.push_creates(rrm) + self.assert_remote_branches() + self.assert_no_remote_branches() + + def test_keep_after_create_success(self): + with morphlib.branchmanager.RemoteRefManager(False) as rrm: + self.push_creates(rrm) + self.assert_remote_branches() + + def test_deferred_rollback_after_create_success(self): + with morphlib.branchmanager.RemoteRefManager(False) as rrm: + self.push_creates(rrm) + rrm.close() + self.assert_no_remote_branches() + + def test_rollback_after_create_failure(self): + failure_exception = Exception() + with self.assertRaises(Exception) as cm: + with morphlib.branchmanager.RemoteRefManager() as rrm: + self.push_creates(rrm) + raise failure_exception + self.assertEqual(cm.exception, failure_exception) + self.assert_no_remote_branches() + + @unittest.skip('No way to have conditional delete until Git 1.8.5') + def test_rollback_after_create_cleanup_failure(self): + failure_exception = Exception() + with self.assertRaises(morphlib.branchmanager.RefCleanupError) as cm: + with morphlib.branchmanager.RemoteRefManager() as rrm: + self.push_creates(rrm) + + # Break rollback with a new non-ff commit on master + no_ff = self.sgd.resolve_ref_to_commit('no-ff') + master = 'refs/heads/master' + master_commit = \ + self.sgd.resolve_ref_to_commit('refs/heads/master') + for name, dirname, gd in self.remotes: + r = self.sgd.get_remote(name) + r.push(morphlib.gitdir.RefSpec(source=no_ff, target=master, + require=master_commit, + force=True)) + + raise failure_exception + self.assertEqual(cm.exception.primary_exception, failure_exception) + self.assert_no_remote_branches() + + def test_rollback_after_deletes_success(self): + for name, dirname, gd in self.remotes: + self.sgd.get_remote(name).push( + morphlib.gitdir.RefSpec('master'), + morphlib.gitdir.RefSpec('dev-branch')) + self.assert_remote_branches() + with morphlib.branchmanager.RemoteRefManager() as rrm: + self.push_deletes(rrm) + self.assert_no_remote_branches() + self.assert_remote_branches() + + def test_keep_after_deletes_success(self): + for name, dirname, gd in self.remotes: + self.sgd.get_remote(name).push( + morphlib.gitdir.RefSpec('master'), + morphlib.gitdir.RefSpec('dev-branch')) + self.assert_remote_branches() + with morphlib.branchmanager.RemoteRefManager(False) as rrm: + self.push_deletes(rrm) + self.assert_no_remote_branches() + + def test_deferred_rollback_after_deletes_success(self): + for name, dirname, gd in self.remotes: + self.sgd.get_remote(name).push( + morphlib.gitdir.RefSpec('master'), + morphlib.gitdir.RefSpec('dev-branch')) + self.assert_remote_branches() + with morphlib.branchmanager.RemoteRefManager(False) as rrm: + self.push_deletes(rrm) + rrm.close() + self.assert_remote_branches() + + def test_rollback_after_deletes_failure(self): + failure_exception = Exception() + for name, dirname, gd in self.remotes: + self.sgd.get_remote(name).push( + morphlib.gitdir.RefSpec('master'), + morphlib.gitdir.RefSpec('dev-branch')) + self.assert_remote_branches() + with self.assertRaises(Exception) as cm: + with morphlib.branchmanager.RemoteRefManager() as rrm: + self.push_deletes(rrm) + raise failure_exception + self.assertEqual(cm.exception, failure_exception) + self.assert_remote_branches() + + def test_rollback_after_deletes_cleanup_failure(self): + failure_exception = Exception() + for name, dirname, gd in self.remotes: + self.sgd.get_remote(name).push( + morphlib.gitdir.RefSpec('master'), + morphlib.gitdir.RefSpec('dev-branch')) + with self.assertRaises(morphlib.branchmanager.RefCleanupError) as cm: + with morphlib.branchmanager.RemoteRefManager() as rrm: + self.push_deletes(rrm) + + # Break rollback with a new non-ff commit on master + no_ff = self.sgd.resolve_ref_to_commit('no-ff') + master = 'refs/heads/master' + master_commit = \ + self.sgd.resolve_ref_to_commit('refs/heads/master') + for name, dirname, gd in self.remotes: + r = self.sgd.get_remote(name) + r.push(morphlib.gitdir.RefSpec(source=no_ff, target=master, + require=master_commit)) + + raise failure_exception + self.assertEqual(cm.exception.primary_exception, failure_exception) + diff --git a/morphlib/buildbranch.py b/morphlib/buildbranch.py new file mode 100644 index 00000000..638350e3 --- /dev/null +++ b/morphlib/buildbranch.py @@ -0,0 +1,323 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import collections +import contextlib +import os +import urlparse + +import cliapp +import fs.tempfs + +import morphlib + + +class BuildBranchCleanupError(cliapp.AppException): + def __init__(self, bb, exceptions): + self.bb = bb + self.exceptions = exceptions + ex_nr = len(exceptions) + cliapp.AppException.__init__( + self, '%(ex_nr)d exceptions caught when cleaning up build branch' + % locals()) + + +class BuildBranch(object): + '''Represent the sources modified in a system branch. + + This is an abstraction on top of SystemBranchDirectories, providing + the ability to add uncommitted changes to the temporary build branch, + push temporary build branches and retrieve the correct repository + URI and ref to build the system. + + ''' + + # TODO: This currently always uses the temporary build ref. It + # would be better to not use local repositories and temporary refs, + # so building from a workspace appears to be identical to using + # `morph build-morphology` + def __init__(self, sb, build_ref_prefix): + + self._sb = sb + + self._cleanup = collections.deque() + self._to_push = {} + self._td = fs.tempfs.TempFS() + self._register_cleanup(self._td.close) + + self._branch_root = sb.get_config('branch.root') + branch_uuid = sb.get_config('branch.uuid') + + for gd in sb.list_git_directories(): + try: + repo_uuid = gd.get_config('morph.uuid') + except cliapp.AppException: + # Not a repository cloned by morph, ignore + break + build_ref = os.path.join('refs/heads', build_ref_prefix, + branch_uuid, repo_uuid) + # index is commit of workspace + uncommitted changes may want + # to change to use user's index instead of user's commit, + # so they can add new files first + index = gd.get_index(self._td.getsyspath(repo_uuid)) + index.set_to_tree(gd.resolve_ref_to_tree(gd.HEAD)) + self._to_push[gd] = (build_ref, index) + + rootinfo, = ((gd, index) for gd, (build_ref, index) + in self._to_push.iteritems() + if gd.get_config('morph.repository') == self._branch_root) + self._root, self._root_index = rootinfo + + def _register_cleanup(self, func, *args, **kwargs): + self._cleanup.append((func, args, kwargs)) + + def add_uncommitted_changes(self, add_cb=lambda **kwargs: None): + '''Add any uncommitted changes to temporary build GitIndexes''' + changes_made = False + for gd, (build_ref, index) in self._to_push.iteritems(): + changed = [to_path for code, to_path, from_path + in index.get_uncommitted_changes()] + if not changed: + continue + add_cb(gd=gd, build_ref=gd, changed=changed) + changes_made = True + index.add_files_from_working_tree(changed) + return changes_made + + @staticmethod + def _hash_morphologies(gd, morphologies, loader): + '''Hash morphologies and return object info''' + for morphology in morphologies: + loader.unset_defaults(morphology) + sha1 = gd.store_blob(loader.save_to_string(morphology)) + yield 0100644, sha1, morphology.filename + + def inject_build_refs(self, loader, use_local_repos, + inject_cb=lambda **kwargs: None): + '''Update system and stratum morphologies to point to our branch. + + For all edited repositories, this alter the temporary GitIndex + of the morphs repositories to point their temporary build branch + versions. + + `loader` is a MorphologyLoader that is used to convert morphology + files into their in-memory representations and back again. + + ''' + root_repo = self._root.get_config('morph.repository') + root_ref = self._root.HEAD + morphs = morphlib.morphset.MorphologySet() + for morph in self._sb.load_all_morphologies(loader): + morphs.add_morphology(morph) + + sb_info = {} + for gd, (build_ref, index) in self._to_push.iteritems(): + repo, ref = gd.get_config('morph.repository'), gd.HEAD + sb_info[repo, ref] = (gd, build_ref) + + def filter(m, kind, spec): + return (spec.get('repo'), spec.get('ref')) in sb_info + def process(m, kind, spec): + repo, ref = spec['repo'], spec['ref'] + gd, build_ref = sb_info[repo, ref] + if (repo, ref) == (root_repo, root_ref): + spec['repo'] = None + spec['ref'] = None + return True + if use_local_repos: + spec['repo'] = urlparse.urljoin('file://', gd.dirname) + spec['ref'] = build_ref + return True + + morphs.traverse_specs(process, filter) + + if any(m.dirty for m in morphs.morphologies): + inject_cb(gd=self._root) + + # TODO: Prevent it hashing unchanged morphologies, while still + # hashing uncommitted ones. + self._root_index.add_files_from_index_info( + self._hash_morphologies(self._root, morphs.morphologies, loader)) + + def update_build_refs(self, name, email, uuid, + commit_cb=lambda **kwargs: None): + '''Commit changes in temporary GitIndexes to temporary branches. + + `name` and `email` are required to construct the commit author info. + `uuid` is used to identify each build uniquely and is included + in the commit message. + + A new commit is added to the temporary build branch of each of + the repositories in the SystemBranch with: + 1. The tree of anything currently in the temporary GitIndex. + This is the same as the current commit on HEAD unless + `add_uncommitted_changes` or `inject_build_refs` have + been called. + 2. the parent of the previous temporary commit, or the last + commit of the working tree if there has been no previous + commits + 3. author and committer email as specified by `email`, author + name of `name` and committer name of 'Morph (on behalf of + `name`)' + 4. commit message describing the current build using `uuid` + + ''' + commit_message = 'Morph build %s\n\nSystem branch: %s\n' % \ + (uuid, self._sb.system_branch_name) + author_name = name + committer_name = 'Morph (on behalf of %s)' % name + author_email = committer_email = email + + with morphlib.branchmanager.LocalRefManager() as lrm: + for gd, (build_ref, index) in self._to_push.iteritems(): + tree = index.write_tree() + try: + parent = gd.resolve_ref_to_commit(build_ref) + except morphlib.gitdir.InvalidRefError: + parent = gd.resolve_ref_to_commit(gd.HEAD) + else: + # Skip updating ref if we already have a temporary + # build branch and have this tree on the branch + if tree == gd.resolve_ref_to_tree(build_ref): + continue + + commit_cb(gd=gd, build_ref=build_ref) + + commit = gd.commit_tree(tree, parent=parent, + committer_name=committer_name, + committer_email=committer_email, + author_name=author_name, + author_email=author_email, + message=commit_message) + try: + old_commit = gd.resolve_ref_to_commit(build_ref) + except morphlib.gitdir.InvalidRefError: + lrm.add(gd, build_ref, commit) + else: + # NOTE: This will fail if build_ref pointed to a tag, + # due to resolve_ref_to_commit returning the + # commit id of tags, but since it's only morph + # that touches those refs, it should not be + # a problem. + lrm.update(gd, build_ref, commit, old_commit) + + def get_unpushed_branches(self): + '''Work out which, if any, local branches need to be pushed to build + + NOTE: This assumes that the refs in the morphologies and the + refs in the local checkouts match. + + ''' + for gd, (build_ref, index) in self._to_push.iteritems(): + head_ref = gd.HEAD + upstream_ref = gd.get_upstream_of_branch(head_ref) + if upstream_ref is None: + yield gd + continue + head_sha1 = gd.resolve_ref_to_commit(head_ref) + upstream_sha1 = gd.resolve_ref_to_commit(upstream_ref) + if head_sha1 != upstream_sha1: + yield gd + + def push_build_branches(self, push_cb=lambda **kwargs: None): + '''Push all temporary build branches to the remote repositories. + ''' + with morphlib.branchmanager.RemoteRefManager(False) as rrm: + for gd, (build_ref, index) in self._to_push.iteritems(): + remote = gd.get_remote('origin') + refspec = morphlib.gitdir.RefSpec(build_ref) + push_cb(gd=gd, build_ref=build_ref, + remote=remote, refspec=refspec) + rrm.push(remote, refspec) + self._register_cleanup(rrm.close) + + @property + def root_repo_url(self): + '''URI of the repository that systems may be found in.''' + return self._sb.get_config('branch.root') + + @property + def root_ref(self): + return self._sb.get_config('branch.name') + + @property + def root_local_repo_url(self): + return urlparse.urljoin('file://', self._root.dirname) + + @property + def root_build_ref(self): + '''Name of the ref of the repository that systems may be found in.''' + build_ref, index = self._to_push[self._root] + return build_ref + + def close(self): + '''Clean up any resources acquired during operation.''' + # TODO: This is a common pattern for our context managers, + # we could do with a way to share the common code. I suggest the + # ExitStack from python3.4 or the contextlib2 module. + exceptions = [] + while self._cleanup: + func, args, kwargs = self._cleanup.pop() + try: + func(*args, **kwargs) + except Exception, e: + exceptions.append(e) + if exceptions: + raise BuildBranchCleanupError(self, exceptions) + + +@contextlib.contextmanager +def pushed_build_branch(bb, loader, changes_need_pushing, name, email, + build_uuid, status): + with contextlib.closing(bb) as bb: + def report_add(gd, build_ref, changed): + status(msg='Adding uncommitted changes '\ + 'in %(dirname)s to %(ref)s', + dirname=gd.dirname, ref=build_ref, chatty=True) + changes_made = bb.add_uncommitted_changes(add_cb=report_add) + unpushed = any(bb.get_unpushed_branches()) + + if not changes_made and not unpushed: + yield bb.root_repo_url, bb.root_ref + return + + def report_inject(gd): + status(msg='Injecting temporary build refs '\ + 'into morphologies in %(dirname)s', + dirname=gd.dirname, chatty=True) + bb.inject_build_refs(loader=loader, + use_local_repos=not changes_need_pushing, + inject_cb=report_inject) + + def report_commit(gd, build_ref): + status(msg='Committing changes in %(dirname)s '\ + 'to %(ref)s', + dirname=gd.dirname, ref=build_ref, + chatty=True) + bb.update_build_refs(name, email, build_uuid, + commit_cb=report_commit) + + if changes_need_pushing: + def report_push(gd, build_ref, remote, refspec): + status(msg='Pushing %(ref)s in %(dirname)s '\ + 'to %(remote)s', + ref=build_ref, dirname=gd.dirname, + remote=remote.get_push_url(), chatty=True) + bb.push_build_branches(push_cb=report_push) + + yield bb.root_repo_url, bb.root_build_ref + else: + yield bb.root_local_repo_url, bb.root_build_ref diff --git a/morphlib/buildcommand.py b/morphlib/buildcommand.py new file mode 100644 index 00000000..edd2f0c5 --- /dev/null +++ b/morphlib/buildcommand.py @@ -0,0 +1,575 @@ +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import itertools +import os +import shutil +import logging +import tempfile + +import morphlib +import distbuild + + +class MultipleRootArtifactsError(morphlib.Error): + + def __init__(self, artifacts): + self.msg = ('System build has multiple root artifacts: %r' + % [a.name for a in artifacts]) + self.artifacts = artifacts + + +class BuildCommand(object): + + '''High level logic for building. + + This controls how the whole build process goes. This is a separate + class to enable easy experimentation of different approaches to + the various parts of the process. + + ''' + + def __init__(self, app, build_env = None): + self.supports_local_build = True + + self.app = app + self.lac, self.rac = self.new_artifact_caches() + self.lrc, self.rrc = self.new_repo_caches() + + def build(self, args): + '''Build triplets specified on command line.''' + + self.app.status(msg='Build starts', chatty=True) + + for repo_name, ref, filename in self.app.itertriplets(args): + self.app.status(msg='Building %(repo_name)s %(ref)s %(filename)s', + repo_name=repo_name, ref=ref, filename=filename) + self.app.status(msg='Deciding on task order') + srcpool = self.create_source_pool(repo_name, ref, filename) + self.validate_sources(srcpool) + root_artifact = self.resolve_artifacts(srcpool) + self.build_in_order(root_artifact) + + self.app.status(msg='Build ends successfully') + + def new_artifact_caches(self): + '''Create interfaces for the build artifact caches. + + This includes creating the directories on disk if they are missing. + + ''' + return morphlib.util.new_artifact_caches(self.app.settings) + + def new_repo_caches(self): + return morphlib.util.new_repo_caches(self.app) + + def new_build_env(self, arch): + '''Create a new BuildEnvironment instance.''' + return morphlib.buildenvironment.BuildEnvironment(self.app.settings, + arch) + + def create_source_pool(self, repo_name, ref, filename): + '''Find the source objects required for building a the given artifact + + The SourcePool will contain every stratum and chunk dependency of the + given artifact (which must be a system) but will not take into account + any Git submodules which are required in the build. + + ''' + self.app.status(msg='Creating source pool', chatty=True) + srcpool = self.app.create_source_pool( + self.lrc, self.rrc, repo_name, ref, filename) + + return srcpool + + def validate_sources(self, srcpool): + self.app.status( + msg='Validating cross-morphology references', chatty=True) + self._validate_cross_morphology_references(srcpool) + + self.app.status(msg='Validating for there being non-bootstrap chunks', + chatty=True) + self._validate_has_non_bootstrap_chunks(srcpool) + + def _validate_root_artifact(self, root_artifact): + self._validate_root_kind(root_artifact) + self._validate_architecture(root_artifact) + + @staticmethod + def _validate_root_kind(root_artifact): + root_kind = root_artifact.source.morphology['kind'] + if root_kind != 'system': + raise morphlib.Error( + 'Building a %s directly is not supported' % root_kind) + + def _validate_architecture(self, root_artifact): + '''Perform the validation between root and target architectures.''' + + root_arch = root_artifact.source.morphology['arch'] + host_arch = morphlib.util.get_host_architecture() + if root_arch != host_arch: + raise morphlib.Error( + 'Are you trying to cross-build? ' + 'Host architecture is %s but target is %s' + % (host_arch, root_arch)) + + @staticmethod + def _validate_has_non_bootstrap_chunks(srcpool): + stratum_sources = [src for src in srcpool + if src.morphology['kind'] == 'stratum'] + # any will return true for an empty iterable, which will give + # a false positive when there are no strata. + # This is an error by itself, but the source of this error can + # be better diagnosed later, so we abort validating here. + if not stratum_sources: + return + + if not any(spec.get('build-mode', 'staging') != 'bootstrap' + for src in stratum_sources + for spec in src.morphology['chunks']): + raise morphlib.Error('No non-bootstrap chunks found.') + + def resolve_artifacts(self, srcpool): + '''Resolve the artifacts that will be built for a set of sources''' + + self.app.status(msg='Creating artifact resolver', chatty=True) + ar = morphlib.artifactresolver.ArtifactResolver() + + self.app.status(msg='Resolving artifacts', chatty=True) + artifacts = ar.resolve_artifacts(srcpool) + + self.app.status(msg='Computing build order', chatty=True) + root_artifacts = self._find_root_artifacts(artifacts) + if len(root_artifacts) > 1: + # Validate root artifacts, since validation covers errors + # such as trying to build a chunk or stratum directly, + # and this is one cause for having multiple root artifacts + for root_artifact in root_artifacts: + self._validate_root_artifact(root_artifact) + raise MultipleRootArtifactsError(root_artifacts) + root_artifact = root_artifacts[0] + + # Validate the root artifact here, since it's a costly function + # to finalise it, so any pre finalisation validation is better + # done before that happens, but we also don't want to expose + # the root artifact until it's finalised. + self.app.status(msg='Validating root artifact', chatty=True) + self._validate_root_artifact(root_artifact) + arch = root_artifact.source.morphology['arch'] + + self.app.status(msg='Creating build environment for %(arch)s', + arch=arch, chatty=True) + build_env = self.new_build_env(arch) + + self.app.status(msg='Computing cache keys', chatty=True) + ckc = morphlib.cachekeycomputer.CacheKeyComputer(build_env) + for source in set(a.source for a in artifacts): + source.cache_key = ckc.compute_key(source) + source.cache_id = ckc.get_cache_id(source) + + root_artifact.build_env = build_env + return root_artifact + + def _validate_cross_morphology_references(self, srcpool): + '''Perform validation across all morphologies involved in the build''' + + stratum_names = [] + + for src in srcpool: + kind = src.morphology['kind'] + + # Verify that chunks pointed to by strata really are chunks, etc. + method_name = '_validate_cross_refs_for_%s' % kind + if hasattr(self, method_name): + logging.debug('Calling %s' % method_name) + getattr(self, method_name)(src, srcpool) + else: + logging.warning('No %s' % method_name) + + # Verify stratum build-depends agree with the system's contents. + # It is permissible for a stratum to build-depend on a stratum that + # isn't specified in the target system morphology. + # Multiple references to the same stratum are permitted. This is + # handled by the SourcePool deduplicating added Sources. + # It is forbidden to have two different strata with the same name. + # Hence if a Stratum is defined in the System, and in a Stratum as + # a build-dependency, then they must both have the same Repository + # and Ref specified. + if src.morphology['kind'] == 'stratum': + name = src.name + ref = src.sha1[:7] + self.app.status(msg='Stratum [%(name)s] version is %(ref)s', + name=name, ref=ref) + if name in stratum_names: + raise morphlib.Error( + "Conflicting versions of stratum '%s' appear in the " + "build. Check the contents of the system against the " + "build-depends of the strata." % name) + stratum_names.append(name) + + def _validate_cross_refs_for_system(self, src, srcpool): + self._validate_cross_refs_for_xxx( + src, srcpool, src.morphology['strata'], 'stratum') + + def _validate_cross_refs_for_stratum(self, src, srcpool): + self._validate_cross_refs_for_xxx( + src, srcpool, src.morphology['chunks'], 'chunk') + + def _validate_cross_refs_for_xxx(self, src, srcpool, specs, wanted): + for spec in specs: + repo_name = spec.get('repo') or src.repo_name + ref = spec.get('ref') or src.original_ref + filename = morphlib.util.sanitise_morphology_path( + spec.get('morph', spec.get('name'))) + logging.debug( + 'Validating cross ref to %s:%s:%s' % + (repo_name, ref, filename)) + for other in srcpool.lookup(repo_name, ref, filename): + if other.morphology['kind'] != wanted: + raise morphlib.Error( + '%s %s references %s:%s:%s which is a %s, ' + 'instead of a %s' % + (src.morphology['kind'], + src.name, + repo_name, + ref, + filename, + other.morphology['kind'], + wanted)) + + def _find_root_artifacts(self, artifacts): + '''Find all the root artifacts among a set of artifacts in a DAG. + + It would be nice if the ArtifactResolver would return its results in a + more useful order to save us from needing to do this -- the root object + is known already since that's the one the user asked us to build. + + ''' + + return [a for a in artifacts if not a.dependents] + + @staticmethod + def get_ordered_sources(artifacts): + ordered_sources = [] + known_sources = set() + for artifact in artifacts: + if artifact.source not in known_sources: + known_sources.add(artifact.source) + yield artifact.source + + def build_in_order(self, root_artifact): + '''Build everything specified in a build order.''' + + self.app.status(msg='Building a set of sources', chatty=True) + build_env = root_artifact.build_env + ordered_sources = list(self.get_ordered_sources(root_artifact.walk())) + old_prefix = self.app.status_prefix + for i, s in enumerate(ordered_sources): + self.app.status_prefix = ( + old_prefix + '[Build %(index)d/%(total)d] [%(name)s] ' % { + 'index': (i+1), + 'total': len(ordered_sources), + 'name': s.name, + }) + + self.cache_or_build_source(s, build_env) + + self.app.status_prefix = old_prefix + + def cache_or_build_source(self, source, build_env): + '''Make artifacts of the built source available in the local cache. + + This can be done by retrieving from a remote artifact cache, or if + that doesn't work for some reason, by building the source locally. + + ''' + artifacts = source.artifacts.values() + if self.rac is not None: + try: + self.cache_artifacts_locally(artifacts) + except morphlib.remoteartifactcache.GetError: + # Error is logged by the RemoteArtifactCache object. + pass + + if any(not self.lac.has(artifact) for artifact in artifacts): + self.build_source(source, build_env) + + for a in artifacts: + self.app.status(msg='%(kind)s %(name)s is cached at %(cachepath)s', + kind=source.morphology['kind'], name=a.name, + cachepath=self.lac.artifact_filename(a), + chatty=(source.morphology['kind'] != "system")) + + def build_source(self, source, build_env): + '''Build all artifacts for one source. + + All the dependencies are assumed to be built and available + in either the local or remote cache already. + + ''' + self.app.status(msg='Building %(kind)s %(name)s', + name=source.name, + kind=source.morphology['kind']) + + self.fetch_sources(source) + # TODO: Make an artifact.walk() that takes multiple root artifacts. + # as this does a walk for every artifact. This was the status + # quo before build logic was made to work per-source, but we can + # now do better. + deps = self.get_recursive_deps(source.artifacts.values()) + self.cache_artifacts_locally(deps) + + use_chroot = False + setup_mounts = False + if source.morphology['kind'] == 'chunk': + build_mode = source.build_mode + extra_env = {'PREFIX': source.prefix} + + dep_prefix_set = set(a.source.prefix for a in deps + if a.source.morphology['kind'] == 'chunk') + extra_path = [os.path.join(d, 'bin') for d in dep_prefix_set] + + if build_mode not in ['bootstrap', 'staging', 'test']: + logging.warning('Unknown build mode %s for chunk %s. ' + 'Defaulting to staging mode.' % + (build_mode, artifact.name)) + build_mode = 'staging' + + if build_mode == 'staging': + use_chroot = True + setup_mounts = True + + staging_area = self.create_staging_area(build_env, + use_chroot, + extra_env=extra_env, + extra_path=extra_path) + try: + self.install_dependencies(staging_area, deps, source) + except BaseException: + staging_area.abort() + raise + else: + staging_area = self.create_staging_area(build_env, False) + + self.build_and_cache(staging_area, source, setup_mounts) + self.remove_staging_area(staging_area) + + def get_recursive_deps(self, artifacts): + deps = set() + ordered_deps = [] + for artifact in artifacts: + for dep in artifact.walk(): + if dep not in deps and dep not in artifacts: + deps.add(dep) + ordered_deps.append(dep) + return ordered_deps + + def fetch_sources(self, source): + '''Update the local git repository cache with the sources.''' + + repo_name = source.repo_name + if self.app.settings['no-git-update']: + self.app.status(msg='Not updating existing git repository ' + '%(repo_name)s ' + 'because of no-git-update being set', + chatty=True, + repo_name=repo_name) + source.repo = self.lrc.get_repo(repo_name) + return + + if self.lrc.has_repo(repo_name): + source.repo = self.lrc.get_repo(repo_name) + try: + sha1 = source.sha1 + source.repo.resolve_ref(sha1) + self.app.status(msg='Not updating git repository ' + '%(repo_name)s because it ' + 'already contains sha1 %(sha1)s', + chatty=True, repo_name=repo_name, + sha1=sha1) + except morphlib.cachedrepo.InvalidReferenceError: + self.app.status(msg='Updating %(repo_name)s', + repo_name=repo_name) + source.repo.update() + else: + self.app.status(msg='Cloning %(repo_name)s', + repo_name=repo_name) + source.repo = self.lrc.cache_repo(repo_name) + + # Update submodules. + done = set() + self.app.cache_repo_and_submodules( + self.lrc, source.repo.url, + source.sha1, done) + + def cache_artifacts_locally(self, artifacts): + '''Get artifacts missing from local cache from remote cache.''' + + def fetch_files(to_fetch): + '''Fetch a set of files atomically. + + If an error occurs during the transfer of any files, all downloaded + data is deleted, to ensure integrity of the local cache. + + ''' + try: + for remote, local in to_fetch: + shutil.copyfileobj(remote, local) + except BaseException: + for remote, local in to_fetch: + local.abort() + raise + else: + for remote, local in to_fetch: + remote.close() + local.close() + + for artifact in artifacts: + # This block should fetch all artifact files in one go, using the + # 1.0/artifacts method of morph-cache-server. The code to do that + # needs bringing in from the distbuild.worker_build_connection + # module into morphlib.remoteartififactcache first. + to_fetch = [] + if not self.lac.has(artifact): + to_fetch.append((self.rac.get(artifact), + self.lac.put(artifact))) + + if artifact.source.morphology.needs_artifact_metadata_cached: + if not self.lac.has_artifact_metadata(artifact, 'meta'): + to_fetch.append(( + self.rac.get_artifact_metadata(artifact, 'meta'), + self.lac.put_artifact_metadata(artifact, 'meta'))) + + if len(to_fetch) > 0: + self.app.status( + msg='Fetching to local cache: artifact %(name)s', + name=artifact.name) + fetch_files(to_fetch) + + def create_staging_area(self, build_env, use_chroot=True, extra_env={}, + extra_path=[]): + '''Create the staging area for building a single artifact.''' + + self.app.status(msg='Creating staging area') + staging_dir = tempfile.mkdtemp( + dir=os.path.join(self.app.settings['tempdir'], 'staging')) + staging_area = morphlib.stagingarea.StagingArea( + self.app, staging_dir, build_env, use_chroot, extra_env, + extra_path) + return staging_area + + def remove_staging_area(self, staging_area): + '''Remove the staging area.''' + + self.app.status(msg='Removing staging area') + staging_area.remove() + + # Nasty hack to avoid installing chunks built in 'bootstrap' mode in a + # different stratum when constructing staging areas. + # TODO: make nicer by having chunk morphs keep a reference to the + # stratum they were in + def in_same_stratum(self, s1, s2): + '''Checks whether two chunk sources are from the same stratum. + + In the absence of morphologies tracking where they came from, + this checks whether both sources are depended on by artifacts + that belong to sources which have the same morphology. + + ''' + def dependent_stratum_morphs(source): + dependents = set(itertools.chain.from_iterable( + a.dependents for a in source.artifacts.itervalues())) + dependent_strata = set(s for s in dependents + if s.morphology['kind'] == 'stratum') + return set(s.morphology for s in dependent_strata) + return dependent_stratum_morphs(s1) == dependent_stratum_morphs(s2) + + def install_dependencies(self, staging_area, artifacts, target_source): + '''Install chunk artifacts into staging area. + + We only ever care about chunk artifacts as build dependencies, + so this is not a generic artifact installer into staging area. + Any non-chunk artifacts are silently ignored. + + All artifacts MUST be in the local artifact cache already. + + ''' + + for artifact in artifacts: + if artifact.source.morphology['kind'] != 'chunk': + continue + if artifact.source.build_mode == 'bootstrap': + if not self.in_same_stratum(artifact.source, target_source): + continue + self.app.status( + msg='Installing chunk %(chunk_name)s from cache %(cache)s', + chunk_name=artifact.name, + cache=artifact.source.cache_key[:7], + chatty=True) + handle = self.lac.get(artifact) + staging_area.install_artifact(handle) + + if target_source.build_mode == 'staging': + morphlib.builder2.ldconfig(self.app.runcmd, staging_area.dirname) + + def build_and_cache(self, staging_area, source, setup_mounts): + '''Build a source and put its artifacts into the local cache.''' + + self.app.status(msg='Starting actual build: %(name)s ' + '%(sha1)s', + name=source.name, sha1=source.sha1[:7]) + builder = morphlib.builder2.Builder( + self.app, staging_area, self.lac, self.rac, self.lrc, + self.app.settings['max-jobs'], setup_mounts) + return builder.build_and_cache(source) + +class InitiatorBuildCommand(BuildCommand): + + RECONNECT_INTERVAL = 30 # seconds + MAX_RETRIES = 1 + + def __init__(self, app, addr, port): + self.app = app + self.addr = addr + self.port = port + self.app.settings['push-build-branches'] = True + super(InitiatorBuildCommand, self).__init__(app) + + def build(self, args): + '''Initiate a distributed build on a controller''' + + distbuild.add_crash_conditions(self.app.settings['crash-condition']) + + if len(args) != 3: + raise morphlib.Error( + 'Need repo, ref, morphology triplet to build') + + if self.addr == '': + raise morphlib.Error( + 'Need address of controller to run a distbuild') + + self.app.status(msg='Starting distributed build') + loop = distbuild.MainLoop() + cm = distbuild.InitiatorConnectionMachine(self.app, + self.addr, + self.port, + distbuild.Initiator, + [self.app] + args, + self.RECONNECT_INTERVAL, + self.MAX_RETRIES) + + loop.add_state_machine(cm) + loop.run() diff --git a/morphlib/buildenvironment.py b/morphlib/buildenvironment.py new file mode 100644 index 00000000..68e7e756 --- /dev/null +++ b/morphlib/buildenvironment.py @@ -0,0 +1,129 @@ +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import copy +import cliapp +import os + +import morphlib + + +class BuildEnvironment(): + + '''Represents the build environment for an artifact + + This should be as consistent as possible across builds, but some + artifacts will require tweaks. The intention of this object is + to create one once and call populate() to create an initial state + and when changes are required, call clone() to get another instance + which can be modified. + + ''' + + def __init__(self, settings, arch): + '''Create a new BuildEnvironment object''' + + self.extra_path = [] + + self.env = self._clean_env(settings) + self.env.update(self._env_for_arch(arch)) + + + _osenv = os.environ + _ccache_path = '/usr/lib/ccache' + _override_home = '/tmp' + _override_locale = 'C' + _override_shell = '/bin/sh' + _override_term = 'dumb' + _override_username = 'tomjon' + + def _clean_env(self, settings): + '''Create a fresh set of environment variables for a clean build. + + Return a dict with the new environment. + + ''' + + # copy a set of white-listed variables from the original env + copied_vars = dict.fromkeys([ + 'DISTCC_HOSTS', + 'LD_PRELOAD', + 'LD_LIBRARY_PATH', + 'FAKEROOTKEY', + 'FAKED_MODE', + 'FAKEROOT_FD_BASE', + ]) + for name in copied_vars: + copied_vars[name] = self._osenv.get(name, None) + + env = {} + + # apply the copied variables to the clean env + for name in copied_vars: + if copied_vars[name] is not None: + env[name] = copied_vars[name] + + env['TERM'] = self._override_term + env['SHELL'] = self._override_shell + env['USER'] = \ + env['USERNAME'] = \ + env['LOGNAME'] = self._override_username + env['LC_ALL'] = self._override_locale + env['HOME'] = self._override_home + + if not settings['no-ccache']: + self.extra_path.append(self._ccache_path) +# FIXME: we should set CCACHE_BASEDIR so any objects that refer to their +# current directory get corrected. This improve the cache hit rate +# env['CCACHE_BASEDIR'] = self.tempdir.dirname + env['CCACHE_DIR'] = '/tmp/ccache' + env['CCACHE_EXTRAFILES'] = ':'.join( + f for f in ('/baserock/binutils.meta', + '/baserock/eglibc.meta', + '/baserock/gcc.meta') if os.path.exists(f) + ) + if not settings['no-distcc']: + env['CCACHE_PREFIX'] = 'distcc' + + return env + + def _env_for_arch(self, arch): + '''Set build environment variables specific to the target machine + + These are entirely controlled by the 'arch' field in the system + morphology, which is passed to the morphologies as MORPH_ARCH to + do what they like with. + + ''' + + env = {} + env['MORPH_ARCH'] = arch + + # GNU triplets are widely used, so we handle these in Morph rather + # than leaving it up to individual morphologies. + if arch == 'x86_32': + cpu = 'i686' + else: + cpu = arch + + if arch.startswith('arm'): + abi = 'eabi' + else: + abi = '' + + env['TARGET'] = cpu + '-baserock-linux-gnu' + abi + env['TARGET_STAGE1'] = cpu + '-bootstrap-linux-gnu' + abi + + return env diff --git a/morphlib/buildenvironment_tests.py b/morphlib/buildenvironment_tests.py new file mode 100644 index 00000000..7ae7c2d5 --- /dev/null +++ b/morphlib/buildenvironment_tests.py @@ -0,0 +1,112 @@ +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import copy +import unittest + +import morphlib +from morphlib import buildenvironment + + +class BuildEnvironmentTests(unittest.TestCase): + + def setUp(self): + self.settings = { + 'prefix': '/usr', + 'no-ccache': True, + 'no-distcc': True + } + self.fake_env = { + 'PATH': '/fake_bin', + } + + def new_build_env(self, settings=None, target=None, **kws): + settings = settings or self.settings + target = target or self.target + return buildenvironment.BuildEnvironment(settings, target, **kws) + + def new_build_env(self, settings=None, arch='x86_64'): + settings = settings or self.settings + return buildenvironment.BuildEnvironment(settings, arch) + + def test_copies_whitelist_vars(self): + env = self.fake_env + safe = { + 'DISTCC_HOSTS': 'example.com:example.co.uk', + 'LD_PRELOAD': '/buildenv/lib/libbuildenv.so', + 'LD_LIBRARY_PATH': '/buildenv/lib:/buildenv/lib64', + 'FAKEROOTKEY': 'b011de73', + 'FAKED_MODE': 'non-fakeroot', + 'FAKEROOT_FD_BASE': '-1', + } + env.update(safe) + old_osenv = buildenvironment.BuildEnvironment._osenv + buildenvironment.BuildEnvironment._osenv = env + + buildenv = self.new_build_env() + self.assertEqual(sorted(safe.items()), + sorted([(k, buildenv.env[k]) for k in safe.keys()])) + + buildenvironment.BuildEnvironment._osenv = old_osenv + + def test_user_spellings_equal(self): + buildenv = self.new_build_env() + self.assertTrue(buildenv.env['USER'] == buildenv.env['USERNAME'] == + buildenv.env['LOGNAME']) + + def test_environment_overrides(self): + buildenv = self.new_build_env() + self.assertEqual(buildenv.env['TERM'], buildenv._override_term) + self.assertEqual(buildenv.env['SHELL'], buildenv._override_shell) + self.assertEqual(buildenv.env['USER'], buildenv._override_username) + self.assertEqual(buildenv.env['USERNAME'], buildenv._override_username) + self.assertEqual(buildenv.env['LOGNAME'], buildenv._override_username) + self.assertEqual(buildenv.env['LC_ALL'], buildenv._override_locale) + self.assertEqual(buildenv.env['HOME'], buildenv._override_home) + + def test_arch_x86_64(self): + b = self.new_build_env(arch='x86_64') + self.assertEqual(b.env['MORPH_ARCH'], 'x86_64') + self.assertEqual(b.env['TARGET'], 'x86_64-baserock-linux-gnu') + self.assertEqual(b.env['TARGET_STAGE1'], 'x86_64-bootstrap-linux-gnu') + + def test_arch_x86_32(self): + b = self.new_build_env(arch='x86_32') + self.assertEqual(b.env['MORPH_ARCH'], 'x86_32') + self.assertEqual(b.env['TARGET'], 'i686-baserock-linux-gnu') + self.assertEqual(b.env['TARGET_STAGE1'], 'i686-bootstrap-linux-gnu') + + def test_arch_armv7l(self): + b = self.new_build_env(arch='armv7l') + self.assertEqual(b.env['MORPH_ARCH'], 'armv7l') + self.assertEqual(b.env['TARGET'], 'armv7l-baserock-linux-gnueabi') + self.assertEqual(b.env['TARGET_STAGE1'], + 'armv7l-bootstrap-linux-gnueabi') + + def test_arch_armv7b(self): + b = self.new_build_env(arch='armv7b') + self.assertEqual(b.env['MORPH_ARCH'], 'armv7b') + self.assertEqual(b.env['TARGET'], 'armv7b-baserock-linux-gnueabi') + self.assertEqual(b.env['TARGET_STAGE1'], + 'armv7b-bootstrap-linux-gnueabi') + + def test_ccache_vars_set(self): + new_settings = copy.copy(self.settings) + new_settings['no-ccache'] = False + new_settings['no-distcc'] = False + buildenv = self.new_build_env(settings=new_settings) + self.assertTrue(buildenv._ccache_path in buildenv.extra_path) + self.assertEqual(buildenv.env['CCACHE_PREFIX'], 'distcc') diff --git a/morphlib/builder2.py b/morphlib/builder2.py new file mode 100644 index 00000000..9cd3a074 --- /dev/null +++ b/morphlib/builder2.py @@ -0,0 +1,731 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +from collections import defaultdict +import datetime +import errno +import json +import logging +import os +from os.path import relpath +import shutil +import stat +import tarfile +import time +import traceback +import subprocess +import tempfile +import gzip + +import cliapp + +import morphlib +from morphlib.artifactcachereference import ArtifactCacheReference +import morphlib.gitversion + +SYSTEM_INTEGRATION_PATH = os.path.join('baserock', 'system-integration') + +def extract_sources(app, repo_cache, repo, sha1, srcdir): #pragma: no cover + '''Get sources from git to a source directory, including submodules''' + + def extract_repo(repo, sha1, destdir): + app.status(msg='Extracting %(source)s into %(target)s', + source=repo.original_name, + target=destdir) + + repo.checkout(sha1, destdir) + morphlib.git.reset_workdir(app.runcmd, destdir) + submodules = morphlib.git.Submodules(app, repo.path, sha1) + try: + submodules.load() + except morphlib.git.NoModulesFileError: + return [] + else: + tuples = [] + for sub in submodules: + cached_repo = repo_cache.get_repo(sub.url) + sub_dir = os.path.join(destdir, sub.path) + tuples.append((cached_repo, sub.commit, sub_dir)) + return tuples + + todo = [(repo, sha1, srcdir)] + while todo: + repo, sha1, srcdir = todo.pop() + todo += extract_repo(repo, sha1, srcdir) + set_mtime_recursively(srcdir) + +def set_mtime_recursively(root): # pragma: no cover + '''Set the mtime for every file in a directory tree to the same. + + We do this because git checkout does not set the mtime to anything, + and some projects (binutils, gperf for example) include formatted + documentation and try to randomly build things or not because of + the timestamps. This should help us get more reliable builds. + + ''' + + now = time.time() + for dirname, subdirs, basenames in os.walk(root.encode("utf-8"), + topdown=False): + for basename in basenames: + pathname = os.path.join(dirname, basename) + # we need the following check to ignore broken symlinks + if os.path.exists(pathname): + os.utime(pathname, (now, now)) + os.utime(dirname, (now, now)) + +def ldconfig(runcmd, rootdir): # pragma: no cover + '''Run ldconfig for the filesystem below ``rootdir``. + + Essentially, ``rootdir`` specifies the root of a new system. + Only directories below it are considered. + + ``etc/ld.so.conf`` below ``rootdir`` is assumed to exist and + be populated by the right directories, and should assume + the root directory is ``rootdir``. Example: if ``rootdir`` + is ``/tmp/foo``, then ``/tmp/foo/etc/ld.so.conf`` should + contain ``/lib``, not ``/tmp/foo/lib``. + + The ldconfig found via ``$PATH`` is used, not the one in ``rootdir``, + since in bootstrap mode that might not yet exist, the various + implementations should be compatible enough. + + ''' + + # FIXME: use the version in ROOTDIR, since even in + # bootstrap it will now always exist due to being part of build-essential + + conf = os.path.join(rootdir, 'etc', 'ld.so.conf') + if os.path.exists(conf): + logging.debug('Running ldconfig for %s' % rootdir) + cache = os.path.join(rootdir, 'etc', 'ld.so.cache') + + # The following trickery with $PATH is necessary during the Baserock + # bootstrap build: we are not guaranteed that PATH contains the + # directory (/sbin conventionally) that ldconfig is in. Then again, + # it might, and if so, we don't want to hardware a particular + # location. So we add the possible locations to the end of $PATH + env = dict(os.environ) + old_path = env['PATH'] + env['PATH'] = '%s:/sbin:/usr/sbin:/usr/local/sbin' % old_path + runcmd(['ldconfig', '-r', rootdir], env=env) + else: + logging.debug('No %s, not running ldconfig' % conf) + + +def download_depends(constituents, lac, rac, metadatas=None): + for constituent in constituents: + if not lac.has(constituent): + source = rac.get(constituent) + target = lac.put(constituent) + shutil.copyfileobj(source, target) + target.close() + source.close() + if metadatas is not None: + for metadata in metadatas: + if not lac.has_artifact_metadata(constituent, metadata): + if rac.has_artifact_metadata(constituent, metadata): + src = rac.get_artifact_metadata(constituent, metadata) + dst = lac.put_artifact_metadata(constituent, metadata) + shutil.copyfileobj(src, dst) + dst.close() + src.close() + + +def get_chunk_files(f): # pragma: no cover + tar = tarfile.open(fileobj=f) + for member in tar.getmembers(): + if member.type is not tarfile.DIRTYPE: + yield member.name + tar.close() + + +def get_stratum_files(f, lac): # pragma: no cover + for ca in (ArtifactCacheReference(a) + for a in json.load(f, encoding='unicode-escape')): + cf = lac.get(ca) + for filename in get_chunk_files(cf): + yield filename + cf.close() + + +class BuilderBase(object): + + '''Base class for building artifacts.''' + + def __init__(self, app, staging_area, local_artifact_cache, + remote_artifact_cache, source, repo_cache, max_jobs, + setup_mounts): + self.app = app + self.staging_area = staging_area + self.local_artifact_cache = local_artifact_cache + self.remote_artifact_cache = remote_artifact_cache + self.source = source + self.repo_cache = repo_cache + self.max_jobs = max_jobs + self.build_watch = morphlib.stopwatch.Stopwatch() + self.setup_mounts = setup_mounts + + def save_build_times(self): + '''Write the times captured by the stopwatch''' + meta = { + 'build-times': {} + } + for stage in self.build_watch.ticks.iterkeys(): + meta['build-times'][stage] = { + 'start': '%s' % self.build_watch.start_time(stage), + 'stop': '%s' % self.build_watch.stop_time(stage), + 'delta': '%.4f' % self.build_watch.start_stop_seconds(stage) + } + + logging.debug('Writing metadata to the cache') + with self.local_artifact_cache.put_source_metadata( + self.source, self.source.cache_key, + 'meta') as f: + json.dump(meta, f, indent=4, sort_keys=True, + encoding='unicode-escape') + f.write('\n') + + def create_metadata(self, artifact_name, contents=[]): # pragma: no cover + '''Create metadata to artifact to allow it to be reproduced later. + + The metadata is represented as a dict, which later on will be + written out as a JSON file. + + ''' + + assert isinstance(self.source.repo, + morphlib.cachedrepo.CachedRepo) + meta = { + 'artifact-name': artifact_name, + 'source-name': self.source.name, + 'kind': self.source.morphology['kind'], + 'description': self.source.morphology['description'], + 'repo': self.source.repo.url, + 'repo-alias': self.source.repo_name, + 'original_ref': self.source.original_ref, + 'sha1': self.source.sha1, + 'morphology': self.source.filename, + 'cache-key': self.source.cache_key, + 'cache-id': self.source.cache_id, + 'morph-version': { + 'ref': morphlib.gitversion.ref, + 'tree': morphlib.gitversion.tree, + 'commit': morphlib.gitversion.commit, + 'version': morphlib.gitversion.version, + }, + 'contents': contents, + } + + return meta + + # Wrapper around open() to allow it to be overridden by unit tests. + def _open(self, filename, mode): # pragma: no cover + dirname = os.path.dirname(filename) + if not os.path.exists(dirname): + os.makedirs(dirname) + return open(filename, mode) + + def write_metadata(self, instdir, artifact_name, + contents=[]): # pragma: no cover + '''Write the metadata for an artifact. + + The file will be located under the ``baserock`` directory under + instdir, named after ``cache_key`` with ``.meta`` as the suffix. + It will be in JSON format. + + ''' + + meta = self.create_metadata(artifact_name, contents) + + basename = '%s.meta' % artifact_name + filename = os.path.join(instdir, 'baserock', basename) + + # Unit tests use StringIO, which in Python 2.6 isn't usable with + # the "with" statement. So we don't do it with "with". + f = self._open(filename, 'w') + json.dump(meta, f, indent=4, sort_keys=True, encoding='unicode-escape') + f.close() + + def runcmd(self, *args, **kwargs): + return self.staging_area.runcmd(*args, **kwargs) + +class ChunkBuilder(BuilderBase): + + '''Build chunk artifacts.''' + + def create_devices(self, destdir): # pragma: no cover + '''Creates device nodes if the morphology specifies them''' + morphology = self.source.morphology + perms_mask = stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO + if 'devices' in morphology and morphology['devices'] is not None: + for dev in morphology['devices']: + destfile = os.path.join(destdir, './' + dev['filename']) + mode = int(dev['permissions'], 8) & perms_mask + if dev['type'] == 'c': + mode = mode | stat.S_IFCHR + elif dev['type'] == 'b': + mode = mode | stat.S_IFBLK + else: + raise IOError('Cannot create device node %s,' + 'unrecognized device type "%s"' + % (destfile, dev['type'])) + self.app.status(msg="Creating device node %s" + % destfile) + os.mknod(destfile, mode, + os.makedev(dev['major'], dev['minor'])) + os.chown(destfile, dev['uid'], dev['gid']) + + def build_and_cache(self): # pragma: no cover + with self.build_watch('overall-build'): + + builddir, destdir = self.staging_area.chroot_open( + self.source, self.setup_mounts) + + stdout = (self.app.output + if self.app.settings['build-log-on-stdout'] else None) + + cache = self.local_artifact_cache + logpath = cache.get_source_metadata_filename( + self.source, self.source.cache_key, 'build-log') + + _, temppath = tempfile.mkstemp(dir=os.path.dirname(logpath)) + + try: + self.get_sources(builddir) + self.run_commands(builddir, destdir, temppath, stdout) + self.create_devices(destdir) + + os.rename(temppath, logpath) + except BaseException, e: + logging.error('Caught exception: %s' % str(e)) + logging.info('Cleaning up staging area') + self.staging_area.chroot_close() + if os.path.isfile(temppath): + with open(temppath) as f: + for line in f: + logging.error('OUTPUT FROM FAILED BUILD: %s' % + line.rstrip('\n')) + + os.rename(temppath, logpath) + else: + logging.error("Couldn't find build log at %s", temppath) + + self.staging_area.abort() + raise + + self.staging_area.chroot_close() + built_artifacts = self.assemble_chunk_artifacts(destdir) + + self.save_build_times() + return built_artifacts + + + def run_commands(self, builddir, destdir, + logfilepath, stdout=None): # pragma: no cover + m = self.source.morphology + bs = morphlib.buildsystem.lookup_build_system(m['build-system']) + + relative_builddir = self.staging_area.relative(builddir) + relative_destdir = self.staging_area.relative(destdir) + extra_env = { 'DESTDIR': relative_destdir } + + steps = [ + ('pre-configure', False), + ('configure', False), + ('post-configure', False), + ('pre-build', True), + ('build', True), + ('post-build', True), + ('pre-test', False), + ('test', False), + ('post-test', False), + ('pre-install', False), + ('install', False), + ('post-install', False), + ] + for step, in_parallel in steps: + with self.build_watch(step): + key = '%s-commands' % step + cmds = m[key] + if cmds: + with open(logfilepath, 'a') as log: + self.app.status(msg='Running %(key)s', key=key) + log.write('# %s\n' % step) + + for cmd in cmds: + if in_parallel: + max_jobs = self.source.morphology['max-jobs'] + if max_jobs is None: + max_jobs = self.max_jobs + extra_env['MAKEFLAGS'] = '-j%s' % max_jobs + else: + extra_env['MAKEFLAGS'] = '-j1' + + try: + with open(logfilepath, 'a') as log: + log.write('# # %s\n' % cmd) + + # flushing is needed because writes from python and + # writes from being the output in Popen have different + # buffers, but flush handles both + if stdout: + stdout.flush() + + self.runcmd(['sh', '-c', cmd], + extra_env=extra_env, + cwd=relative_builddir, + stdout=stdout or subprocess.PIPE, + stderr=subprocess.STDOUT, + logfile=logfilepath) + + if stdout: + stdout.flush() + except cliapp.AppException, e: + if not stdout: + with open(logfilepath, 'r') as log: + self.app.output.write("%s failed\n" % step) + shutil.copyfileobj(log, self.app.output) + raise e + + def write_system_integration_commands(self, destdir, + integration_commands, artifact_name): # pragma: no cover + + rel_path = SYSTEM_INTEGRATION_PATH + dest_path = os.path.join(destdir, SYSTEM_INTEGRATION_PATH) + + scripts_created = [] + + if not os.path.exists(dest_path): + os.makedirs(dest_path) + + if artifact_name in integration_commands: + prefixes_per_artifact = integration_commands[artifact_name] + for prefix, commands in prefixes_per_artifact.iteritems(): + for index, script in enumerate(commands): + script_name = "%s-%s-%04d" % (prefix, + artifact_name, + index) + script_path = os.path.join(dest_path, script_name) + + with morphlib.savefile.SaveFile(script_path, 'w') as f: + f.write("#!/bin/sh\nset -xeu\n") + f.write(script) + os.chmod(script_path, 0555) + + rel_script_path = os.path.join(SYSTEM_INTEGRATION_PATH, + script_name) + scripts_created += [rel_script_path] + + return scripts_created + + def assemble_chunk_artifacts(self, destdir): # pragma: no cover + built_artifacts = [] + filenames = [] + source = self.source + split_rules = source.split_rules + morphology = source.morphology + sys_tag = 'system-integration' + + def filepaths(destdir): + for dirname, subdirs, basenames in os.walk(destdir): + subdirsymlinks = [os.path.join(dirname, x) for x in subdirs + if os.path.islink(os.path.join(dirname, x))] + filenames = [os.path.join(dirname, x) for x in basenames] + for relpath in (os.path.relpath(x, destdir) for x in + [dirname] + subdirsymlinks + filenames): + yield relpath + + with self.build_watch('determine-splits'): + matches, overlaps, unmatched = \ + split_rules.partition(filepaths(destdir)) + + system_integration = morphology.get(sys_tag) or {} + + with self.build_watch('create-chunks'): + for chunk_artifact_name, chunk_artifact \ + in source.artifacts.iteritems(): + file_paths = matches[chunk_artifact_name] + chunk_artifact = source.artifacts[chunk_artifact_name] + + def all_parents(path): + while path != '': + yield path + path = os.path.dirname(path) + + def parentify(filenames): + names = set() + for name in filenames: + names.update(all_parents(name)) + return sorted(names) + + extra_files = self.write_system_integration_commands( + destdir, system_integration, + chunk_artifact_name) + extra_files += ['baserock/%s.meta' % chunk_artifact_name] + parented_paths = parentify(file_paths + extra_files) + + with self.local_artifact_cache.put(chunk_artifact) as f: + self.write_metadata(destdir, chunk_artifact_name, + parented_paths) + + self.app.status(msg='Creating chunk artifact %(name)s', + name=chunk_artifact_name) + morphlib.bins.create_chunk(destdir, f, parented_paths) + built_artifacts.append(chunk_artifact) + + for dirname, subdirs, files in os.walk(destdir): + if files: + raise Exception('DESTDIR %s is not empty: %s' % + (destdir, files)) + return built_artifacts + + def get_sources(self, srcdir): # pragma: no cover + s = self.source + extract_sources(self.app, self.repo_cache, s.repo, s.sha1, srcdir) + + +class StratumBuilder(BuilderBase): + '''Build stratum artifacts.''' + + def is_constituent(self, artifact): # pragma: no cover + '''True if artifact should be included in the stratum artifact''' + return (artifact.source.morphology['kind'] == 'chunk' and \ + artifact.source.build_mode != 'bootstrap') + + def build_and_cache(self): # pragma: no cover + with self.build_watch('overall-build'): + constituents = [d for d in self.source.dependencies + if self.is_constituent(d)] + + # the only reason the StratumBuilder has to download chunks is to + # check for overlap now that strata are lists of chunks + with self.build_watch('check-chunks'): + for a_name, a in self.source.artifacts.iteritems(): + # download the chunk artifact if necessary + download_depends(constituents, + self.local_artifact_cache, + self.remote_artifact_cache) + + with self.build_watch('create-chunk-list'): + lac = self.local_artifact_cache + for a_name, a in self.source.artifacts.iteritems(): + meta = self.create_metadata( + a_name, + [x.name for x in constituents]) + with lac.put_artifact_metadata(a, 'meta') as f: + json.dump(meta, f, indent=4, sort_keys=True) + with self.local_artifact_cache.put(a) as f: + json.dump([c.basename() for c in constituents], f) + self.save_build_times() + return self.source.artifacts.values() + + +class SystemBuilder(BuilderBase): # pragma: no cover + + '''Build system image artifacts.''' + + def __init__(self, *args, **kwargs): + BuilderBase.__init__(self, *args, **kwargs) + self.args = args + self.kwargs = kwargs + + def build_and_cache(self): + self.app.status(msg='Building system %(system_name)s', + system_name=self.source.name) + + with self.build_watch('overall-build'): + arch = self.source.morphology['arch'] + + for a_name, artifact in self.source.artifacts.iteritems(): + handle = self.local_artifact_cache.put(artifact) + + try: + fs_root = self.staging_area.destdir(self.source) + self.unpack_strata(fs_root) + self.write_metadata(fs_root, a_name) + self.run_system_integration_commands(fs_root) + unslashy_root = fs_root[1:] + def uproot_info(info): + info.name = relpath(info.name, unslashy_root) + if info.islnk(): + info.linkname = relpath(info.linkname, + unslashy_root) + return info + tar = tarfile.open(fileobj=handle, mode="w", name=a_name) + self.app.status(msg='Constructing tarball of rootfs', + chatty=True) + tar.add(fs_root, recursive=True, filter=uproot_info) + tar.close() + except BaseException as e: + logging.error(traceback.format_exc()) + self.app.status(msg='Error while building system', + error=True) + handle.abort() + raise + else: + handle.close() + + self.save_build_times() + return self.source.artifacts.itervalues() + + def unpack_one_stratum(self, stratum_artifact, target): + '''Unpack a single stratum into a target directory''' + + cache = self.local_artifact_cache + with cache.get(stratum_artifact) as stratum_file: + artifact_list = json.load(stratum_file, encoding='unicode-escape') + for chunk in (ArtifactCacheReference(a) for a in artifact_list): + self.app.status(msg='Unpacking chunk %(basename)s', + basename=chunk.basename(), chatty=True) + with cache.get(chunk) as chunk_file: + morphlib.bins.unpack_binary_from_file(chunk_file, target) + + target_metadata = os.path.join( + target, 'baserock', '%s.meta' % stratum_artifact.name) + with cache.get_artifact_metadata(stratum_artifact, 'meta') as meta_src: + with morphlib.savefile.SaveFile(target_metadata, 'w') as meta_dst: + shutil.copyfileobj(meta_src, meta_dst) + + def unpack_strata(self, path): + '''Unpack strata into a directory.''' + + self.app.status(msg='Unpacking strata to %(path)s', + path=path, chatty=True) + with self.build_watch('unpack-strata'): + for a_name, a in self.source.artifacts.iteritems(): + # download the stratum artifacts if necessary + download_depends(self.source.dependencies, + self.local_artifact_cache, + self.remote_artifact_cache, + ('meta',)) + + # download the chunk artifacts if necessary + for stratum_artifact in self.source.dependencies: + f = self.local_artifact_cache.get(stratum_artifact) + chunks = [ArtifactCacheReference(c) for c in json.load(f)] + download_depends(chunks, + self.local_artifact_cache, + self.remote_artifact_cache) + f.close() + + # unpack it from the local artifact cache + for stratum_artifact in self.source.dependencies: + self.unpack_one_stratum(stratum_artifact, path) + + ldconfig(self.app.runcmd, path) + + def write_metadata(self, instdir, artifact_name): + BuilderBase.write_metadata(self, instdir, artifact_name) + + os_release_file = os.path.join(instdir, 'etc', 'os-release') + dirname = os.path.dirname(os_release_file) + if not os.path.exists(dirname): + os.makedirs(dirname) + with morphlib.savefile.SaveFile(os_release_file, 'w') as f: + f.write('NAME="Baserock"\n') + f.write('ID=baserock\n') + f.write('HOME_URL="http://wiki.baserock.org"\n') + f.write('SUPPORT_URL="http://wiki.baserock.org/mailinglist"\n') + f.write('BUG_REPORT_URL="http://wiki.baserock.org/mailinglist"\n') + + os.chmod(os_release_file, 0644) + + def run_system_integration_commands(self, rootdir): # pragma: no cover + ''' Run the system integration commands ''' + + sys_integration_dir = os.path.join(rootdir, SYSTEM_INTEGRATION_PATH) + if not os.path.isdir(sys_integration_dir): + return + + env = { + 'PATH': '/bin:/usr/bin:/sbin:/usr/sbin' + } + + self.app.status(msg='Running the system integration commands', + error=True) + + mounted = [] + to_mount = ( + ('proc', 'proc', 'none'), + ('dev/shm', 'tmpfs', 'none'), + ) + + try: + for mount_point, mount_type, source in to_mount: + logging.debug('Mounting %s in system root filesystem' + % mount_point) + path = os.path.join(rootdir, mount_point) + if not os.path.exists(path): + os.makedirs(path) + morphlib.fsutils.mount(self.app.runcmd, source, path, + mount_type) + mounted.append(path) + + # The single - is just a shell convention to fill $0 when using -c, + # since ordinarily $0 contains the program name. + # -- is used to indicate the end of options for run-parts, + # we don't want SYSTEM_INTEGRATION_PATH to be interpreted + # as an option if it happens to begin with a - + self.app.runcmd(['chroot', rootdir, 'sh', '-c', + 'cd / && run-parts -- "$1"', '-', SYSTEM_INTEGRATION_PATH], + env=env) + except BaseException, e: + self.app.status( + msg='Error while running system integration commands', + error=True) + raise + finally: + for mount_path in reversed(mounted): + logging.debug('Unmounting %s in system root filesystem' + % mount_path) + morphlib.fsutils.unmount(self.app.runcmd, mount_path) + + +class Builder(object): # pragma: no cover + + '''Helper class to build with the right BuilderBase subclass.''' + + classes = { + 'chunk': ChunkBuilder, + 'stratum': StratumBuilder, + 'system': SystemBuilder, + } + + def __init__(self, app, staging_area, local_artifact_cache, + remote_artifact_cache, repo_cache, max_jobs, setup_mounts): + self.app = app + self.staging_area = staging_area + self.local_artifact_cache = local_artifact_cache + self.remote_artifact_cache = remote_artifact_cache + self.repo_cache = repo_cache + self.max_jobs = max_jobs + self.setup_mounts = setup_mounts + + def build_and_cache(self, source): + kind = source.morphology['kind'] + o = self.classes[kind](self.app, self.staging_area, + self.local_artifact_cache, + self.remote_artifact_cache, source, + self.repo_cache, self.max_jobs, + self.setup_mounts) + self.app.status(msg='Builder.build: artifact %s with %s' % + (source.name, repr(o)), + chatty=True) + built_artifacts = o.build_and_cache() + self.app.status(msg='Builder.build: done', + chatty=True) + return built_artifacts diff --git a/morphlib/builder2_tests.py b/morphlib/builder2_tests.py new file mode 100644 index 00000000..4fd0807a --- /dev/null +++ b/morphlib/builder2_tests.py @@ -0,0 +1,221 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import json +import os +import StringIO +import unittest + +import morphlib + + +class FakeBuildSystem(object): + + def __init__(self): + self.build_commands = ['buildsys-it'] + + +class FakeApp(object): + def __init__(self, runcmd=None): + self.runcmd = runcmd + + +class FakeStagingArea(object): + + def __init__(self, runcmd, build_env): + self.runcmd = runcmd + self.env = build_env.env + + +class FakeSource(object): + + def __init__(self): + self.morphology = { + 'name': 'a', + 'kind': 'b', + 'description': 'c', + } + self.name = 'a' + + self.repo = morphlib.cachedrepo.CachedRepo(FakeApp(), 'repo', + 'url', 'path') + self.repo_name = 'url' + self.original_ref = 'e' + self.sha1 = 'f' + self.filename = 'g' + + +class FakeArtifact(object): + + def __init__(self, name): + self.name = name + self.source = FakeSource() + self.cache_key = 'blahblah' + self.cache_id = {} + + +class FakeBuildEnv(object): + + def __init__(self): + self.arch = 'le-arch' + self.env = { + 'PATH': '/le-bin:/le-bon:/le-bin-bon', + } + + +class FakeFileHandle(object): + + def __init__(self, cache, key): + self._string = "" + self._cache = cache + self._key = key + + def __enter__(self): + return self + + def _writeback(self): + self._cache._cached[self._key] = self._string + + def __exit__(self, type, value, traceback): + self._writeback() + + def close(self): + self._writeback() + + def write(self, string): + self._string += string + + +class FakeArtifactCache(object): + + def __init__(self): + self._cached = {} + + def put(self, artifact): + return FakeFileHandle(self, (artifact.cache_key, artifact.name)) + + def put_artifact_metadata(self, artifact, name): + return FakeFileHandle(self, (artifact.cache_key, artifact.name, name)) + + def put_source_metadata(self, source, cachekey, name): + return FakeFileHandle(self, (cachekey, name)) + + def get(self, artifact): + return StringIO.StringIO( + self._cached[(artifact.cache_key, artifact.name)]) + + def get_artifact_metadata(self, artifact, name): + return StringIO.StringIO( + self._cached[(artifact.cache_key, artifact.name, name)]) + + def get_source_metadata(self, source, cachekey, name): + return StringIO.StringIO(self._cached[(cachekey, name)]) + + def has(self, artifact): + return (artifact.cache_key, artifact.name) in self._cached + + def has_artifact_metadata(self, artifact, name): + return (artifact.cache_key, artifact.name, name) in self._cached + + def has_source_metadata(self, source, cachekey, name): + return (cachekey, name) in self._cached + + +class BuilderBaseTests(unittest.TestCase): + + def fake_runcmd(self, argv, *args, **kwargs): + self.commands_run.append(argv) + + def fake_open(self, filename, mode): + self.open_filename = filename + self.open_handle = StringIO.StringIO() + self.open_handle.close = lambda: None + return self.open_handle + + def setUp(self): + self.commands_run = [] + self.app = FakeApp(self.fake_runcmd) + self.staging_area = FakeStagingArea(self.fake_runcmd, FakeBuildEnv()) + self.artifact_cache = FakeArtifactCache() + self.artifact = FakeArtifact('le-artifact') + self.repo_cache = None + self.build_env = FakeBuildEnv() + self.max_jobs = 1 + self.builder = morphlib.builder2.BuilderBase(self.app, + self.staging_area, + self.artifact_cache, + None, + self.artifact, + self.repo_cache, + self.max_jobs, + False) + + def test_runs_desired_command(self): + self.builder.runcmd(['foo', 'bar']) + self.assertEqual(self.commands_run, [['foo', 'bar']]) + + def test_writes_build_times(self): + with self.builder.build_watch('nothing'): + pass + self.builder.save_build_times() + self.assertTrue(self.artifact_cache.has_source_metadata( + self.artifact.source, self.artifact.cache_key, 'meta')) + + def test_watched_events_in_cache(self): + events = ["configure", "build", "install"] + for event in events: + with self.builder.build_watch(event): + pass + self.builder.save_build_times() + meta = json.load(self.artifact_cache.get_source_metadata( + self.artifact.source, self.artifact.cache_key, 'meta')) + self.assertEqual(sorted(events), + sorted(meta['build-times'].keys())) + + def test_downloads_depends(self): + lac = FakeArtifactCache() + rac = FakeArtifactCache() + afacts = [FakeArtifact(name) for name in ('a', 'b', 'c')] + for a in afacts: + fh = rac.put(a) + fh.write(a.name) + fh.close() + morphlib.builder2.download_depends(afacts, lac, rac) + self.assertTrue(all(lac.has(a) for a in afacts)) + + def test_downloads_depends_metadata(self): + lac = FakeArtifactCache() + rac = FakeArtifactCache() + afacts = [FakeArtifact(name) for name in ('a', 'b', 'c')] + for a in afacts: + fh = rac.put(a) + fh.write(a.name) + fh.close() + fh = rac.put_artifact_metadata(a, 'meta') + fh.write('metadata') + fh.close() + morphlib.builder2.download_depends(afacts, lac, rac, ('meta',)) + self.assertTrue(all(lac.has(a) for a in afacts)) + self.assertTrue(all(lac.has_artifact_metadata(a, 'meta') + for a in afacts)) + + +class ChunkBuilderTests(unittest.TestCase): + + def setUp(self): + self.app = FakeApp() + self.build = morphlib.builder2.ChunkBuilder(self.app, None, None, + None, None, None, 1, False) diff --git a/morphlib/buildsystem.py b/morphlib/buildsystem.py new file mode 100644 index 00000000..fb99e70e --- /dev/null +++ b/morphlib/buildsystem.py @@ -0,0 +1,287 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import os + +import morphlib + + +class BuildSystem(object): + + '''An abstraction of an upstream build system. + + Some build systems are well known: autotools, for example. + Others are purely manual: there's a set of commands to run that + are specific for that project, and (almost) no other project uses them. + The Linux kernel would be an example of that. + + This class provides an abstraction for these, including a method + to autodetect well known build systems. + + ''' + + def __init__(self): + self.pre_configure_commands = [] + self.configure_commands = [] + self.post_configure_commands = [] + self.pre_build_commands = [] + self.build_commands = [] + self.post_build_commands = [] + self.pre_test_commands = [] + self.test_commands = [] + self.post_test_commands = [] + self.pre_install_commands = [] + self.install_commands = [] + self.post_install_commands = [] + + def __getitem__(self, key): + key = '_'.join(key.split('-')) + return getattr(self, key) + + def get_morphology(self, name): + '''Return the text of an autodetected chunk morphology.''' + + return morphlib.morphology.Morphology({ + 'name': name, + 'kind': 'chunk', + 'build-system': self.name, + }) + + def used_by_project(self, file_list): + '''Does a project use this build system? + + ``exists`` is a function that returns a boolean telling if a + filename, relative to the project source directory, exists or not. + + ''' + raise NotImplementedError() # pragma: no cover + + +class ManualBuildSystem(BuildSystem): + + '''A manual build system where the morphology must specify all commands.''' + + name = 'manual' + + def used_by_project(self, file_list): + return False + + +class DummyBuildSystem(BuildSystem): + + '''A dummy build system, useful for debugging morphologies.''' + + name = 'dummy' + + def __init__(self): + BuildSystem.__init__(self) + self.configure_commands = ['echo dummy configure'] + self.build_commands = ['echo dummy build'] + self.test_commands = ['echo dummy test'] + self.install_commands = ['echo dummy install'] + + def used_by_project(self, file_list): + return False + + +class AutotoolsBuildSystem(BuildSystem): + + '''The automake/autoconf/libtool holy trinity.''' + + name = 'autotools' + + def __init__(self): + BuildSystem.__init__(self) + self.configure_commands = [ + 'export NOCONFIGURE=1; ' + + 'if [ -e autogen ]; then ./autogen; ' + + 'elif [ -e autogen.sh ]; then ./autogen.sh; ' + + 'elif [ ! -e ./configure ]; then autoreconf -ivf; fi', + './configure --prefix="$PREFIX"', + ] + self.build_commands = [ + 'make', + ] + self.test_commands = [ + ] + self.install_commands = [ + 'make DESTDIR="$DESTDIR" install', + ] + + def used_by_project(self, file_list): + indicators = [ + 'autogen', + 'autogen.sh', + 'configure', + 'configure.ac', + 'configure.in', + 'configure.in.in', + ] + + return any(x in file_list for x in indicators) + + +class PythonDistutilsBuildSystem(BuildSystem): + + '''The Python distutils build systems.''' + + name = 'python-distutils' + + def __init__(self): + BuildSystem.__init__(self) + self.configure_commands = [ + ] + self.build_commands = [ + 'python setup.py build', + ] + self.test_commands = [ + ] + self.install_commands = [ + 'python setup.py install --prefix "$PREFIX" --root "$DESTDIR"', + ] + + def used_by_project(self, file_list): + indicators = [ + 'setup.py', + ] + + return any(x in file_list for x in indicators) + + +class CPANBuildSystem(BuildSystem): + + '''The Perl cpan build system.''' + + name = 'cpan' + + def __init__(self): + BuildSystem.__init__(self) + self.configure_commands = [ + 'perl Makefile.PL INSTALLDIRS=perl ' + 'INSTALLARCHLIB="$PREFIX/lib/perl" ' + 'INSTALLPRIVLIB="$PREFIX/lib/perl" ' + 'INSTALLBIN="$PREFIX/bin" ' + 'INSTALLSCRIPT="$PREFIX/bin" ' + 'INSTALLMAN1DIR="$PREFIX/share/man/man1" ' + 'INSTALLMAN3DIR="$PREFIX/share/man/man3"', + ] + self.build_commands = [ + 'make', + ] + self.test_commands = [ + ] + self.install_commands = [ + 'make DESTDIR="$DESTDIR" install', + ] + + def used_by_project(self, file_list): + indicators = [ + 'Makefile.PL', + ] + + return any(x in file_list for x in indicators) + +class CMakeBuildSystem(BuildSystem): + + '''The cmake build system.''' + + name = 'cmake' + + def __init__(self): + BuildSystem.__init__(self) + self.configure_commands = [ + 'cmake -DCMAKE_INSTALL_PREFIX=/usr' + ] + self.build_commands = [ + 'make', + ] + self.test_commands = [ + ] + self.install_commands = [ + 'make DESTDIR="$DESTDIR" install', + ] + + def used_by_project(self, file_list): + indicators = [ + 'CMakeLists.txt', + ] + + return any(x in file_list for x in indicators) + +class QMakeBuildSystem(BuildSystem): + + '''The Qt build system.''' + + name = 'qmake' + + def __init__(self): + BuildSystem.__init__(self) + self.configure_commands = [ + 'qmake -makefile ' + ] + self.build_commands = [ + 'make', + ] + self.test_commands = [ + ] + self.install_commands = [ + 'make INSTALL_ROOT="$DESTDIR" install', + ] + + def used_by_project(self, file_list): + indicator = '.pro' + + for x in file_list: + if x.endswith(indicator): + return True + + return False + +build_systems = [ + ManualBuildSystem(), + AutotoolsBuildSystem(), + PythonDistutilsBuildSystem(), + CPANBuildSystem(), + CMakeBuildSystem(), + QMakeBuildSystem(), + DummyBuildSystem(), +] + + +def detect_build_system(file_list): + '''Automatically detect the build system, if possible. + + If the build system cannot be detected automatically, return None. + For ``exists`` see the ``BuildSystem.exists`` method. + + ''' + for bs in build_systems: + if bs.used_by_project(file_list): + return bs + return None + + +def lookup_build_system(name): + '''Return build system that corresponds to the name. + + If the name does not match any build system, raise ``KeyError``. + + ''' + + for bs in build_systems: + if bs.name == name: + return bs + raise KeyError('Unknown build system: %s' % name) diff --git a/morphlib/buildsystem_tests.py b/morphlib/buildsystem_tests.py new file mode 100644 index 00000000..56ba64d7 --- /dev/null +++ b/morphlib/buildsystem_tests.py @@ -0,0 +1,172 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import os +import shutil +import tempfile +import unittest + +import morphlib + + +def touch(pathname): + with open(pathname, 'w'): + pass + +manual_project = [] +autotools_project = ['configure.in'] +qmake_project = ['foo.pro'] +cmake_project = ['CMakeLists.txt'] + + +class BuildSystemTests(unittest.TestCase): + + def setUp(self): + self.bs = morphlib.buildsystem.BuildSystem() + + def test_has_configure_commands(self): + self.assertEqual(self.bs['configure-commands'], []) + + def test_has_build_commands(self): + self.assertEqual(self.bs['build-commands'], []) + + def test_has_test_commands(self): + self.assertEqual(self.bs['test-commands'], []) + + def test_has_install_commands(self): + self.assertEqual(self.bs['install-commands'], []) + + def test_returns_morphology(self): + self.bs.name = 'fake' + morph = self.bs.get_morphology('foobar') + self.assertTrue(morph.__class__.__name__ == 'Morphology') + + +class ManualBuildSystemTests(unittest.TestCase): + + def setUp(self): + self.bs = morphlib.buildsystem.ManualBuildSystem() + + def test_does_not_autodetect_empty(self): + self.assertFalse(self.bs.used_by_project(manual_project)) + + def test_does_not_autodetect_autotools(self): + self.assertFalse(self.bs.used_by_project(autotools_project)) + + def test_does_not_autodetect_qmake(self): + self.assertFalse(self.bs.used_by_project(qmake_project)) + + def test_does_not_autodetect_cmake(self): + self.assertFalse(self.bs.used_by_project(cmake_project)) + + +class DummyBuildSystemTests(unittest.TestCase): + + def setUp(self): + self.bs = morphlib.buildsystem.DummyBuildSystem() + + def test_does_not_autodetect_empty(self): + self.assertFalse(self.bs.used_by_project(manual_project)) + + def test_does_not_autodetect_autotools(self): + self.assertFalse(self.bs.used_by_project(autotools_project)) + + def test_does_not_autodetect_cmake(self): + self.assertFalse(self.bs.used_by_project(cmake_project)) + + def test_does_not_autodetect_qmake(self): + self.assertFalse(self.bs.used_by_project(qmake_project)) + + +class AutotoolsBuildSystemTests(unittest.TestCase): + + def setUp(self): + self.bs = morphlib.buildsystem.AutotoolsBuildSystem() + + def test_does_not_autodetect_empty(self): + self.assertFalse(self.bs.used_by_project(manual_project)) + + def test_autodetects_autotools(self): + self.assertTrue(self.bs.used_by_project(autotools_project)) + +class CMakeBuildSystemTests(unittest.TestCase): + + def setUp(self): + self.bs = morphlib.buildsystem.CMakeBuildSystem() + + def test_does_not_autodetect_empty(self): + self.assertFalse(self.bs.used_by_project(manual_project)) + + def test_autodetects_cmake(self): + self.assertTrue(self.bs.used_by_project(cmake_project)) + +class QMakeBuildSystemTests(unittest.TestCase): + + def setUp(self): + self.bs = morphlib.buildsystem.QMakeBuildSystem() + + def test_does_not_autodetect_empty(self): + self.assertFalse(self.bs.used_by_project(manual_project)) + + def test_autodetects_qmake(self): + self.assertTrue(self.bs.used_by_project(qmake_project)) + +class DetectBuildSystemTests(unittest.TestCase): + + def test_does_not_autodetect_manual(self): + bs = morphlib.buildsystem.detect_build_system(manual_project) + self.assertEqual(bs, None) + + def test_autodetects_autotools(self): + bs = morphlib.buildsystem.detect_build_system(autotools_project) + self.assertEqual(type(bs), morphlib.buildsystem.AutotoolsBuildSystem) + + def test_autodetects_cmake(self): + bs = morphlib.buildsystem.detect_build_system(cmake_project) + self.assertEqual(type(bs), morphlib.buildsystem.CMakeBuildSystem) + + def test_autodetects_qmake(self): + bs = morphlib.buildsystem.detect_build_system(qmake_project) + self.assertEqual(type(bs), morphlib.buildsystem.QMakeBuildSystem) + + +class LookupBuildSystemTests(unittest.TestCase): + + def lookup(self, name): + return morphlib.buildsystem.lookup_build_system(name) + + def test_raises_keyerror_for_unknown_name(self): + self.assertRaises(KeyError, self.lookup, 'unknown') + + def test_looks_up_manual(self): + self.assertEqual(type(self.lookup('manual')), + morphlib.buildsystem.ManualBuildSystem) + + def test_looks_up_autotools(self): + self.assertEqual(type(self.lookup('autotools')), + morphlib.buildsystem.AutotoolsBuildSystem) + + def test_looks_up_cmake(self): + self.assertEqual(type(self.lookup('cmake')), + morphlib.buildsystem.CMakeBuildSystem) + + def test_looks_up_qmake(self): + self.assertEqual(type(self.lookup('qmake')), + morphlib.buildsystem.QMakeBuildSystem) + + def test_looks_up_dummy(self): + self.assertEqual(type(self.lookup('dummy')), + morphlib.buildsystem.DummyBuildSystem) diff --git a/morphlib/cachedrepo.py b/morphlib/cachedrepo.py new file mode 100644 index 00000000..2fc7cfa5 --- /dev/null +++ b/morphlib/cachedrepo.py @@ -0,0 +1,308 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import logging +import os + +import morphlib + + +class InvalidReferenceError(cliapp.AppException): + + def __init__(self, repo, ref): + cliapp.AppException.__init__( + self, 'Ref %s is an invalid reference for repo %s' % (ref, repo)) + + +class UnresolvedNamedReferenceError(cliapp.AppException): + + def __init__(self, repo, ref): + cliapp.AppException.__init__( + self, 'Ref %s is not a SHA1 ref for repo %s' % (ref, repo)) + + +class CheckoutDirectoryExistsError(cliapp.AppException): + + def __init__(self, repo, target_dir): + cliapp.AppException.__init__( + self, + 'Checkout directory %s for repo %s already exists' % + (target_dir, repo)) + + +class CloneError(cliapp.AppException): + + def __init__(self, repo, target_dir): + cliapp.AppException.__init__( + self, + 'Failed to clone %s into %s' % (repo.original_name, target_dir)) + + +class CopyError(cliapp.AppException): + + def __init__(self, repo, target_dir): + cliapp.AppException.__init__( + self, + 'Failed to copy %s into %s' % (repo.original_name, target_dir)) + + +class CheckoutError(cliapp.AppException): + + def __init__(self, repo, ref, target_dir): + cliapp.AppException.__init__( + self, + 'Failed to check out ref %s in %s' % (ref, target_dir)) + + +class UpdateError(cliapp.AppException): + + def __init__(self, repo): + cliapp.AppException.__init__( + self, 'Failed to update cached version of repo %s' % repo) + + +class CachedRepo(object): + + '''A locally cached Git repository with an origin remote set up. + + On instance of this class represents a locally cached version of a + remote Git repository. This remote repository is set up as the + 'origin' remote. + + Cached repositories are bare mirrors of the upstream. Locally created + branches will be lost the next time the repository updates. + + CachedRepo objects can resolve Git refs into SHA1s. Given a SHA1 + ref, they can also be asked to return the contents of a file via the + cat() method. They can furthermore check out the repository into + a local directory using a SHA1 ref. Last but not least, any cached + repo may be updated from it's origin remote using the update() + method. + + ''' + + def __init__(self, app, original_name, url, path): + '''Creates a new CachedRepo for a repo name, URL and local path.''' + + self.app = app + self.original_name = original_name + self.url = url + self.path = path + self.is_mirror = not url.startswith('file://') + self.already_updated = False + + def ref_exists(self, ref): + '''Returns True if the given ref exists in the repo''' + + try: + self._rev_parse(ref) + except cliapp.AppException: + return False + return True + + def resolve_ref(self, ref): + '''Attempts to resolve a ref into its SHA1 and tree SHA1. + + Raises an InvalidReferenceError if the ref is not found in the + repository. + + ''' + + try: + absref = self._rev_parse(ref) + except cliapp.AppException: + raise InvalidReferenceError(self, ref) + + try: + tree = self._show_tree_hash(absref) + except cliapp.AppException: + raise InvalidReferenceError(self, ref) + + return absref, tree + + def cat(self, ref, filename): + '''Attempts to read a file given a SHA1 ref. + + Raises an UnresolvedNamedReferenceError if the ref is not a SHA1 + ref. Raises an InvalidReferenceError if the SHA1 ref is not found + in the repository. Raises an IOError if the requested file is not + found in the ref. + + ''' + + if not morphlib.git.is_valid_sha1(ref): + raise UnresolvedNamedReferenceError(self, ref) + try: + sha1 = self._rev_parse(ref) + except cliapp.AppException: + raise InvalidReferenceError(self, ref) + + try: + return self._cat_file(sha1, filename) + except cliapp.AppException: + raise IOError('File %s does not exist in ref %s of repo %s' % + (filename, ref, self)) + + def clone_checkout(self, ref, target_dir): + '''Clone from the cache into the target path and check out a given ref. + + Raises a CheckoutDirectoryExistsError if the target + directory already exists. Raises an InvalidReferenceError if the + ref is not found in the repository. Raises a CheckoutError if + something else goes wrong while copying the repository or checking + out the SHA1 ref. + + ''' + + if os.path.exists(target_dir): + raise CheckoutDirectoryExistsError(self, target_dir) + + self.resolve_ref(ref) + + self._clone_into(target_dir, ref) + + def checkout(self, ref, target_dir): + '''Unpacks the repository in a directory and checks out a commit ref. + + Raises an InvalidReferenceError if the ref is not found in the + repository. Raises a CopyError if something goes wrong with the copy + of the repository. Raises a CheckoutError if something else goes wrong + while copying the repository or checking out the SHA1 ref. + + ''' + + if not os.path.exists(target_dir): + os.mkdir(target_dir) + + # Note, we copy instead of cloning because it's much faster in the case + # that the target is on a different filesystem from the cache. We then + # take care to turn the copy into something as good as a real clone. + self._copy_repository(self.path, target_dir) + + self._checkout_ref(ref, target_dir) + + def ls_tree(self, ref): + '''Return file names found in root tree. Does not recurse to subtrees. + + Raises an UnresolvedNamedReferenceError if the ref is not a SHA1 + ref. Raises an InvalidReferenceError if the SHA1 ref is not found + in the repository. + + ''' + + if not morphlib.git.is_valid_sha1(ref): + raise UnresolvedNamedReferenceError(self, ref) + try: + sha1 = self._rev_parse(ref) + except cliapp.AppException: + raise InvalidReferenceError(self, ref) + + return self._ls_tree(sha1) + + def requires_update_for_ref(self, ref): + '''Returns False if there's no need to update this cached repo. + + If the ref points to a specific commit that's already available + locally, there's never any need to update. If it's a named ref and this + repo wasn't already updated in the lifetime of the current process, + it's necessary to update. + + ''' + if not self.is_mirror: + # Repos with file:/// URLs don't ever need updating. + return False + + if self.already_updated: + return False + + # Named refs that are valid SHA1s will confuse this code. + ref_can_change = not morphlib.git.is_valid_sha1(ref) + + if ref_can_change or not self.ref_exists(ref): + return True + else: + return False + + def update(self): + '''Updates the cached repository using its origin remote. + + Raises an UpdateError if anything goes wrong while performing + the update. + + ''' + + if not self.is_mirror: + return + + try: + self._update() + self.already_updated = True + except cliapp.AppException, e: + raise UpdateError(self) + + def _runcmd(self, *args, **kwargs): # pragma: no cover + if not 'cwd' in kwargs: + kwargs['cwd'] = self.path + return self.app.runcmd(*args, **kwargs) + + def _rev_parse(self, ref): # pragma: no cover + return morphlib.git.gitcmd(self._runcmd, 'rev-parse', '--verify', + '%s^{commit}' % ref)[0:40] + + def _show_tree_hash(self, absref): # pragma: no cover + return morphlib.git.gitcmd(self._runcmd, 'rev-parse', '--verify', + '%s^{tree}' % absref).strip() + + def _ls_tree(self, ref): # pragma: no cover + result = morphlib.git.gitcmd(self._runcmd, 'ls-tree', + '--name-only', ref) + return result.split('\n') + + def _cat_file(self, ref, filename): # pragma: no cover + return morphlib.git.gitcmd(self._runcmd, 'cat-file', 'blob', + '%s:%s' % (ref, filename)) + + def _clone_into(self, target_dir, ref): #pragma: no cover + '''Actually perform the clone''' + try: + morphlib.git.clone_into(self._runcmd, self.path, target_dir, ref) + except cliapp.AppException: + raise CloneError(self, target_dir) + + def _copy_repository(self, source_dir, target_dir): # pragma: no cover + try: + morphlib.git.copy_repository( + self._runcmd, source_dir, target_dir, self.is_mirror) + except cliapp.AppException: + raise CopyError(self, target_dir) + + def _checkout_ref(self, ref, target_dir): # pragma: no cover + try: + morphlib.git.checkout_ref(self._runcmd, target_dir, ref) + except cliapp.AppException: + raise CheckoutError(self, ref, target_dir) + + def _update(self): # pragma: no cover + try: + morphlib.git.gitcmd(self._runcmd, 'remote', 'update', + 'origin', '--prune') + except cliapp.AppException, ae: + morphlib.git.gitcmd(self._runcmd, 'remote', 'prune', 'origin') + morphlib.git.gitcmd(self._runcmd, 'remote', 'update', 'origin') + + def __str__(self): # pragma: no cover + return self.url diff --git a/morphlib/cachedrepo_tests.py b/morphlib/cachedrepo_tests.py new file mode 100644 index 00000000..d3ae331a --- /dev/null +++ b/morphlib/cachedrepo_tests.py @@ -0,0 +1,267 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import logging +import os +import unittest + +import fs.tempfs +import cliapp + +import morphlib + + +class CachedRepoTests(unittest.TestCase): + + EXAMPLE_MORPH = '''{ + "name": "foo", + "kind": "chunk" + }''' + + known_commit = 'a4da32f5a81c8bc6d660404724cedc3bc0914a75' + bad_sha1_known_to_rev_parse = 'cafecafecafecafecafecafecafecafecafecafe' + + def rev_parse(self, ref): + output = { + self.bad_sha1_known_to_rev_parse: self.bad_sha1_known_to_rev_parse, + 'a4da32f5a81c8bc6d660404724cedc3bc0914a75': + 'a4da32f5a81c8bc6d660404724cedc3bc0914a75', + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9': + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + 'master': 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + 'baserock/morph': '8b780e2e6f102fcf400ff973396566d36d730501' + } + try: + return output[ref] + except KeyError: + raise cliapp.AppException('git rev-parse --verify %s' % ref) + + def show_tree_hash(self, absref): + output = { + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9': + 'eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee', + '8b780e2e6f102fcf400ff973396566d36d730501': + 'ffffffffffffffffffffffffffffffffffffffff', + 'a4da32f5a81c8bc6d660404724cedc3bc0914a75': + 'dddddddddddddddddddddddddddddddddddddddd' + } + try: + return output[absref] + except KeyError: + raise cliapp.AppException('git log -1 --format=format:%%T %s' % + absref) + + def cat_file(self, ref, filename): + output = { + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9:foo.morph': + self.EXAMPLE_MORPH + } + try: + return output['%s:%s' % (ref, filename)] + except KeyError: + raise cliapp.AppException( + 'git cat-file blob %s:%s' % (ref, filename)) + + def copy_repository(self, source_dir, target_dir): + if target_dir.endswith('failed-checkout'): + raise morphlib.cachedrepo.CopyError(self.repo, target_dir) + + def checkout_ref(self, ref, target_dir): + if ref == 'a4da32f5a81c8bc6d660404724cedc3bc0914a75': + raise morphlib.cachedrepo.CloneError(self.repo, target_dir) + elif ref == '079bbfd447c8534e464ce5d40b80114c2022ebf4': + raise morphlib.cachedrepo.CheckoutError(self.repo, ref, target_dir) + else: + with open(os.path.join(target_dir, 'foo.morph'), 'w') as f: + f.write('contents of foo.morph') + + def ls_tree(self, ref): + output = { + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9': + ['foo.morph'] + } + try: + return output[ref] + except KeyError: + raise cliapp.AppException('git ls-tree --name-only %s' % (ref)) + + def clone_into(self, target_dir, ref): + if target_dir.endswith('failed-checkout'): + raise morphlib.cachedrepo.CloneError(self.repo, target_dir) + self.clone_target = target_dir + self.clone_ref = ref + + def update_successfully(self): + pass + + def update_with_failure(self): + raise cliapp.AppException('git remote update origin') + + def setUp(self): + self.repo_name = 'foo' + self.repo_url = 'git://foo.bar/foo.git' + self.repo_path = '/tmp/foo' + self.repo = morphlib.cachedrepo.CachedRepo( + object(), self.repo_name, self.repo_url, self.repo_path) + self.repo._rev_parse = self.rev_parse + self.repo._show_tree_hash = self.show_tree_hash + self.repo._cat_file = self.cat_file + self.repo._copy_repository = self.copy_repository + self.repo._checkout_ref = self.checkout_ref + self.repo._ls_tree = self.ls_tree + self.repo._clone_into = self.clone_into + self.tempfs = fs.tempfs.TempFS() + + def test_constructor_sets_name_and_url_and_path(self): + self.assertEqual(self.repo.original_name, self.repo_name) + self.assertEqual(self.repo.url, self.repo_url) + self.assertEqual(self.repo.path, self.repo_path) + + def test_ref_exists(self): + self.assertEqual(self.repo.ref_exists('master'), True) + + def test_ref_does_not_exist(self): + self.assertEqual(self.repo.ref_exists('non-existant-ref'), False) + + def test_resolve_named_ref_master(self): + sha1, tree = self.repo.resolve_ref('master') + self.assertEqual(sha1, 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9') + self.assertEqual(tree, 'eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee') + + def test_resolve_named_ref_baserock_morph(self): + sha1, tree = self.repo.resolve_ref('baserock/morph') + self.assertEqual(sha1, '8b780e2e6f102fcf400ff973396566d36d730501') + self.assertEqual(tree, 'ffffffffffffffffffffffffffffffffffffffff') + + def test_fail_resolving_invalid_named_ref(self): + self.assertRaises(morphlib.cachedrepo.InvalidReferenceError, + self.repo.resolve_ref, 'foo/bar') + + def test_resolve_sha1_ref(self): + sha1, tree = self.repo.resolve_ref( + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9') + self.assertEqual(sha1, 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9') + self.assertEqual(tree, 'eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee') + + def test_fail_resolving_an_invalid_sha1_ref(self): + self.assertRaises(morphlib.cachedrepo.InvalidReferenceError, + self.repo.resolve_ref, + self.bad_sha1_known_to_rev_parse) + + def test_cat_existing_file_in_existing_ref(self): + data = self.repo.cat('e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + 'foo.morph') + self.assertEqual(data, self.EXAMPLE_MORPH) + + def test_fail_cat_file_in_invalid_ref(self): + self.assertRaises( + morphlib.cachedrepo.InvalidReferenceError, self.repo.cat, + '079bbfd447c8534e464ce5d40b80114c2022ebf4', + 'doesnt-matter-whether-this-file-exists') + + def test_fail_cat_non_existent_file_in_existing_ref(self): + self.assertRaises(IOError, self.repo.cat, + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + 'file-that-does-not-exist') + + def test_fail_cat_non_existent_file_in_invalid_ref(self): + self.assertRaises( + morphlib.cachedrepo.InvalidReferenceError, self.repo.cat, + '079bbfd447c8534e464ce5d40b80114c2022ebf4', + 'file-that-does-not-exist') + + def test_fail_because_cat_in_named_ref_is_not_allowed(self): + self.assertRaises(morphlib.cachedrepo.UnresolvedNamedReferenceError, + self.repo.cat, 'master', 'doesnt-matter') + + def test_fail_clone_checkout_into_existing_directory(self): + self.assertRaises(morphlib.cachedrepo.CheckoutDirectoryExistsError, + self.repo.clone_checkout, + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + self.tempfs.root_path) + + def test_fail_checkout_due_to_clone_error(self): + self.assertRaises( + morphlib.cachedrepo.CloneError, self.repo.clone_checkout, + 'a4da32f5a81c8bc6d660404724cedc3bc0914a75', + self.tempfs.getsyspath('failed-checkout')) + + def test_fail_checkout_due_to_copy_error(self): + self.assertRaises(morphlib.cachedrepo.CopyError, self.repo.checkout, + 'a4da32f5a81c8bc6d660404724cedc3bc0914a75', + self.tempfs.getsyspath('failed-checkout')) + + def test_fail_checkout_from_invalid_ref(self): + self.assertRaises( + morphlib.cachedrepo.CheckoutError, self.repo.checkout, + '079bbfd447c8534e464ce5d40b80114c2022ebf4', + self.tempfs.getsyspath('checkout-from-invalid-ref')) + + def test_checkout_from_existing_ref_into_new_directory(self): + unpack_dir = self.tempfs.getsyspath('unpack-dir') + self.repo.checkout('e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + unpack_dir) + self.assertTrue(os.path.exists(unpack_dir)) + + morph_filename = os.path.join(unpack_dir, 'foo.morph') + self.assertTrue(os.path.exists(morph_filename)) + + def test_ls_tree_in_existing_ref(self): + data = self.repo.ls_tree('e28a23812eadf2fce6583b8819b9c5dbd36b9fb9') + self.assertEqual(data, ['foo.morph']) + + def test_fail_ls_tree_in_invalid_ref(self): + self.assertRaises( + morphlib.cachedrepo.InvalidReferenceError, self.repo.ls_tree, + '079bbfd447c8534e464ce5d40b80114c2022ebf4') + + def test_fail_because_ls_tree_in_named_ref_is_not_allowed(self): + self.assertRaises(morphlib.cachedrepo.UnresolvedNamedReferenceError, + self.repo.ls_tree, 'master') + + def test_successful_update(self): + self.repo._update = self.update_successfully + self.repo.update() + + def test_failing_update(self): + self.repo._update = self.update_with_failure + self.assertRaises(morphlib.cachedrepo.UpdateError, self.repo.update) + + def test_no_update_if_local(self): + self.repo = morphlib.cachedrepo.CachedRepo( + object(), 'local:repo', 'file:///local/repo/', '/local/repo/') + self.repo._update = self.update_with_failure + self.assertFalse(self.repo.requires_update_for_ref(self.known_commit)) + self.repo.update() + + def test_clone_checkout(self): + self.repo.clone_checkout('master', '/.DOES_NOT_EXIST') + self.assertEqual(self.clone_target, '/.DOES_NOT_EXIST') + self.assertEqual(self.clone_ref, 'master') + + def test_no_need_to_update_repo_for_existing_sha1(self): + # If the SHA1 is present locally already there's no need to update. + # If it's a named ref then it might have changed in the remote, so we + # must still update. + self.assertFalse(self.repo.requires_update_for_ref(self.known_commit)) + self.assertTrue(self.repo.requires_update_for_ref('named_ref')) + + def test_no_need_to_update_repo_if_already_updated(self): + self.repo._update = self.update_successfully + + self.assertTrue(self.repo.requires_update_for_ref('named_ref')) + self.repo.update() + self.assertFalse(self.repo.requires_update_for_ref('named_ref')) diff --git a/morphlib/cachekeycomputer.py b/morphlib/cachekeycomputer.py new file mode 100644 index 00000000..c3a01b9e --- /dev/null +++ b/morphlib/cachekeycomputer.py @@ -0,0 +1,131 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import hashlib +import logging + +import morphlib + + +class CacheKeyComputer(object): + + def __init__(self, build_env): + self._build_env = build_env + self._calculated = {} + self._hashed = {} + + def _filterenv(self, env): + keys = ["LOGNAME", "MORPH_ARCH", "TARGET", "TARGET_STAGE1", + "USER", "USERNAME"] + return dict([(k, env[k]) for k in keys]) + + def compute_key(self, source): + try: + return self._hashed[source] + except KeyError: + ret = self._hash_id(self.get_cache_id(source)) + self._hashed[source] = ret + logging.debug( + 'computed cache key %s for artifact %s from source ', + ret, (source.repo_name, source.sha1, source.filename)) + return ret + + def _hash_id(self, cache_id): + sha = hashlib.sha256() + self._hash_dict(sha, cache_id) + return sha.hexdigest() + + def _hash_thing(self, sha, thing): + if type(thing) == dict: + self._hash_dict(sha, thing) + elif type(thing) == list: + self._hash_list(sha, thing) + elif type(thing) == tuple: + self._hash_tuple(sha, thing) + else: + sha.update(str(thing)) + + def _hash_dict(self, sha, d): + for tup in sorted(d.iteritems()): + self._hash_thing(sha, tup) + + def _hash_list(self, sha, l): + for item in l: + self._hash_thing(sha, item) + + def _hash_tuple(self, sha, tup): + for item in tup: + self._hash_thing(sha, item) + + def get_cache_id(self, source): + try: + ret = self._calculated[source] + return ret + except KeyError: + cacheid = self._calculate(source) + self._calculated[source] = cacheid + return cacheid + + def _calculate(self, source): + keys = { + 'env': self._filterenv(self._build_env.env), + 'kids': [{'artifact': a.name, + 'cache-key': self.compute_key(a.source)} + for a in source.dependencies], + 'metadata-version': 1 + } + + morphology = source.morphology + kind = morphology['kind'] + if kind == 'chunk': + keys['build-mode'] = source.build_mode + keys['prefix'] = source.prefix + keys['tree'] = source.tree + keys['split-rules'] = [(a, [rgx.pattern for rgx in r._regexes]) + for (a, r) in source.split_rules] + + # Include morphology contents, since it doesn't always come + # from the source tree + keys['devices'] = morphology.get('devices') + keys['max-jobs'] = morphology.get('max-jobs') + keys['system-integration'] = morphology.get('system-integration', + {}) + # products is omitted as they are part of the split-rules + # include {pre-,,post-}{configure,build,test,install}-commands + # in morphology key + for prefix in ('pre-', '', 'post-'): + for cmdtype in ('configure', 'build', 'test', 'install'): + cmd_field = prefix + cmdtype + '-commands' + keys[cmd_field] = morphology[cmd_field] + elif kind in ('system', 'stratum'): + morph_dict = dict((k, morphology[k]) for k in morphology.keys()) + + # Disregard all fields of a morphology that aren't important + ignored_fields = ( + 'description', # purely cosmetic, doesn't change builds + # The following are used to determine dependencies, + # so are already handled by the 'kids' field. + 'strata', 'build-depends', 'chunks', + 'products') + for key in morph_dict: + if key not in ignored_fields: + keys[key] = morph_dict[key] + if kind == 'stratum': + keys['stratum-format-version'] = 1 + elif kind == 'system': + keys['system-compatibility-version'] = "2~ (upgradable, root rw)" + + return keys diff --git a/morphlib/cachekeycomputer_tests.py b/morphlib/cachekeycomputer_tests.py new file mode 100644 index 00000000..55936f94 --- /dev/null +++ b/morphlib/cachekeycomputer_tests.py @@ -0,0 +1,162 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import copy +import unittest + +import morphlib + + +class DummyBuildEnvironment: + '''Fake build environment class that doesn't need + settings to pick the environment, it just gets passed + a dict representing it + ''' + def __init__(self, env, arch): + self.env = env + + +class CacheKeyComputerTests(unittest.TestCase): + + def setUp(self): + loader = morphlib.morphloader.MorphologyLoader() + self.source_pool = morphlib.sourcepool.SourcePool() + for name, text in { + 'chunk.morph': ''' + name: chunk + kind: chunk + description: A test chunk + ''', + 'chunk2.morph': ''' + name: chunk2 + kind: chunk + description: A test chunk + ''', + 'chunk3.morph': ''' + name: chunk3 + kind: chunk + description: A test chunk + ''', + 'stratum.morph': ''' + name: stratum + kind: stratum + build-depends: [] + chunks: + - name: chunk + repo: repo + ref: original/ref + build-depends: [] + ''', + 'stratum2.morph': ''' + name: stratum2 + kind: stratum + build-depends: [] + chunks: + - name: chunk2 + repo: repo + ref: original/ref + build-depends: [] + - name: chunk3 + repo: repo + ref: original/ref + build-depends: [] + ''', + 'system.morph': ''' + name: system + kind: system + arch: testarch + strata: + - morph: stratum + - morph: stratum2 + ''', + }.iteritems(): + morph = loader.load_from_string(text) + sources = morphlib.source.make_sources('repo', 'original/ref', + name, 'sha1', + 'tree', morph) + for source in sources: + self.source_pool.add(source) + # FIXME: This should use MorphologyFactory + m = source.morphology + self.build_env = DummyBuildEnvironment({ + "LOGNAME": "foouser", + "MORPH_ARCH": "dummy", + "TARGET": "dummy-baserock-linux-gnu", + "TARGET_STAGE1": "dummy-baserock-linux-gnu", + "USER": "foouser", + "USERNAME": "foouser"}, 'dummy') + self.artifact_resolver = morphlib.artifactresolver.ArtifactResolver() + self.artifacts = self.artifact_resolver.resolve_artifacts( + self.source_pool) + self.ckc = morphlib.cachekeycomputer.CacheKeyComputer(self.build_env) + + def _find_artifact(self, name): + for artifact in self.artifacts: + if artifact.name == name: + return artifact + + def test_compute_key_hashes_all_types(self): + runcount = {'thing': 0, 'dict': 0, 'list': 0, 'tuple': 0} + + def inccount(func, name): + def f(sha, item): + runcount[name] = runcount[name] + 1 + func(sha, item) + return f + + self.ckc._hash_thing = inccount(self.ckc._hash_thing, 'thing') + self.ckc._hash_dict = inccount(self.ckc._hash_dict, 'dict') + self.ckc._hash_list = inccount(self.ckc._hash_list, 'list') + self.ckc._hash_tuple = inccount(self.ckc._hash_tuple, 'tuple') + + artifact = self._find_artifact('system-rootfs') + self.ckc.compute_key(artifact.source) + + self.assertNotEqual(runcount['thing'], 0) + self.assertNotEqual(runcount['dict'], 0) + self.assertNotEqual(runcount['list'], 0) + self.assertNotEqual(runcount['tuple'], 0) + + def _valid_sha256(self, s): + validchars = '0123456789abcdef' + return len(s) == 64 and all([c in validchars for c in s]) + + def test_compute_twice_same_key(self): + artifact = self._find_artifact('system-rootfs') + self.assertEqual(self.ckc.compute_key(artifact.source), + self.ckc.compute_key(artifact.source)) + + def test_compute_twice_same_id(self): + artifact = self._find_artifact('system-rootfs') + id1 = self.ckc.get_cache_id(artifact.source) + id2 = self.ckc.get_cache_id(artifact.source) + hash1 = self.ckc._hash_id(id1) + hash2 = self.ckc._hash_id(id2) + self.assertEqual(hash1, hash2) + + def test_compute_key_returns_sha256(self): + artifact = self._find_artifact('system-rootfs') + self.assertTrue(self._valid_sha256( + self.ckc.compute_key(artifact.source))) + + def test_different_env_gives_different_key(self): + artifact = self._find_artifact('system-rootfs') + oldsha = self.ckc.compute_key(artifact.source) + build_env = copy.deepcopy(self.build_env) + build_env.env["USER"] = "brian" + ckc = morphlib.cachekeycomputer.CacheKeyComputer(build_env) + + self.assertNotEqual(oldsha, ckc.compute_key(artifact.source)) diff --git a/morphlib/extensions.py b/morphlib/extensions.py new file mode 100644 index 00000000..af6ba279 --- /dev/null +++ b/morphlib/extensions.py @@ -0,0 +1,261 @@ +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import asyncore +import asynchat +import glob +import logging +import os +import stat +import subprocess +import tempfile + +import cliapp + +import morphlib +import sysbranchdir + + +class ExtensionError(morphlib.Error): + pass + +class ExtensionNotFoundError(ExtensionError): + pass + +class ExtensionNotExecutableError(ExtensionError): + pass + +def _get_root_repo(): + system_branch = morphlib.sysbranchdir.open_from_within('.') + root_repo_dir = morphlib.gitdir.GitDirectory( + system_branch.get_git_directory_name( + system_branch.root_repository_url)) + return root_repo_dir + +def _get_morph_extension_directory(): + code_dir = os.path.dirname(morphlib.__file__) + return os.path.join(code_dir, 'exts') + +def _list_repo_extension_filenames(kind): #pragma: no cover + repo_dir = _get_root_repo() + files = repo_dir.list_files() + return (f for f in files if os.path.splitext(f)[1] == kind) + +def _list_morph_extension_filenames(kind): + return glob.glob(os.path.join(_get_morph_extension_directory(), + '*' + kind)) + +def _get_extension_name(filename): + return os.path.basename(filename) + +def _get_repo_extension_contents(name, kind): + repo_dir = _get_root_repo() + return repo_dir.read_file(name + kind) + +def _get_morph_extension_filename(name, kind): + return os.path.join(_get_morph_extension_directory(), name + kind) + +def _is_executable(filename): + st = os.stat(filename) + mask = stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH + return (stat.S_IMODE(st.st_mode) & mask) != 0 + +def _list_extensions(kind): + repo_extension_filenames = [] + try: + repo_extension_filenames = \ + _list_repo_extension_filenames(kind) + except (sysbranchdir.NotInSystemBranch): + # Squash this and just return no system branch extensions + pass + morph_extension_filenames = _list_morph_extension_filenames(kind) + + repo_extension_names = \ + (_get_extension_name(f) for f in repo_extension_filenames) + morph_extension_names = \ + (_get_extension_name(f) for f in morph_extension_filenames) + + extension_names = set(repo_extension_names) + extension_names.update(set(morph_extension_names)) + return list(extension_names) + +def list_extensions(kind=None): + """ + List all available extensions by 'kind'. + + 'kind' should be one of '.write' or '.configure'. + If 'kind' is not provided available extensions of both + types will be returned. + + '.check' extensions are not listed here as they should + be associated with a '.write' extension of the same name. + """ + if kind: + return _list_extensions(kind) + else: + configure_extensions = _list_extensions('.configure') + write_extensions = _list_extensions('.write') + + return configure_extensions + write_extensions + +class get_extension_filename(): + """ + Find the filename of an extension by its 'name' and 'kind'. + + 'kind' should be one of '.configure', '.write' or '.check'. + + '.help' files for the extensions may also be retrieved by + passing the kind as '.write.help' or '.configure.help'. + + If the extension is in the build repository then a temporary + file will be created, which will be deleted on exting the with block. + """ + def __init__(self, name, kind, executable=True): + self.name = name + self.kind = kind + self.executable = executable + self.delete = False + + def __enter__(self): + ext_filename = None + try: + ext_contents = _get_repo_extension_contents(self.name, + self.kind) + except (IOError, cliapp.AppException, sysbranchdir.NotInSystemBranch): + # Not found: look for it in the Morph code. + ext_filename = _get_morph_extension_filename(self.name, self.kind) + if not os.path.exists(ext_filename): + raise ExtensionNotFoundError( + 'Could not find extension %s%s' % (self.name, self.kind)) + if self.executable and not _is_executable(ext_filename): + raise ExtensionNotExecutableError( + 'Extension not executable: %s' % ext_filename) + else: + # Found it in the system morphology's repository. + fd, ext_filename = tempfile.mkstemp() + os.write(fd, ext_contents) + os.close(fd) + os.chmod(ext_filename, 0700) + self.delete = True + + self.ext_filename = ext_filename + return ext_filename + + def __exit__(self, type, value, trace): + if self.delete: + os.remove(self.ext_filename) + + +class _EOFWrapper(asyncore.file_wrapper): + '''File object that reports when it hits EOF + + The async_chat class doesn't notice that its input file has hit EOF, + so if we give it one of these instead, it will mark the chatter for + closiure and ensure any in-progress buffers are flushed. + ''' + def __init__(self, dispatcher, fd): + self._dispatcher = dispatcher + asyncore.file_wrapper.__init__(self, fd) + + def recv(self, *args): + data = asyncore.file_wrapper.recv(self, *args) + if not data: + self._dispatcher.close_when_done() + # ensure any unterminated data is flushed + return self._dispatcher.get_terminator() + return data + + +class _OutputDispatcher(asynchat.async_chat, asyncore.file_dispatcher): + '''asyncore dispatcher that calls line_handler per line.''' + def __init__(self, fd, line_handler, map=None): + asynchat.async_chat.__init__(self, sock=None, map=map) + asyncore.file_dispatcher.__init__(self, fd=fd, map=map) + self.set_terminator('\n') + self._line_handler = line_handler + collect_incoming_data = asynchat.async_chat._collect_incoming_data + def set_file(self, fd): + self.socket = _EOFWrapper(self, fd) + self._fileno = self.socket.fileno() + self.add_channel() + def found_terminator(self): + self._line_handler(''.join(self.incoming)) + self.incoming = [] + +class ExtensionSubprocess(object): + + def __init__(self, report_stdout, report_stderr, report_logger): + self._report_stdout = report_stdout + self._report_stderr = report_stderr + self._report_logger = report_logger + + def run(self, filename, args, cwd, env): + '''Run an extension. + + Anything written by the extension to stdout is passed to status(), thus + normally echoed to Morph's stdout. An extra FD is passed in the + environment variable MORPH_LOG_FD, and anything written here will be + included as debug messages in Morph's log file. + + ''' + + log_read_fd, log_write_fd = os.pipe() + + try: + new_env = env.copy() + new_env['MORPH_LOG_FD'] = str(log_write_fd) + + # Because we don't have python 3.2's pass_fds, we have to + # play games with preexec_fn to close the fds we don't + # need to inherit + def close_read_end(): + os.close(log_read_fd) + p = subprocess.Popen( + [filename] + args, cwd=cwd, env=new_env, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + preexec_fn=close_read_end) + os.close(log_write_fd) + log_write_fd = None + + return self._watch_extension_subprocess(p, log_read_fd) + finally: + os.close(log_read_fd) + if log_write_fd is not None: + os.close(log_write_fd) + + def _watch_extension_subprocess(self, p, log_read_fd): + '''Follow stdout, stderr and log output of an extension subprocess.''' + + try: + socket_map = {} + for handler, fd in ((self._report_stdout, p.stdout), + (self._report_stderr, p.stderr), + (self._report_logger, log_read_fd)): + _OutputDispatcher(line_handler=handler, fd=fd, + map=socket_map) + asyncore.loop(use_poll=True, map=socket_map) + + returncode = p.wait() + assert returncode is not None + except BaseException as e: + logging.debug('Received exception %r watching extension' % e) + p.terminate() + p.wait() + raise + finally: + p.stdout.close() + p.stderr.close() + + return returncode diff --git a/morphlib/extractedtarball.py b/morphlib/extractedtarball.py new file mode 100644 index 00000000..fd98cd92 --- /dev/null +++ b/morphlib/extractedtarball.py @@ -0,0 +1,66 @@ +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import gzip +import logging +import os +import tempfile +import shutil + +import morphlib + + +class ExtractedTarball(object): # pragma: no cover + + '''Tarball extracted in a temporary directory. + + This can be used e.g. to inspect the contents of a rootfs tarball. + + ''' + def __init__(self, app, tarball): + self.app = app + self.tarball = tarball + + def setup(self): + self.app.status(msg='Preparing tarball %(tarball)s', + tarball=os.path.basename(self.tarball), chatty=True) + self.app.status(msg=' Extracting...', chatty=True) + self.tempdir = tempfile.mkdtemp(dir=self.app.settings['tempdir']) + try: + morphlib.bins.unpack_binary(self.tarball, self.tempdir) + except BaseException, e: + logging.error('Caught exception: %s' % str(e)) + logging.debug('Removing temporary directory %s' % self.tempdir) + shutil.rmtree(self.tempdir) + raise + return self.tempdir + + def cleanup(self): + self.app.status(msg='Cleanup extracted tarball %(tarball)s', + tarball=os.path.basename(self.tarball), chatty=True) + try: + shutil.rmtree(self.tempdir) + except BaseException, e: + logging.warning( + 'Error when removing temporary directory %s: %s' % + (self.tempdir, str(e))) + + def __enter__(self): + return self.setup() + + def __exit__(self, exctype, excvalue, exctraceback): + self.cleanup() diff --git a/morphlib/exts/add-config-files.configure b/morphlib/exts/add-config-files.configure new file mode 100755 index 00000000..0094cf6b --- /dev/null +++ b/morphlib/exts/add-config-files.configure @@ -0,0 +1,27 @@ +#!/bin/sh +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Copy all files located in $SRC_CONFIG_DIR to the image /etc. + + +set -e + +if [ "x${SRC_CONFIG_DIR}" != x ] +then + cp -r "$SRC_CONFIG_DIR"/* "$1/etc/" +fi + diff --git a/morphlib/exts/fstab.configure b/morphlib/exts/fstab.configure new file mode 100755 index 00000000..a1287ea4 --- /dev/null +++ b/morphlib/exts/fstab.configure @@ -0,0 +1,40 @@ +#!/usr/bin/python +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import os +import sys + + +def asciibetical(strings): + + def key(s): + return [ord(c) for c in s] + + return sorted(strings, key=key) + + +fstab_filename = os.path.join(sys.argv[1], 'etc', 'fstab') + +fstab_vars = asciibetical(x for x in os.environ if x.startswith('FSTAB_')) +with open(fstab_filename, 'a') as f: + for var in fstab_vars: + f.write('%s\n' % os.environ[var]) + +os.chown(fstab_filename, 0, 0) +os.chmod(fstab_filename, 0644) diff --git a/morphlib/exts/initramfs.write b/morphlib/exts/initramfs.write new file mode 100755 index 00000000..f8af6d84 --- /dev/null +++ b/morphlib/exts/initramfs.write @@ -0,0 +1,27 @@ +#!/bin/sh +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + +set -e + +ROOTDIR="$1" +INITRAMFS_PATH="$2" + +(cd "$ROOTDIR" && + find . -print0 | + cpio -0 -H newc -o) | + gzip -c | install -D -m644 /dev/stdin "$INITRAMFS_PATH" diff --git a/morphlib/exts/initramfs.write.help b/morphlib/exts/initramfs.write.help new file mode 100644 index 00000000..29a9d266 --- /dev/null +++ b/morphlib/exts/initramfs.write.help @@ -0,0 +1,35 @@ +help: | + Create an initramfs for a system by taking an existing system and + converting it to the appropriate format. + + The system must have a `/init` executable as the userland entry-point. + This can have a different path, if `rdinit=$path` is added to + the kernel command line. This can be added to the `rawdisk`, + `virtualbox-ssh` and `kvm` write extensions with the `KERNEL_CMDLINE` + option. + + It is possible to use a ramfs as the final rootfs without a `/init` + executable, by setting `root=/dev/mem`, or `rdinit=/sbin/init`, + but this is beyond the scope for the `initramfs.write` extension. + + The intended use of initramfs.write is to be part of a nested + deployment, so the parent system has an initramfs stored as + `/boot/initramfs.gz`. See the following example: + + name: initramfs-test + kind: cluster + systems: + - morph: minimal-system-x86_64-generic + deploy: + system: + type: rawdisk + location: initramfs-system-x86_64.img + DISK_SIZE: 1G + HOSTNAME: initramfs-system + INITRAMFS_PATH: boot/initramfs.gz + subsystems: + - morph: initramfs-x86_64 + deploy: + initramfs: + type: initramfs + location: boot/initramfs.gz diff --git a/morphlib/exts/install-files.configure b/morphlib/exts/install-files.configure new file mode 100755 index 00000000..04dc5f18 --- /dev/null +++ b/morphlib/exts/install-files.configure @@ -0,0 +1,104 @@ +#!/usr/bin/python +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +''' A Morph configuration extension for adding arbitrary files to a system + +It will read the manifest files specified in the environment variable +INSTALL_FILES, then use the contens of those files to determine which files +to install into the target system. + +''' + +import cliapp +import os +import re +import sys +import shlex +import shutil +import stat + +class InstallFilesConfigureExtension(cliapp.Application): + + def process_args(self, args): + if not 'INSTALL_FILES' in os.environ: + return + target_root = args[0] + manifests = shlex.split(os.environ['INSTALL_FILES']) + for manifest in manifests: + self.install_manifest(manifest, target_root) + + def install_manifest(self, manifest, target_root): + manifest_dir = os.path.dirname(manifest) + with open(manifest) as f: + entries = f.readlines() + for entry in entries: + self.install_entry(entry, manifest_dir, target_root) + + def install_entry(self, entry, manifest_root, target_root): + m = re.match('(overwrite )?([0-7]+) ([0-9]+) ([0-9]+) (\S+)', entry) + + if m: + overwrite = m.group(1) + mode = int(m.group(2), 8) # mode is octal + uid = int(m.group(3)) + gid = int(m.group(4)) + path = m.group(5) + else: + raise cliapp.AppException('Invalid manifest entry, ' + 'format: [overwrite] <octal mode> <uid decimal> <gid decimal> ' + '<filename>') + + dest_path = os.path.join(target_root, './' + path) + if stat.S_ISDIR(mode): + if os.path.exists(dest_path) and not overwrite: + dest_stat = os.stat(dest_path) + if (mode != dest_stat.st_mode + or uid != dest_stat.st_uid + or gid != dest_stat.st_gid): + raise cliapp.AppException('"%s" exists and is not ' + 'identical to directory ' + '"%s"' % (dest_path, entry)) + else: + os.mkdir(dest_path, mode) + os.chown(dest_path, uid, gid) + os.chmod(dest_path, mode) + + elif stat.S_ISLNK(mode): + if os.path.lexists(dest_path) and not overwrite: + raise cliapp.AppException('Symlink already exists at %s' + % dest_path) + else: + linkdest = os.readlink(os.path.join(manifest_root, + './' + path)) + os.symlink(linkdest, dest_path) + os.lchown(dest_path, uid, gid) + + elif stat.S_ISREG(mode): + if os.path.lexists(dest_path) and not overwrite: + raise cliapp.AppException('File already exists at %s' + % dest_path) + else: + shutil.copyfile(os.path.join(manifest_root, './' + path), + dest_path) + os.chown(dest_path, uid, gid) + os.chmod(dest_path, mode) + + else: + raise cliapp.AppException('Mode given in "%s" is not a file,' + ' symlink or directory' % entry) + +InstallFilesConfigureExtension().run() diff --git a/morphlib/exts/install-files.configure.help b/morphlib/exts/install-files.configure.help new file mode 100644 index 00000000..eb3aab0c --- /dev/null +++ b/morphlib/exts/install-files.configure.help @@ -0,0 +1,60 @@ +help: | + Install a set of files onto a system + + To use this extension you create a directory of files you want to install + onto the target system. + + In this example we want to copy some ssh keys onto a system + + % mkdir sshkeyfiles + % mkdir -p sshkeyfiles/root/.ssh + % cp id_rsa sshkeyfiles/root/.ssh + % cp id_rsa.pub sshkeyfiles/root/.ssh + + Now we need to create a manifest file to set the file modes + and persmissions. The manifest file should be created inside the + directory that contains the files we're trying to install. + + cat << EOF > sshkeyfiles/manifest + 0040755 0 0 /root/.ssh + 0100600 0 0 /root/.ssh/id_rsa + 0100644 0 0 /root/.ssh/id_rsa.pub + EOF + + Then we add the path to our manifest to our cluster morph, + this path should be relative to the system definitions repository. + + INSTALL_FILES: sshkeysfiles/manifest + + More generally entries in the manifest are formatted as: + [overwrite] <octal mode> <uid decimal> <gid decimal> <filename> + + NOTE: Directories on the target must be created if they do not exist. + + The extension supports files, symlinks and directories. + + For example, + + 0100644 0 0 /etc/issue + + creates a regular file at /etc/issue with 644 permissions, + uid 0 and gid 0, if the file doesn't already exist. + + overwrite 0100644 0 0 /etc/issue + + creates a regular file at /etc/issue with 644 permissions, + uid 0 and gid 0, if the file already exists it is overwritten. + + 0100755 0 0 /usr/bin/foo + + creates an executable file at /usr/bin/foo + + 0040755 0 0 /etc/foodir + + creates a directory with 755 permissions + + 0120000 0 0 /usr/bin/bar + + creates a symlink at /usr/bin/bar + + NOTE: You will still need to make a symlink in the manifest directory. diff --git a/morphlib/exts/kvm.check b/morphlib/exts/kvm.check new file mode 100755 index 00000000..1bb4007a --- /dev/null +++ b/morphlib/exts/kvm.check @@ -0,0 +1,84 @@ +#!/usr/bin/python +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +'''Preparatory checks for Morph 'kvm' write extension''' + +import cliapp +import re +import urlparse + +import morphlib.writeexts + + +class KvmPlusSshCheckExtension(morphlib.writeexts.WriteExtension): + + location_pattern = '^/(?P<guest>[^/]+)(?P<path>/.+)$' + + def process_args(self, args): + if len(args) != 1: + raise cliapp.AppException('Wrong number of command line args') + + self.require_btrfs_in_deployment_host_kernel() + + upgrade = self.get_environment_boolean('UPGRADE') + if upgrade: + raise cliapp.AppException( + 'Use the `ssh-rsync` write extension to deploy upgrades to an ' + 'existing remote system.') + + location = args[0] + ssh_host, vm_name, vm_path = self.check_and_parse_location(location) + + self.check_ssh_connectivity(ssh_host) + self.check_no_existing_libvirt_vm(ssh_host, vm_name) + self.check_extra_disks_exist(ssh_host, self.parse_attach_disks()) + + def check_and_parse_location(self, location): + '''Check and parse the location argument to get relevant data.''' + + x = urlparse.urlparse(location) + + if x.scheme != 'kvm+ssh': + raise cliapp.AppException( + 'URL schema must be kvm+ssh in %s' % location) + + m = re.match(self.location_pattern, x.path) + if not m: + raise cliapp.AppException('Cannot parse location %s' % location) + + return x.netloc, m.group('guest'), m.group('path') + + def check_no_existing_libvirt_vm(self, ssh_host, vm_name): + try: + cliapp.ssh_runcmd(ssh_host, + ['virsh', '--connect', 'qemu:///system', 'domstate', vm_name]) + except cliapp.AppException as e: + pass + else: + raise cliapp.AppException( + 'Host %s already has a VM named %s. You can use the ssh-rsync ' + 'write extension to deploy upgrades to existing machines.' % + (ssh_host, vm_name)) + + def check_extra_disks_exist(self, ssh_host, filename_list): + for filename in filename_list: + try: + cliapp.ssh_runcmd(ssh_host, ['ls', filename]) + except cliapp.AppException as e: + raise cliapp.AppException('Did not find file %s on host %s' % + (filename, ssh_host)) + +KvmPlusSshCheckExtension().run() diff --git a/morphlib/exts/kvm.write b/morphlib/exts/kvm.write new file mode 100755 index 00000000..16f188b5 --- /dev/null +++ b/morphlib/exts/kvm.write @@ -0,0 +1,138 @@ +#!/usr/bin/python +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''A Morph deployment write extension for deploying to KVM+libvirt.''' + + +import cliapp +import os +import re +import sys +import tempfile +import urlparse + +import morphlib.writeexts + + +class KvmPlusSshWriteExtension(morphlib.writeexts.WriteExtension): + + '''Create a KVM/LibVirt virtual machine during Morph's deployment. + + The location command line argument is the pathname of the disk image + to be created. The user is expected to provide the location argument + using the following syntax: + + kvm+ssh://HOST/GUEST/PATH + + where: + + * HOST is the host on which KVM/LibVirt is running + * GUEST is the name of the guest virtual machine on that host + * PATH is the path to the disk image that should be created, + on that host + + The extension will connect to HOST via ssh to run libvirt's + command line management tools. + + ''' + + location_pattern = '^/(?P<guest>[^/]+)(?P<path>/.+)$' + + def process_args(self, args): + if len(args) != 2: + raise cliapp.AppException('Wrong number of command line args') + + temp_root, location = args + ssh_host, vm_name, vm_path = self.parse_location(location) + autostart = self.get_environment_boolean('AUTOSTART') + + fd, raw_disk = tempfile.mkstemp() + os.close(fd) + self.create_local_system(temp_root, raw_disk) + + try: + self.transfer(raw_disk, ssh_host, vm_path) + self.create_libvirt_guest(ssh_host, vm_name, vm_path, autostart) + except BaseException: + sys.stderr.write('Error deploying to libvirt') + os.remove(raw_disk) + cliapp.ssh_runcmd(ssh_host, ['rm', '-f', vm_path]) + raise + else: + os.remove(raw_disk) + + self.status( + msg='Virtual machine %(vm_name)s has been created', + vm_name=vm_name) + + def parse_location(self, location): + '''Parse the location argument to get relevant data.''' + + x = urlparse.urlparse(location) + m = re.match('^/(?P<guest>[^/]+)(?P<path>/.+)$', x.path) + return x.netloc, m.group('guest'), m.group('path') + + def transfer(self, raw_disk, ssh_host, vm_path): + '''Transfer raw disk image to libvirt host.''' + + self.status(msg='Transferring disk image') + + xfer_hole_path = morphlib.util.get_data_path('xfer-hole') + recv_hole = morphlib.util.get_data('recv-hole') + + ssh_remote_cmd = [ + 'sh', '-c', recv_hole, 'dummy-argv0', 'file', vm_path + ] + + cliapp.runcmd( + ['python', xfer_hole_path, raw_disk], + ['ssh', ssh_host] + map(cliapp.shell_quote, ssh_remote_cmd), + stdout=None, stderr=None) + + def create_libvirt_guest(self, ssh_host, vm_name, vm_path, autostart): + '''Create the libvirt virtual machine.''' + + self.status(msg='Creating libvirt/kvm virtual machine') + + attach_disks = self.parse_attach_disks() + attach_opts = [] + for disk in attach_disks: + attach_opts.extend(['--disk', 'path=%s' % disk]) + + if 'NIC_CONFIG' in os.environ: + nics = os.environ['NIC_CONFIG'].split() + for nic in nics: + attach_opts.extend(['--network', nic]) + + ram_mebibytes = str(self.get_ram_size() / (1024**2)) + + vcpu_count = str(self.get_vcpu_count()) + + cmdline = ['virt-install', '--connect', 'qemu:///system', + '--import', '--name', vm_name, '--vnc', + '--ram', ram_mebibytes, '--vcpus', vcpu_count, + '--disk', 'path=%s,bus=ide' % vm_path] + attach_opts + if not autostart: + cmdline += ['--noreboot'] + cliapp.ssh_runcmd(ssh_host, cmdline) + + if autostart: + cliapp.ssh_runcmd(ssh_host, + ['virsh', '--connect', 'qemu:///system', 'autostart', vm_name]) + +KvmPlusSshWriteExtension().run() + diff --git a/morphlib/exts/kvm.write.help b/morphlib/exts/kvm.write.help new file mode 100644 index 00000000..8b5053a5 --- /dev/null +++ b/morphlib/exts/kvm.write.help @@ -0,0 +1,4 @@ +help: | + The INITRAMFS_PATH option can be used to specify the location of an + initramfs for syslinux to tell Linux to use, rather than booting + the rootfs directly. diff --git a/morphlib/exts/nfsboot.check b/morphlib/exts/nfsboot.check new file mode 100755 index 00000000..806e560a --- /dev/null +++ b/morphlib/exts/nfsboot.check @@ -0,0 +1,96 @@ +#!/usr/bin/python +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +'''Preparatory checks for Morph 'nfsboot' write extension''' + +import cliapp +import os + +import morphlib.writeexts + + +class NFSBootCheckExtension(morphlib.writeexts.WriteExtension): + + _nfsboot_root = '/srv/nfsboot' + + def process_args(self, args): + if len(args) != 1: + raise cliapp.AppException('Wrong number of command line args') + + location = args[0] + + upgrade = self.get_environment_boolean('UPGRADE') + if upgrade: + raise cliapp.AppException( + 'Upgrading is not currently supported for NFS deployments.') + + hostname = os.environ.get('HOSTNAME', None) + if hostname is None: + raise cliapp.AppException('You must specify a HOSTNAME.') + if hostname == 'baserock': + raise cliapp.AppException('It is forbidden to nfsboot a system ' + 'with hostname "%s"' % hostname) + + self.test_good_server(location) + + version_label = os.getenv('VERSION_LABEL', 'factory') + versioned_root = os.path.join(self._nfsboot_root, hostname, 'systems', + version_label) + if self.version_exists(versioned_root, location): + raise cliapp.AppException( + 'Root file system for host %s (version %s) already exists on ' + 'the NFS server %s. Deployment aborted.' % (hostname, + version_label, location)) + + def test_good_server(self, server): + self.check_ssh_connectivity(server) + + # Is an NFS server + try: + cliapp.ssh_runcmd( + 'root@%s' % server, ['test', '-e', '/etc/exports']) + except cliapp.AppException: + raise cliapp.AppException('server %s is not an nfs server' + % server) + try: + cliapp.ssh_runcmd( + 'root@%s' % server, ['systemctl', 'is-enabled', + 'nfs-server.service']) + + except cliapp.AppException: + raise cliapp.AppException('server %s does not control its ' + 'nfs server by systemd' % server) + + # TFTP server exports /srv/nfsboot/tftp + tftp_root = os.path.join(self._nfsboot_root, 'tftp') + try: + cliapp.ssh_runcmd( + 'root@%s' % server, ['test' , '-d', tftp_root]) + except cliapp.AppException: + raise cliapp.AppException('server %s does not export %s' % + (tftp_root, server)) + + def version_exists(self, versioned_root, location): + try: + cliapp.ssh_runcmd('root@%s' % location, + ['test', '-d', versioned_root]) + except cliapp.AppException: + return False + + return True + + +NFSBootCheckExtension().run() diff --git a/morphlib/exts/nfsboot.configure b/morphlib/exts/nfsboot.configure new file mode 100755 index 00000000..660d9c39 --- /dev/null +++ b/morphlib/exts/nfsboot.configure @@ -0,0 +1,31 @@ +#!/bin/sh +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Remove all networking interfaces. On nfsboot systems, eth0 is set up +# during kernel init, and the normal ifup@eth0.service systemd unit +# would break the NFS connection and cause the system to hang. + + +set -e +if [ "$NFSBOOT_CONFIGURE" ]; then + # Remove all networking interfaces but loopback + cat > "$1/etc/network/interfaces" <<EOF +auto lo +iface lo inet loopback +EOF + +fi diff --git a/morphlib/exts/nfsboot.write b/morphlib/exts/nfsboot.write new file mode 100755 index 00000000..8d3d6df7 --- /dev/null +++ b/morphlib/exts/nfsboot.write @@ -0,0 +1,194 @@ +#!/usr/bin/python +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''A Morph deployment write extension for deploying to an nfsboot server + +An nfsboot server is defined as a baserock system that has tftp and nfs +servers running, the tftp server is exporting the contents of +/srv/nfsboot/tftp/ and the user has sufficient permissions to create nfs roots +in /srv/nfsboot/nfs/ + +''' + + +import cliapp +import os +import glob + +import morphlib.writeexts + + +class NFSBootWriteExtension(morphlib.writeexts.WriteExtension): + + '''Create an NFS root and kernel on TFTP during Morph's deployment. + + The location command line argument is the hostname of the nfsboot server. + The user is expected to provide the location argument + using the following syntax: + + HOST + + where: + + * HOST is the host of the nfsboot server + + The extension will connect to root@HOST via ssh to copy the kernel and + rootfs, and configure the nfs server. + + It requires root because it uses systemd, and reads/writes to /etc. + + ''' + + _nfsboot_root = '/srv/nfsboot' + + def process_args(self, args): + if len(args) != 2: + raise cliapp.AppException('Wrong number of command line args') + + temp_root, location = args + + version_label = os.getenv('VERSION_LABEL', 'factory') + hostname = os.environ['HOSTNAME'] + + versioned_root = os.path.join(self._nfsboot_root, hostname, 'systems', + version_label) + + self.copy_rootfs(temp_root, location, versioned_root, hostname) + self.copy_kernel(temp_root, location, versioned_root, version_label, + hostname) + self.configure_nfs(location, hostname) + + def create_local_state(self, location, hostname): + statedir = os.path.join(self._nfsboot_root, hostname, 'state') + subdirs = [os.path.join(statedir, 'home'), + os.path.join(statedir, 'opt'), + os.path.join(statedir, 'srv')] + cliapp.ssh_runcmd('root@%s' % location, + ['mkdir', '-p'] + subdirs) + + def copy_kernel(self, temp_root, location, versioned_root, version, + hostname): + bootdir = os.path.join(temp_root, 'boot') + image_names = ['vmlinuz', 'zImage', 'uImage'] + for name in image_names: + try_path = os.path.join(bootdir, name) + if os.path.exists(try_path): + kernel_src = try_path + break + else: + raise cliapp.AppException( + 'Could not find a kernel in the system: none of ' + '%s found' % ', '.join(image_names)) + + kernel_dest = os.path.join(versioned_root, 'orig', 'kernel') + rsync_dest = 'root@%s:%s' % (location, kernel_dest) + self.status(msg='Copying kernel') + cliapp.runcmd( + ['rsync', '-s', kernel_src, rsync_dest]) + + # Link the kernel to the right place + self.status(msg='Creating links to kernel in tftp directory') + tftp_dir = os.path.join(self._nfsboot_root , 'tftp') + versioned_kernel_name = "%s-%s" % (hostname, version) + kernel_name = hostname + try: + cliapp.ssh_runcmd('root@%s' % location, + ['ln', '-f', kernel_dest, + os.path.join(tftp_dir, versioned_kernel_name)]) + + cliapp.ssh_runcmd('root@%s' % location, + ['ln', '-sf', versioned_kernel_name, + os.path.join(tftp_dir, kernel_name)]) + except cliapp.AppException: + raise cliapp.AppException('Could not create symlinks to the ' + 'kernel at %s in %s on %s' + % (kernel_dest, tftp_dir, location)) + + def copy_rootfs(self, temp_root, location, versioned_root, hostname): + rootfs_src = temp_root + '/' + orig_path = os.path.join(versioned_root, 'orig') + run_path = os.path.join(versioned_root, 'run') + + self.status(msg='Creating destination directories') + try: + cliapp.ssh_runcmd('root@%s' % location, + ['mkdir', '-p', orig_path, run_path]) + except cliapp.AppException: + raise cliapp.AppException('Could not create dirs %s and %s on %s' + % (orig_path, run_path, location)) + + self.status(msg='Creating \'orig\' rootfs') + cliapp.runcmd( + ['rsync', '-asXSPH', '--delete', rootfs_src, + 'root@%s:%s' % (location, orig_path)]) + + self.status(msg='Creating \'run\' rootfs') + try: + cliapp.ssh_runcmd('root@%s' % location, + ['rm', '-rf', run_path]) + cliapp.ssh_runcmd('root@%s' % location, + ['cp', '-al', orig_path, run_path]) + cliapp.ssh_runcmd('root@%s' % location, + ['rm', '-rf', os.path.join(run_path, 'etc')]) + cliapp.ssh_runcmd('root@%s' % location, + ['cp', '-a', os.path.join(orig_path, 'etc'), + os.path.join(run_path, 'etc')]) + except cliapp.AppException: + raise cliapp.AppException('Could not create \'run\' rootfs' + ' from \'orig\'') + + self.status(msg='Linking \'default\' to latest system') + try: + cliapp.ssh_runcmd('root@%s' % location, + ['ln', '-sfn', versioned_root, + os.path.join(self._nfsboot_root, hostname, 'systems', + 'default')]) + except cliapp.AppException: + raise cliapp.AppException('Could not link \'default\' to %s' + % versioned_root) + + def configure_nfs(self, location, hostname): + exported_path = os.path.join(self._nfsboot_root, hostname) + exports_path = '/etc/exports' + # If that path is not already exported: + try: + cliapp.ssh_runcmd( + 'root@%s' % location, ['grep', '-q', exported_path, + exports_path]) + except cliapp.AppException: + ip_mask = '*' + options = 'rw,no_subtree_check,no_root_squash,async' + exports_string = '%s %s(%s)\n' % (exported_path, ip_mask, options) + exports_append_sh = '''\ +set -eu +target="$1" +temp=$(mktemp) +cat "$target" > "$temp" +cat >> "$temp" +mv "$temp" "$target" +''' + cliapp.ssh_runcmd( + 'root@%s' % location, + ['sh', '-c', exports_append_sh, '--', exports_path], + feed_stdin=exports_string) + cliapp.ssh_runcmd( + 'root@%s' % location, ['systemctl', 'restart', + 'nfs-server.service']) + + +NFSBootWriteExtension().run() + diff --git a/morphlib/exts/nfsboot.write.help b/morphlib/exts/nfsboot.write.help new file mode 100644 index 00000000..598b1b23 --- /dev/null +++ b/morphlib/exts/nfsboot.write.help @@ -0,0 +1,12 @@ +help: | + Deploy a system image and kernel to an nfsboot server. + + An nfsboot server is defined as a baserock system that has + tftp and nfs servers running, the tftp server is exporting + the contents of /srv/nfsboot/tftp/ and the user has sufficient + permissions to create nfs roots in /srv/nfsboot/nfs/. + + The `location` argument is the hostname of the nfsboot server. + + The extension will connect to root@HOST via ssh to copy the + kernel and rootfs, and configure the nfs server. diff --git a/morphlib/exts/openstack.check b/morphlib/exts/openstack.check new file mode 100755 index 00000000..edc37cc1 --- /dev/null +++ b/morphlib/exts/openstack.check @@ -0,0 +1,85 @@ +#!/usr/bin/python +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +'''Preparatory checks for Morph 'openstack' write extension''' + +import cliapp +import os +import urlparse + +import morphlib.writeexts + + +class OpenStackCheckExtension(morphlib.writeexts.WriteExtension): + def process_args(self, args): + if len(args) != 1: + raise cliapp.AppException('Wrong number of command line args') + + self.require_btrfs_in_deployment_host_kernel() + + upgrade = self.get_environment_boolean('UPGRADE') + if upgrade: + raise cliapp.AppException( + 'Use the `ssh-rsync` write extension to deploy upgrades to an ' + 'existing remote system.') + + location = args[0] + self.check_location(location) + + os_params = self.get_openstack_parameters() + + self.check_openstack_parameters(location, os_params) + + def get_openstack_parameters(self): + '''Check the environment variables needed and returns all. + + The environment variables are described in the class documentation. + ''' + + keys = ('OPENSTACK_USER', 'OPENSTACK_TENANT', + 'OPENSTACK_IMAGENAME', 'OPENSTACK_PASSWORD') + for key in keys: + if key not in os.environ: + raise cliapp.AppException(key + ' was not given') + return (os.environ[key] for key in keys) + + + def check_location(self, location): + x = urlparse.urlparse(location) + if x.scheme not in ['http', 'https']: + raise cliapp.AppException('URL schema must be http or https in %s'\ + % location) + if (x.path != '/v2.0' and x.path != '/v2.0/'): + raise cliapp.AppException('API version must be v2.0 in %s'\ + % location) + + def check_openstack_parameters(self, auth_url, os_params): + '''Check OpenStack credentials using glance image-list''' + self.status(msg='Checking OpenStack credentials...') + + username, tenant_name, image_name, password = os_params + cmdline = ['glance', + '--os-username', username, + '--os-tenant-name', tenant_name, + '--os-password', password, + '--os-auth-url', auth_url, + 'image-list'] + try: + cliapp.runcmd(cmdline) + except cliapp.AppException: + raise cliapp.AppException('Wrong OpenStack credentals.') + +OpenStackCheckExtension().run() diff --git a/morphlib/exts/openstack.write b/morphlib/exts/openstack.write new file mode 100755 index 00000000..516fe367 --- /dev/null +++ b/morphlib/exts/openstack.write @@ -0,0 +1,127 @@ +#!/usr/bin/python +# Copyright (C) 2013 - 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''A Morph deployment write extension for deploying to OpenStack.''' + + +import cliapp +import os +import tempfile +import urlparse + +import morphlib.writeexts + + +class OpenStackWriteExtension(morphlib.writeexts.WriteExtension): + + '''Configure a raw disk image into an OpenStack host. + + The raw disk image is created during Morph's deployment and the + image is deployed in OpenStack using python-glanceclient. + + The location command line argument is the authentification url + of the OpenStack server using the following syntax: + + http://HOST:PORT/VERSION + + where + + * HOST is the host running OpenStack + * PORT is the port which is using OpenStack for authentifications. + * VERSION is the authentification version of OpenStack (Only v2.0 + supported) + + This extension needs in the environment the following variables: + + * OPENSTACK_USER is the username to use in the deployment. + * OPENSTACK_TENANT is the project name to use in the deployment. + * OPENSTACK_IMAGENAME is the name of the image to create. + * OPENSTACK_PASSWORD is the password of the user. + + + The extension will connect to OpenStack using python-glanceclient + to configure a raw image. + + ''' + + def process_args(self, args): + if len(args) != 2: + raise cliapp.AppException('Wrong number of command line args') + + temp_root, location = args + + os_params = self.get_openstack_parameters() + + fd, raw_disk = tempfile.mkstemp() + os.close(fd) + self.create_local_system(temp_root, raw_disk) + self.status(msg='Temporary disk image has been created at %s' + % raw_disk) + + self.set_extlinux_root_to_virtio(raw_disk) + + self.configure_openstack_image(raw_disk, location, os_params) + + def set_extlinux_root_to_virtio(self, raw_disk): + '''Re-configures extlinux to use virtio disks''' + self.status(msg='Updating extlinux.conf') + mp = self.mount(raw_disk) + try: + path = os.path.join(mp, 'extlinux.conf') + + with open(path) as f: + extlinux_conf = f.read() + + extlinux_conf = extlinux_conf.replace('root=/dev/sda', + 'root=/dev/vda') + with open(path, "w") as f: + f.write(extlinux_conf) + + finally: + self.unmount(mp) + + def get_openstack_parameters(self): + '''Get the environment variables needed. + + The environment variables are described in the class documentation. + ''' + + keys = ('OPENSTACK_USER', 'OPENSTACK_TENANT', + 'OPENSTACK_IMAGENAME', 'OPENSTACK_PASSWORD') + return (os.environ[key] for key in keys) + + def configure_openstack_image(self, raw_disk, auth_url, os_params): + '''Configure the image in OpenStack using glance-client''' + self.status(msg='Configuring OpenStack image...') + + username, tenant_name, image_name, password = os_params + cmdline = ['glance', + '--os-username', username, + '--os-tenant-name', tenant_name, + '--os-password', password, + '--os-auth-url', auth_url, + 'image-create', + '--name=%s' % image_name, + '--disk-format=raw', + '--container-format', 'bare', + '--file', raw_disk] + cliapp.runcmd(cmdline) + + self.status(msg='Image configured.') + +OpenStackWriteExtension().run() + diff --git a/morphlib/exts/rawdisk.check b/morphlib/exts/rawdisk.check new file mode 100755 index 00000000..acdc4de1 --- /dev/null +++ b/morphlib/exts/rawdisk.check @@ -0,0 +1,52 @@ +#!/usr/bin/python +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +'''Preparatory checks for Morph 'rawdisk' write extension''' + +import cliapp + +import morphlib.writeexts + +import os + + +class RawdiskCheckExtension(morphlib.writeexts.WriteExtension): + def process_args(self, args): + if len(args) != 1: + raise cliapp.AppException('Wrong number of command line args') + + self.require_btrfs_in_deployment_host_kernel() + + location = args[0] + upgrade = self.get_environment_boolean('UPGRADE') + if upgrade: + if not os.path.isfile(location): + raise cliapp.AppException( + 'Cannot upgrade %s: it is not an existing disk image' % + location) + + version_label = os.environ.get('VERSION_LABEL') + if version_label is None: + raise cliapp.AppException( + 'VERSION_LABEL was not given. It is required when ' + 'upgrading an existing system.') + else: + if os.path.exists(location): + raise cliapp.AppException( + 'Target %s already exists. Use `morph upgrade` if you ' + 'want to update an existing image.' % location) + +RawdiskCheckExtension().run() diff --git a/morphlib/exts/rawdisk.write b/morphlib/exts/rawdisk.write new file mode 100755 index 00000000..1c2c5a84 --- /dev/null +++ b/morphlib/exts/rawdisk.write @@ -0,0 +1,114 @@ +#!/usr/bin/python +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''A Morph deployment write extension for raw disk images.''' + + +import cliapp +import os +import sys +import time +import tempfile + +import morphlib.writeexts + + +class RawDiskWriteExtension(morphlib.writeexts.WriteExtension): + + '''Create a raw disk image during Morph's deployment. + + If the image already exists, it is upgraded. + + The location command line argument is the pathname of the disk image + to be created/upgraded. + + ''' + + def process_args(self, args): + if len(args) != 2: + raise cliapp.AppException('Wrong number of command line args') + + temp_root, location = args + if os.path.isfile(location): + self.upgrade_local_system(location, temp_root) + else: + try: + self.create_local_system(temp_root, location) + self.status(msg='Disk image has been created at %s' % location) + except Exception: + self.status(msg='Failure to create disk image at %s' % + location) + if os.path.exists(location): + os.remove(location) + raise + + def upgrade_local_system(self, raw_disk, temp_root): + self.complete_fstab_for_btrfs_layout(temp_root) + + mp = self.mount(raw_disk) + + version_label = self.get_version_label(mp) + self.status(msg='Updating image to a new version with label %s' % + version_label) + + version_root = os.path.join(mp, 'systems', version_label) + os.mkdir(version_root) + + old_orig = os.path.join(mp, 'systems', 'factory', 'orig') + new_orig = os.path.join(version_root, 'orig') + cliapp.runcmd( + ['btrfs', 'subvolume', 'snapshot', old_orig, new_orig]) + + cliapp.runcmd( + ['rsync', '-a', '--checksum', '--numeric-ids', '--delete', + temp_root + os.path.sep, new_orig]) + + self.create_run(version_root) + + default_path = os.path.join(mp, 'systems', 'default') + if os.path.exists(default_path): + os.remove(default_path) + else: + # we are upgrading and old system that does + # not have an updated extlinux config file + if self.bootloader_config_is_wanted(): + self.generate_bootloader_config(mp) + self.install_bootloader(mp) + os.symlink(version_label, default_path) + + if self.bootloader_config_is_wanted(): + self.install_kernel(version_root, temp_root) + + self.unmount(mp) + + def get_version_label(self, mp): + version_label = os.environ.get('VERSION_LABEL') + + if version_label is None: + self.unmount(mp) + raise cliapp.AppException('VERSION_LABEL was not given') + + if os.path.exists(os.path.join(mp, 'systems', version_label)): + self.unmount(mp) + raise cliapp.AppException('VERSION_LABEL %s already exists' + % version_label) + + return version_label + + +RawDiskWriteExtension().run() + diff --git a/morphlib/exts/rawdisk.write.help b/morphlib/exts/rawdisk.write.help new file mode 100644 index 00000000..298d441c --- /dev/null +++ b/morphlib/exts/rawdisk.write.help @@ -0,0 +1,11 @@ +help: | + Create a raw disk image during Morph's deployment. + + If the image already exists, it is upgraded. + + The `location` argument is a pathname to the image to be + created or upgraded. + + The INITRAMFS_PATH option can be used to specify the location of an + initramfs for syslinux to tell Linux to use, rather than booting + the rootfs directly. diff --git a/morphlib/exts/set-hostname.configure b/morphlib/exts/set-hostname.configure new file mode 100755 index 00000000..e44c5d56 --- /dev/null +++ b/morphlib/exts/set-hostname.configure @@ -0,0 +1,27 @@ +#!/bin/sh +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Set hostname on system from HOSTNAME. + + +set -e + +if [ -n "$HOSTNAME" ] +then + echo "$HOSTNAME" > "$1/etc/hostname" +fi + diff --git a/morphlib/exts/simple-network.configure b/morphlib/exts/simple-network.configure new file mode 100755 index 00000000..b98b202c --- /dev/null +++ b/morphlib/exts/simple-network.configure @@ -0,0 +1,143 @@ +#!/usr/bin/python +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +'''A Morph deployment configuration extension to handle /etc/network/interfaces + +This extension prepares /etc/network/interfaces with the interfaces specified +during deployment. + +If no network configuration is provided, eth0 will be configured for DHCP +with the hostname of the system. +''' + + +import os +import sys +import cliapp + +import morphlib + + +class SimpleNetworkError(morphlib.Error): + '''Errors associated with simple network setup''' + pass + + +class SimpleNetworkConfigurationExtension(cliapp.Application): + '''Configure /etc/network/interfaces + + Reading NETWORK_CONFIG, this extension sets up /etc/network/interfaces. + ''' + + def process_args(self, args): + network_config = os.environ.get( + "NETWORK_CONFIG", "lo:loopback;eth0:dhcp,hostname=$(hostname)") + + self.status(msg="Processing NETWORK_CONFIG=%(nc)s", nc=network_config) + + stanzas = self.parse_network_stanzas(network_config) + iface_file = self.generate_iface_file(stanzas) + + with open(os.path.join(args[0], "etc/network/interfaces"), "w") as f: + f.write(iface_file) + + def generate_iface_file(self, stanzas): + """Generate an interfaces file from the provided stanzas. + + The interfaces will be sorted by name, with loopback sorted first. + """ + + def cmp_iface_names(a, b): + a = a['name'] + b = b['name'] + if a == "lo": + return -1 + elif b == "lo": + return 1 + else: + return cmp(a,b) + + return "\n".join(self.generate_iface_stanza(stanza) + for stanza in sorted(stanzas, cmp=cmp_iface_names)) + + def generate_iface_stanza(self, stanza): + """Generate an interfaces stanza from the provided data.""" + + name = stanza['name'] + itype = stanza['type'] + lines = ["auto %s" % name, "iface %s inet %s" % (name, itype)] + lines += [" %s %s" % elem for elem in stanza['args'].items()] + lines += [""] + return "\n".join(lines) + + + def parse_network_stanzas(self, config): + """Parse a network config environment variable into stanzas. + + Network config stanzas are semi-colon separated. + """ + + return [self.parse_network_stanza(s) for s in config.split(";")] + + def parse_network_stanza(self, stanza): + """Parse a network config stanza into name, type and arguments. + + Each stanza is of the form name:type[,arg=value]... + + For example: + lo:loopback + eth0:dhcp + eth1:static,address=10.0.0.1,netmask=255.255.0.0 + """ + elements = stanza.split(",") + lead = elements.pop(0).split(":") + if len(lead) != 2: + raise SimpleNetworkError("Stanza '%s' is missing its type" % + stanza) + iface = lead[0] + iface_type = lead[1] + + if iface_type not in ['loopback', 'static', 'dhcp']: + raise SimpleNetworkError("Stanza '%s' has unknown interface type" + " '%s'" % (stanza, iface_type)) + + argpairs = [element.split("=", 1) for element in elements] + output_stanza = { "name": iface, + "type": iface_type, + "args": {} } + for argpair in argpairs: + if len(argpair) != 2: + raise SimpleNetworkError("Stanza '%s' has bad argument '%r'" + % (stanza, argpair.pop(0))) + if argpair[0] in output_stanza["args"]: + raise SimpleNetworkError("Stanza '%s' has repeated argument" + " %s" % (stanza, argpair[0])) + output_stanza["args"][argpair[0]] = argpair[1] + + return output_stanza + + def status(self, **kwargs): + '''Provide status output. + + The ``msg`` keyword argument is the actual message, + the rest are values for fields in the message as interpolated + by %. + + ''' + + self.output.write('%s\n' % (kwargs['msg'] % kwargs)) + +SimpleNetworkConfigurationExtension().run() diff --git a/morphlib/exts/ssh-rsync.check b/morphlib/exts/ssh-rsync.check new file mode 100755 index 00000000..6a776ce9 --- /dev/null +++ b/morphlib/exts/ssh-rsync.check @@ -0,0 +1,60 @@ +#!/usr/bin/python +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +'''Preparatory checks for Morph 'ssh-rsync' write extension''' + +import cliapp + +import morphlib.writeexts + + +class SshRsyncCheckExtension(morphlib.writeexts.WriteExtension): + def process_args(self, args): + if len(args) != 1: + raise cliapp.AppException('Wrong number of command line args') + + upgrade = self.get_environment_boolean('UPGRADE') + if not upgrade: + raise cliapp.AppException( + 'The ssh-rsync write is for upgrading existing remote ' + 'Baserock machines. It cannot be used for an initial ' + 'deployment.') + + location = args[0] + self.check_ssh_connectivity(location) + self.check_is_baserock_system(location) + + # The new system that being deployed as an upgrade must contain + # baserock-system-config-sync and system-version-manager. However, the + # old system simply needs to have SSH and rsync. + self.check_command_exists(location, 'rsync') + + def check_is_baserock_system(self, location): + output = cliapp.ssh_runcmd(location, ['sh', '-c', + 'test -d /baserock || echo -n dirnotfound']) + if output == 'dirnotfound': + raise cliapp.AppException('%s is not a baserock system' + % location) + + def check_command_exists(self, location, command): + test = 'type %s > /dev/null 2>&1 || echo -n cmdnotfound' % command + output = cliapp.ssh_runcmd(location, ['sh', '-c', test]) + if output == 'cmdnotfound': + raise cliapp.AppException( + "%s does not have %s" % (location, command)) + + +SshRsyncCheckExtension().run() diff --git a/morphlib/exts/ssh-rsync.write b/morphlib/exts/ssh-rsync.write new file mode 100755 index 00000000..c139b6c0 --- /dev/null +++ b/morphlib/exts/ssh-rsync.write @@ -0,0 +1,148 @@ +#!/usr/bin/python +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''A Morph deployment write extension for upgrading systems over ssh.''' + + +import cliapp +import os +import sys +import time +import tempfile + +import morphlib.writeexts + + +def ssh_runcmd_ignore_failure(location, command, **kwargs): + try: + return cliapp.ssh_runcmd(location, command, **kwargs) + except cliapp.AppException: + pass + + +class SshRsyncWriteExtension(morphlib.writeexts.WriteExtension): + + '''Upgrade a running baserock system with ssh and rsync. + + It assumes the system is baserock-based and has a btrfs partition. + + The location command line argument is the 'user@hostname' string + that will be passed to ssh and rsync + + ''' + + def process_args(self, args): + if len(args) != 2: + raise cliapp.AppException('Wrong number of command line args') + + temp_root, location = args + + self.upgrade_remote_system(location, temp_root) + + def upgrade_remote_system(self, location, temp_root): + self.complete_fstab_for_btrfs_layout(temp_root) + + root_disk = self.find_root_disk(location) + version_label = os.environ.get('VERSION_LABEL') + autostart = self.get_environment_boolean('AUTOSTART') + + self.status(msg='Creating remote mount point') + remote_mnt = cliapp.ssh_runcmd(location, ['mktemp', '-d']).strip() + try: + self.status(msg='Mounting root disk') + cliapp.ssh_runcmd(location, ['mount', root_disk, remote_mnt]) + except Exception as e: + ssh_runcmd_ignore_failure(location, ['rmdir', remote_mnt]) + raise e + + try: + version_root = os.path.join(remote_mnt, 'systems', version_label) + orig_dir = os.path.join(version_root, 'orig') + + self.status(msg='Creating %s' % version_root) + cliapp.ssh_runcmd(location, ['mkdir', version_root]) + + self.create_remote_orig(location, version_root, remote_mnt, + temp_root) + + # Use the system-version-manager from the new system we just + # installed, so that we can upgrade from systems that don't have + # it installed. + self.status(msg='Calling system-version-manager to deploy upgrade') + deployment = os.path.join('/systems', version_label, 'orig') + system_config_sync = os.path.join( + remote_mnt, 'systems', version_label, 'orig', 'usr', 'bin', + 'baserock-system-config-sync') + system_version_manager = os.path.join( + remote_mnt, 'systems', version_label, 'orig', 'usr', 'bin', + 'system-version-manager') + cliapp.ssh_runcmd(location, + ['env', 'BASEROCK_SYSTEM_CONFIG_SYNC='+system_config_sync, + system_version_manager, 'deploy', deployment]) + + self.status(msg='Setting %s as the new default system' % + version_label) + cliapp.ssh_runcmd(location, + [system_version_manager, 'set-default', version_label]) + except Exception as e: + self.status(msg='Deployment failed') + ssh_runcmd_ignore_failure( + location, ['btrfs', 'subvolume', 'delete', orig_dir]) + ssh_runcmd_ignore_failure( + location, ['rm', '-rf', version_root]) + raise e + finally: + self.status(msg='Removing temporary mounts') + cliapp.ssh_runcmd(location, ['umount', remote_mnt]) + cliapp.ssh_runcmd(location, ['rmdir', remote_mnt]) + + if autostart: + self.status(msg="Rebooting into new system ...") + ssh_runcmd_ignore_failure(location, ['reboot']) + + def create_remote_orig(self, location, version_root, remote_mnt, + temp_root): + '''Create the subvolume version_root/orig on location''' + + self.status(msg='Creating "orig" subvolume') + old_orig = self.get_old_orig(location, remote_mnt) + new_orig = os.path.join(version_root, 'orig') + cliapp.ssh_runcmd(location, ['btrfs', 'subvolume', 'snapshot', + old_orig, new_orig]) + + cliapp.runcmd(['rsync', '-as', '--checksum', '--numeric-ids', + '--delete', temp_root + os.path.sep, + '%s:%s' % (location, new_orig)]) + + def get_old_orig(self, location, remote_mnt): + '''Identify which subvolume to snapshot from''' + + # rawdisk upgrades use 'factory' + return os.path.join(remote_mnt, 'systems', 'factory', 'orig') + + def find_root_disk(self, location): + '''Read /proc/mounts on location to find which device contains "/"''' + + self.status(msg='Finding device that contains "/"') + contents = cliapp.ssh_runcmd(location, ['cat', '/proc/mounts']) + for line in contents.splitlines(): + line_words = line.split() + if (line_words[1] == '/' and line_words[0] != 'rootfs'): + return line_words[0] + + +SshRsyncWriteExtension().run() diff --git a/morphlib/exts/sysroot.write b/morphlib/exts/sysroot.write new file mode 100755 index 00000000..1ae4864f --- /dev/null +++ b/morphlib/exts/sysroot.write @@ -0,0 +1,29 @@ +#!/bin/sh +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# A Morph write extension to deploy to another directory + +set -eu + +# Ensure the target is an empty directory +mkdir -p "$2" +find "$2" -mindepth 1 -delete + +# Move the contents of our source directory to our target +# Previously we would (cd "$1" && find -print0 | cpio -0pumd "$absolute_path") +# to do this, but the source directory is disposable anyway, so we can move +# its contents to save time +find "$1" -maxdepth 1 -mindepth 1 -exec mv {} "$2/." + diff --git a/morphlib/exts/tar.check b/morphlib/exts/tar.check new file mode 100755 index 00000000..cbeaf163 --- /dev/null +++ b/morphlib/exts/tar.check @@ -0,0 +1,24 @@ +#!/bin/sh +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Preparatory checks for Morph 'tar' write extension + +set -eu + +if [ "$UPGRADE" == "yes" ]; then + echo >&2 "ERROR: Cannot upgrade a tar file deployment." + exit 1 +fi diff --git a/morphlib/exts/tar.write b/morphlib/exts/tar.write new file mode 100755 index 00000000..333626b5 --- /dev/null +++ b/morphlib/exts/tar.write @@ -0,0 +1,21 @@ +#!/bin/sh +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# A Morph write extension to deploy to a .tar file + +set -eu + +tar -C "$1" -cf "$2" . diff --git a/morphlib/exts/tar.write.help b/morphlib/exts/tar.write.help new file mode 100644 index 00000000..f052ac03 --- /dev/null +++ b/morphlib/exts/tar.write.help @@ -0,0 +1,5 @@ +help: | + Create a .tar file of the deployed system. + + The `location` argument is a pathname to the .tar file to be + created. diff --git a/morphlib/exts/vdaboot.configure b/morphlib/exts/vdaboot.configure new file mode 100755 index 00000000..b88eb3a8 --- /dev/null +++ b/morphlib/exts/vdaboot.configure @@ -0,0 +1,34 @@ +#!/bin/sh +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Change the "/" mount point to /dev/vda to use virtio disks. + +set -e + +if [ "$OPENSTACK_USER" ] +then + # Modifying fstab + if [ -f "$1/etc/fstab" ] + then + mv "$1/etc/fstab" "$1/etc/fstab.old" + awk 'BEGIN {print "/dev/vda / btrfs defaults,rw,noatime 0 1"}; + $2 != "/" {print $0 };' "$1/etc/fstab.old" > "$1/etc/fstab" + rm "$1/etc/fstab.old" + else + echo "/dev/vda / btrfs defaults,rw,noatime 0 1"> "$1/etc/fstab" + fi +fi diff --git a/morphlib/exts/virtualbox-ssh.check b/morphlib/exts/virtualbox-ssh.check new file mode 100755 index 00000000..57d54db1 --- /dev/null +++ b/morphlib/exts/virtualbox-ssh.check @@ -0,0 +1,37 @@ +#!/usr/bin/python +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +'''Preparatory checks for Morph 'virtualbox-ssh' write extension''' + +import cliapp + +import morphlib.writeexts + + +class VirtualBoxPlusSshCheckExtension(morphlib.writeexts.WriteExtension): + def process_args(self, args): + if len(args) != 1: + raise cliapp.AppException('Wrong number of command line args') + + self.require_btrfs_in_deployment_host_kernel() + + upgrade = self.get_environment_boolean('UPGRADE') + if upgrade: + raise cliapp.AppException( + 'Use the `ssh-rsync` write extension to deploy upgrades to an ' + 'existing remote system.') + +VirtualBoxPlusSshCheckExtension().run() diff --git a/morphlib/exts/virtualbox-ssh.write b/morphlib/exts/virtualbox-ssh.write new file mode 100755 index 00000000..39ea8f86 --- /dev/null +++ b/morphlib/exts/virtualbox-ssh.write @@ -0,0 +1,245 @@ +#!/usr/bin/python +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''A Morph deployment write extension for deploying to VirtualBox via ssh. + +VirtualBox is assumed to be running on a remote machine, which is +accessed over ssh. The machine gets created, but not started. + +''' + + +import cliapp +import os +import re +import sys +import time +import tempfile +import urlparse + +import morphlib.writeexts + + +class VirtualBoxPlusSshWriteExtension(morphlib.writeexts.WriteExtension): + + '''Create a VirtualBox virtual machine during Morph's deployment. + + The location command line argument is the pathname of the disk image + to be created. The user is expected to provide the location argument + using the following syntax: + + vbox+ssh://HOST/GUEST/PATH + + where: + + * HOST is the host on which VirtualBox is running + * GUEST is the name of the guest virtual machine on that host + * PATH is the path to the disk image that should be created, + on that host + + The extension will connect to HOST via ssh to run VirtualBox's + command line management tools. + + ''' + + def process_args(self, args): + if len(args) != 2: + raise cliapp.AppException('Wrong number of command line args') + + temp_root, location = args + ssh_host, vm_name, vdi_path = self.parse_location(location) + autostart = self.get_environment_boolean('AUTOSTART') + + vagrant = self.get_environment_boolean('VAGRANT') + + fd, raw_disk = tempfile.mkstemp() + os.close(fd) + self.create_local_system(temp_root, raw_disk) + + try: + self.transfer_and_convert_to_vdi( + raw_disk, ssh_host, vdi_path) + self.create_virtualbox_guest(ssh_host, vm_name, vdi_path, + autostart, vagrant) + except BaseException: + sys.stderr.write('Error deploying to VirtualBox') + os.remove(raw_disk) + cliapp.ssh_runcmd(ssh_host, ['rm', '-f', vdi_path]) + raise + else: + os.remove(raw_disk) + self.status( + msg='Virtual machine %(vm_name)s has been created', + vm_name=vm_name) + + def parse_location(self, location): + '''Parse the location argument to get relevant data.''' + + x = urlparse.urlparse(location) + if x.scheme != 'vbox+ssh': + raise cliapp.AppException( + 'URL schema must be vbox+ssh in %s' % location) + m = re.match('^/(?P<guest>[^/]+)(?P<path>/.+)$', x.path) + if not m: + raise cliapp.AppException('Cannot parse location %s' % location) + return x.netloc, m.group('guest'), m.group('path') + + def transfer_and_convert_to_vdi(self, raw_disk, ssh_host, vdi_path): + '''Transfer raw disk image to VirtualBox host, and convert to VDI.''' + + self.status(msg='Transfer disk and convert to VDI') + + st = os.lstat(raw_disk) + xfer_hole_path = morphlib.util.get_data_path('xfer-hole') + recv_hole = morphlib.util.get_data('recv-hole') + + ssh_remote_cmd = [ + 'sh', '-c', recv_hole, + 'dummy-argv0', 'vbox', vdi_path, str(st.st_size), + ] + + cliapp.runcmd( + ['python', xfer_hole_path, raw_disk], + ['ssh', ssh_host] + map(cliapp.shell_quote, ssh_remote_cmd), + stdout=None, stderr=None) + + def virtualbox_version(self, ssh_host): + 'Get the version number of the VirtualBox running on the remote host.' + + # --version gives a build id, which looks something like + # 1.2.3r456789, so we need to strip the suffix off and get a tuple + # of the (major, minor, patch) version, since comparing with a + # tuple is more reliable than a string and more convenient than + # comparing against the major, minor and patch numbers directly + self.status(msg='Checking version of remote VirtualBox') + build_id = cliapp.ssh_runcmd(ssh_host, ['VBoxManage', '--version']) + version_string = re.match(r"^([0-9\.]+).*$", build_id.strip()).group(1) + return tuple(int(s or '0') for s in version_string.split('.')) + + def create_virtualbox_guest(self, ssh_host, vm_name, vdi_path, autostart, + vagrant): + '''Create the VirtualBox virtual machine.''' + + self.status(msg='Create VirtualBox virtual machine') + + ram_mebibytes = str(self.get_ram_size() / (1024**2)) + + vcpu_count = str(self.get_vcpu_count()) + + if not vagrant: + hostonly_iface = self.get_host_interface(ssh_host) + + if self.virtualbox_version(ssh_host) < (4, 3, 0): + sataportcount_option = '--sataportcount' + else: + sataportcount_option = '--portcount' + + commands = [ + ['createvm', '--name', vm_name, '--ostype', 'Linux26_64', + '--register'], + ['modifyvm', vm_name, '--ioapic', 'on', + '--memory', ram_mebibytes, '--cpus', vcpu_count], + ['storagectl', vm_name, '--name', 'SATA Controller', + '--add', 'sata', '--bootable', 'on', sataportcount_option, '2'], + ['storageattach', vm_name, '--storagectl', 'SATA Controller', + '--port', '0', '--device', '0', '--type', 'hdd', '--medium', + vdi_path], + ] + if vagrant: + commands[1].extend(['--nic1', 'nat', + '--natnet1', 'default']) + else: + commands[1].extend(['--nic1', 'hostonly', + '--hostonlyadapter1', hostonly_iface, + '--nic2', 'nat', '--natnet2', 'default']) + + attach_disks = self.parse_attach_disks() + for device_no, disk in enumerate(attach_disks, 1): + cmd = ['storageattach', vm_name, + '--storagectl', 'SATA Controller', + '--port', str(device_no), + '--device', '0', + '--type', 'hdd', + '--medium', disk] + commands.append(cmd) + + if autostart: + commands.append(['startvm', vm_name]) + + for command in commands: + argv = ['VBoxManage'] + command + cliapp.ssh_runcmd(ssh_host, argv) + + def get_host_interface(self, ssh_host): + host_ipaddr = os.environ.get('HOST_IPADDR') + netmask = os.environ.get('NETMASK') + network_config = os.environ.get("NETWORK_CONFIG") + + if network_config is None: + raise cliapp.AppException('NETWORK_CONFIG was not given') + + if "eth0:" not in network_config: + raise cliapp.AppException( + 'NETWORK_CONFIG does not contain ' + 'the eth0 configuration') + + if "eth1:" not in network_config: + raise cliapp.AppException( + 'NETWORK_CONFIG does not contain ' + 'the eth1 configuration') + + if host_ipaddr is None: + raise cliapp.AppException('HOST_IPADDR was not given') + + if netmask is None: + raise cliapp.AppException('NETMASK was not given') + + # 'VBoxManage list hostonlyifs' retrieves a list with the hostonly + # interfaces on the host. For each interface, the following lines + # are shown on top: + # + # Name: vboxnet0 + # GUID: 786f6276-656e-4074-8000-0a0027000000 + # Dhcp: Disabled + # IPAddress: 192.168.100.1 + # + # The following command tries to retrieve the hostonly interface + # name (e.g. vboxnet0) associated with the given ip address. + iface = None + lines = cliapp.ssh_runcmd(ssh_host, + ['VBoxManage', 'list', 'hostonlyifs']).splitlines() + for i, v in enumerate(lines): + if host_ipaddr in v: + iface = lines[i-3].split()[1] + break + + if iface is None: + iface = cliapp.ssh_runcmd(ssh_host, + ['VBoxManage', 'hostonlyif', 'create']) + # 'VBoxManage hostonlyif create' shows the name of the + # created hostonly interface inside single quotes + iface = iface[iface.find("'") + 1 : iface.rfind("'")] + cliapp.ssh_runcmd(ssh_host, + ['VBoxManage', 'hostonlyif', + 'ipconfig', iface, + '--ip', host_ipaddr, + '--netmask', netmask]) + + return iface + +VirtualBoxPlusSshWriteExtension().run() + diff --git a/morphlib/exts/virtualbox-ssh.write.help b/morphlib/exts/virtualbox-ssh.write.help new file mode 100644 index 00000000..8b5053a5 --- /dev/null +++ b/morphlib/exts/virtualbox-ssh.write.help @@ -0,0 +1,4 @@ +help: | + The INITRAMFS_PATH option can be used to specify the location of an + initramfs for syslinux to tell Linux to use, rather than booting + the rootfs directly. diff --git a/morphlib/fsutils.py b/morphlib/fsutils.py new file mode 100644 index 00000000..751f73f6 --- /dev/null +++ b/morphlib/fsutils.py @@ -0,0 +1,139 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import os +import re + + +def create_image(runcmd, image_name, size): # pragma: no cover + # FIXME a pure python implementation may be better + runcmd(['dd', 'if=/dev/zero', 'of=' + image_name, 'bs=1', + 'seek=%d' % size, 'count=0']) + + +def partition_image(runcmd, image_name): # pragma: no cover + # FIXME make this more flexible with partitioning options + runcmd(['sfdisk', image_name], feed_stdin='1,,83,*\n') + + +def setup_device_mapping(runcmd, image_name): # pragma: no cover + findstart = re.compile(r"start=\s+(\d+),") + out = runcmd(['sfdisk', '-d', image_name]) + for line in out.splitlines(): + match = findstart.search(line) + if match is None: + continue + start = int(match.group(1)) * 512 + if start != 0: + break + + device = runcmd(['losetup', '--show', '-o', str(start), '-f', image_name]) + return device.strip() + + +def create_fs(runcmd, partition): # pragma: no cover + runcmd(['mkfs.btrfs', '-L', 'baserock', partition]) + + +def mount(runcmd, partition, mount_point, fstype=None): # pragma: no cover + if not os.path.exists(mount_point): + os.mkdir(mount_point) + if not fstype: + fstype = [] + else: + fstype = ['-t', fstype] + runcmd(['mount', partition, mount_point] + fstype) + + +def unmount(runcmd, mount_point): # pragma: no cover + runcmd(['umount', mount_point]) + + +def undo_device_mapping(runcmd, image_name): # pragma: no cover + out = runcmd(['losetup', '-j', image_name]) + for line in out.splitlines(): + i = line.find(':') + device = line[:i] + runcmd(['losetup', '-d', device]) + + +def invert_paths(tree_walker, paths): + '''List paths from `tree_walker` that are not in `paths`. + + Given a traversal of a tree and a set of paths separated by os.sep, + return the files and directories that are not part of the set of + paths, culling directories that do not need to be recursed into, + if the traversal supports this. + + `tree_walker` is expected to follow similar behaviour to `os.walk()`. + + This function will remove directores from the ones listed, to avoid + traversing into these subdirectories, if it doesn't need to. + + As such, if a directory is returned, it is implied that its contents + are also not in the set of paths. + + If the tree walker does not support culling the traversal this way, + such as `os.walk(root, topdown=False)`, then the contents will also + be returned. + + The purpose for this is to list the directories that can be made + read-only, such that it would leave everything in paths writable. + + Each path in `paths` is expected to begin with the same path as + yielded by the tree walker. + + ''' + + def is_subpath(prefix, path): + prefix_components = prefix.split(os.sep) + path_components = path.split(os.sep) + return path_components[:len(prefix_components)] == prefix_components + + for dirpath, dirnames, filenames in tree_walker: + + dn_copy = list(dirnames) + for subdir in dn_copy: + subdirpath = os.path.join(dirpath, subdir) + + if any(p == subdirpath for p in paths): + # Subdir is an exact match for a path + # Don't recurse into it, so remove from list + # Don't yield it, since we don't return listed paths + dirnames.remove(subdir) + elif any(is_subpath(subdirpath, p) for p in paths): + # This directory is a parent directory of one + # of our paths + # Recurse into it, so don't remove it from the list + # Don't yield it, since we don't return listed paths + pass + else: + # This directory is neither one marked for writing, + # nor a parent of a file marked for writing + # Don't recurse, so remove it from the list + # Yield it, since we return listed paths + dirnames.remove(subdir) + yield subdirpath + + for filename in filenames: + fullpath = os.path.join(dirpath, filename) + if any(is_subpath(p, fullpath) for p in paths): + # The file path is a child of one of the paths + # or is equal. + # Don't yield because either it is one of the specified + # paths, or is a file in a directory specified by a path + pass + else: + yield fullpath diff --git a/morphlib/fsutils_tests.py b/morphlib/fsutils_tests.py new file mode 100644 index 00000000..7b159665 --- /dev/null +++ b/morphlib/fsutils_tests.py @@ -0,0 +1,99 @@ +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import os +import unittest + +import morphlib + + +def dummy_top_down_walker(root, treedict): + '''Something that imitates os.walk, but with a dict''' + + subdirs = [k for k in treedict if isinstance(treedict[k], dict)] + files = [k for k in treedict if not isinstance(treedict[k], dict)] + yield root, subdirs, files + for subdir in subdirs: + subwalker = dummy_top_down_walker(os.path.join(root, subdir), + treedict[subdir]) + for result in subwalker: + yield result + + +class InvertPathsTests(unittest.TestCase): + + def setUp(self): + self.flat_tree = {"foo": None, "bar": None, "baz": None} + self.nested_tree = { + "foo": { + "bar": None, + "baz": None, + }, + "fs": { + "btrfs": None, + "ext2": None, + "ext3": None, + "ext4": None, + "nfs": None, + }, + } + + def test_flat_lists_single_files(self): + walker = dummy_top_down_walker('.', self.flat_tree) + self.assertEqual(sorted(["./foo", "./bar", "./baz"]), + sorted(morphlib.fsutils.invert_paths(walker, []))) + + def test_flat_excludes_listed_files(self): + walker = dummy_top_down_walker('.', self.flat_tree) + self.assertTrue( + "./bar" not in morphlib.fsutils.invert_paths(walker, ["./bar"])) + + def test_nested_excludes_listed_files(self): + walker = dummy_top_down_walker('.', self.nested_tree) + excludes = ["./foo/bar", "./fs/nfs"] + found = frozenset(morphlib.fsutils.invert_paths(walker, excludes)) + self.assertTrue(all(path not in found for path in excludes)) + + def test_nested_excludes_whole_dir(self): + walker = dummy_top_down_walker('.', self.nested_tree) + found = frozenset(morphlib.fsutils.invert_paths(walker, ["./foo"])) + unexpected = ("./foo", "./foo/bar", "./foo/baz") + self.assertTrue(all(path not in found for path in unexpected)) + + def test_lower_mount_precludes(self): + walker = dummy_top_down_walker('.', { + "tmp": { + "morph": { + "staging": { + "build": None, + "inst": None, + }, + }, + "ccache": { + "0": None + }, + }, + "bin": { + }, + }) + found = frozenset(morphlib.fsutils.invert_paths( + walker, [ + "./tmp/morph/staging/build", + "./tmp/morph/staging/inst", + "./tmp", + ])) + expected = ("./bin",) + self.assertEqual(sorted(found), sorted(expected)) diff --git a/morphlib/git.py b/morphlib/git.py new file mode 100644 index 00000000..d897de3b --- /dev/null +++ b/morphlib/git.py @@ -0,0 +1,338 @@ +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import binascii +import cliapp +import ConfigParser +import logging +import os +import re +import string +import StringIO +import time + + +import cliapp + +import morphlib + + +class NoModulesFileError(cliapp.AppException): + + def __init__(self, repo, ref): + Exception.__init__(self, + '%s:%s has no .gitmodules file.' % (repo, ref)) + + +class Submodule(object): + + def __init__(self, name, url, path): + self.name = name + self.url = url + self.path = path + + +class InvalidSectionError(cliapp.AppException): + + def __init__(self, repo, ref, section): + Exception.__init__(self, + '%s:%s:.gitmodules: Found a misformatted section ' + 'title: [%s]' % (repo, ref, section)) + + +class MissingSubmoduleCommitError(cliapp.AppException): + + def __init__(self, repo, ref, submodule): + Exception.__init__(self, + '%s:%s:.gitmodules: No commit object found for ' + 'submodule "%s"' % (repo, ref, submodule)) + + +class Submodules(object): + + def __init__(self, app, repo, ref): + self.app = app + self.repo = repo + self.ref = ref + self.submodules = [] + + def load(self): + content = self._read_gitmodules_file() + + io = StringIO.StringIO(content) + parser = ConfigParser.RawConfigParser() + parser.readfp(io) + + self._validate_and_read_entries(parser) + + def _read_gitmodules_file(self): + try: + # try to read the .gitmodules file from the repo/ref + content = gitcmd(self.app.runcmd, 'cat-file', 'blob', + '%s:.gitmodules' % self.ref, cwd=self.repo, + ignore_fail=True) + + # drop indentation in sections, as RawConfigParser cannot handle it + return '\n'.join([line.strip() for line in content.splitlines()]) + except cliapp.AppException: + raise NoModulesFileError(self.repo, self.ref) + + def _validate_and_read_entries(self, parser): + for section in parser.sections(): + # validate section name against the 'section "foo"' pattern + section_pattern = r'submodule "(.*)"' + if re.match(section_pattern, section): + # parse the submodule name, URL and path + name = re.sub(section_pattern, r'\1', section) + url = parser.get(section, 'url') + path = parser.get(section, 'path') + + # create a submodule object + submodule = Submodule(name, url, path) + try: + # list objects in the parent repo tree to find the commit + # object that corresponds to the submodule + commit = gitcmd(self.app.runcmd, 'ls-tree', self.ref, + submodule.name, cwd=self.repo) + + # read the commit hash from the output + fields = commit.split() + if len(fields) >= 2 and fields[1] == 'commit': + submodule.commit = commit.split()[2] + + # fail if the commit hash is invalid + if len(submodule.commit) != 40: + raise MissingSubmoduleCommitError(self.repo, + self.ref, + submodule.name) + + # add a submodule object to the list + self.submodules.append(submodule) + else: + logging.warning('Skipping submodule "%s" as %s:%s has ' + 'a non-commit object for it' % + (submodule.name, self.repo, self.ref)) + except cliapp.AppException: + raise MissingSubmoduleCommitError(self.repo, self.ref, + submodule.name) + else: + raise InvalidSectionError(self.repo, self.ref, section) + + def __iter__(self): + for submodule in self.submodules: + yield submodule + + def __len__(self): + return len(self.submodules) + + +def update_submodules(app, repo_dir): # pragma: no cover + '''Set up repo submodules, rewriting the URLs to expand prefixes + + We do this automatically rather than leaving it to the user so that they + don't have to worry about the prefixed URLs manually. + ''' + + if os.path.exists(os.path.join(repo_dir, '.gitmodules')): + resolver = morphlib.repoaliasresolver.RepoAliasResolver( + app.settings['repo-alias']) + gitcmd(app.runcmd, 'submodule', 'init', cwd=repo_dir) + submodules = Submodules(app, repo_dir, 'HEAD') + submodules.load() + for submodule in submodules: + gitcmd(app.runcmd, 'config', 'submodule.%s.url' % submodule.name, + resolver.pull_url(submodule.url), cwd=repo_dir) + gitcmd(app.runcmd, 'submodule', 'update', cwd=repo_dir) + + +class ConfigNotSetException(cliapp.AppException): + + def __init__(self, missing, defaults): + self.missing = missing + self.defaults = defaults + if len(missing) == 1: + self.preamble = ('Git configuration for %s has not been set. ' + 'Please set it with:' % missing[0]) + else: + self.preamble = ('Git configuration for keys %s and %s ' + 'have not been set. Please set them with:' + % (', '.join(missing[:-1]), missing[-1])) + + def __str__(self): + lines = [self.preamble] + lines.extend('git config --global %s \'%s\'' % (k, self.defaults[k]) + for k in self.missing) + return '\n '.join(lines) + + +class IdentityNotSetException(ConfigNotSetException): + + preamble = 'Git user info incomplete. Please set your identity, using:' + + def __init__(self, missing): + self.defaults = {"user.name": "My Name", + "user.email": "me@example.com"} + self.missing = missing + + +def get_user_name(runcmd): + '''Get user.name configuration setting. Complain if none was found.''' + if 'GIT_AUTHOR_NAME' in os.environ: + return os.environ['GIT_AUTHOR_NAME'].strip() + try: + config = check_config_set(runcmd, keys={"user.name": "My Name"}) + return config['user.name'] + except ConfigNotSetException, e: + raise IdentityNotSetException(e.missing) + + +def get_user_email(runcmd): + '''Get user.email configuration setting. Complain if none was found.''' + if 'GIT_AUTHOR_EMAIL' in os.environ: + return os.environ['GIT_AUTHOR_EMAIL'].strip() + try: + cfg = check_config_set(runcmd, keys={"user.email": "me@example.com"}) + return cfg['user.email'] + except ConfigNotSetException, e: + raise IdentityNotSetException(e.missing) + +def check_config_set(runcmd, keys, cwd='.'): + ''' Check whether the given keys have values in git config. ''' + missing = [] + found = {} + for key in keys: + try: + value = gitcmd(runcmd, 'config', key, cwd=cwd, + print_command=False).strip() + found[key] = value + except cliapp.AppException: + missing.append(key) + if missing: + raise ConfigNotSetException(missing, keys) + return found + + +def set_remote(runcmd, gitdir, name, url): + '''Set remote with name 'name' use a given url at gitdir''' + return gitcmd(runcmd, 'remote', 'set-url', name, url, cwd=gitdir) + + +def copy_repository(runcmd, repo, destdir, is_mirror=True): + '''Copies a cached repository into a directory using cp. + + This also fixes up the repository afterwards, so that it can contain + code etc. It does not leave any given branch ready for use. + + ''' + if is_mirror == False: + runcmd(['cp', '-a', os.path.join(repo, '.git'), + os.path.join(destdir, '.git')]) + return + + runcmd(['cp', '-a', repo, os.path.join(destdir, '.git')]) + # core.bare should be false so that git believes work trees are possible + gitcmd(runcmd, 'config', 'core.bare', 'false', cwd=destdir) + # we do not want the origin remote to behave as a mirror for pulls + gitcmd(runcmd, 'config', '--unset', 'remote.origin.mirror', cwd=destdir) + # we want a traditional refs/heads -> refs/remotes/origin ref mapping + gitcmd(runcmd, 'config', 'remote.origin.fetch', + '+refs/heads/*:refs/remotes/origin/*', cwd=destdir) + # set the origin url to the cached repo so that we can quickly clean up + gitcmd(runcmd, 'config', 'remote.origin.url', repo, cwd=destdir) + # by packing the refs, we can then edit then en-masse easily + gitcmd(runcmd, 'pack-refs', '--all', '--prune', cwd=destdir) + # turn refs/heads/* into refs/remotes/origin/* in the packed refs + # so that the new copy behaves more like a traditional clone. + logging.debug("Adjusting packed refs for %s" % destdir) + with open(os.path.join(destdir, ".git", "packed-refs"), "r") as ref_fh: + pack_lines = ref_fh.read().split("\n") + with open(os.path.join(destdir, ".git", "packed-refs"), "w") as ref_fh: + ref_fh.write(pack_lines.pop(0) + "\n") + for refline in pack_lines: + if ' refs/remotes/' in refline: + continue + if ' refs/heads/' in refline: + sha, ref = refline[:40], refline[41:] + if ref.startswith("refs/heads/"): + ref = "refs/remotes/origin/" + ref[11:] + refline = "%s %s" % (sha, ref) + ref_fh.write("%s\n" % (refline)) + # Finally run a remote update to clear up the refs ready for use. + gitcmd(runcmd, 'remote', 'update', 'origin', '--prune', cwd=destdir) + + +def checkout_ref(runcmd, gitdir, ref): + '''Checks out a specific ref/SHA1 in a git working tree.''' + gitcmd(runcmd, 'checkout', ref, cwd=gitdir) + gd = morphlib.gitdir.GitDirectory(gitdir) + if gd.has_fat(): + gd.fat_init() + gd.fat_pull() + + +def index_has_changes(runcmd, gitdir): + '''Returns True if there are no staged changes to commit''' + try: + gitcmd(runcmd, 'diff-index', '--cached', '--quiet', + '--ignore-submodules', 'HEAD', cwd=gitdir) + except cliapp.AppException: + return True + return False + + +def reset_workdir(runcmd, gitdir): + '''Removes any differences between the current commit ''' + '''and the status of the working directory''' + gitcmd(runcmd, 'clean', '-fxd', cwd=gitdir) + gitcmd(runcmd, 'reset', '--hard', 'HEAD', cwd=gitdir) + + +def clone_into(runcmd, srcpath, targetpath, ref=None): + '''Clones a repo in srcpath into targetpath, optionally directly at ref.''' + + if ref is None: + gitcmd(runcmd, 'clone', srcpath, targetpath) + elif is_valid_sha1(ref): + gitcmd(runcmd, 'clone', srcpath, targetpath) + gitcmd(runcmd, 'checkout', ref, cwd=targetpath) + else: + gitcmd(runcmd, 'clone', '-b', ref, srcpath, targetpath) + gd = morphlib.gitdir.GitDirectory(targetpath) + if gd.has_fat(): + gd.fat_init() + gd.fat_pull() + +def is_valid_sha1(ref): + '''Checks whether a string is a valid SHA1.''' + + return len(ref) == 40 and all(x in string.hexdigits for x in ref) + +def rev_parse(runcmd, gitdir, ref): + '''Find the sha1 for the given ref''' + return gitcmd(runcmd, 'rev-parse', '--verify', ref, cwd=gitdir)[0:40] + + +def gitcmd(runcmd, *args, **kwargs): + '''Run git commands safely''' + if 'env' not in kwargs: + kwargs['env'] = dict(os.environ) + # git replace means we can't trust that just the sha1 of the branch + # is enough to say what it contains, so we turn it off by setting + # the right flag in an environment variable. + kwargs['env']['GIT_NO_REPLACE_OBJECTS'] = '1' + cmdline = ['git'] + cmdline.extend(args) + return runcmd(cmdline, **kwargs) diff --git a/morphlib/gitdir.py b/morphlib/gitdir.py new file mode 100644 index 00000000..9fef4f1e --- /dev/null +++ b/morphlib/gitdir.py @@ -0,0 +1,733 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import cliapp +import itertools +import os +import re + +import morphlib + + +class NoWorkingTreeError(cliapp.AppException): + + def __init__(self, repo): + cliapp.AppException.__init__( + self, 'Git directory %s has no working tree ' + '(is bare).' % repo.dirname) + + +class InvalidRefError(cliapp.AppException): + def __init__(self, repo, ref): + cliapp.AppException.__init__( + self, 'Git directory %s has no commit ' + 'at ref %s.' %(repo.dirname, ref)) + + +class ExpectedSha1Error(cliapp.AppException): + + def __init__(self, ref): + self.ref = ref + cliapp.AppException.__init__( + self, 'SHA1 expected, got %s' % ref) + + +class RefChangeError(cliapp.AppException): + pass + + +class RefAddError(RefChangeError): + + def __init__(self, gd, ref, sha1, original_exception): + self.gd = gd + self.dirname = dirname = gd.dirname + self.ref = ref + self.sha1 = sha1 + self.original_exception = original_exception + RefChangeError.__init__(self, 'Adding ref %(ref)s '\ + 'with commit %(sha1)s failed in git repository '\ + 'located at %(dirname)s: %(original_exception)r' % locals()) + + +class RefUpdateError(RefChangeError): + + def __init__(self, gd, ref, old_sha1, new_sha1, original_exception): + self.gd = gd + self.dirname = dirname = gd.dirname + self.ref = ref + self.old_sha1 = old_sha1 + self.new_sha1 = new_sha1 + self.original_exception = original_exception + RefChangeError.__init__(self, 'Updating ref %(ref)s '\ + 'from %(old_sha1)s to %(new_sha1)s failed in git repository '\ + 'located at %(dirname)s: %(original_exception)r' % locals()) + + +class RefDeleteError(RefChangeError): + + def __init__(self, gd, ref, sha1, original_exception): + self.gd = gd + self.dirname = dirname = gd.dirname + self.ref = ref + self.sha1 = sha1 + self.original_exception = original_exception + RefChangeError.__init__(self, 'Deleting ref %(ref)s '\ + 'expecting commit %(sha1)s failed in git repository '\ + 'located at %(dirname)s: %(original_exception)r' % locals()) + + +class InvalidRefSpecError(cliapp.AppException): + + def __init__(self, source, target): + self.source = source + self.target = target + cliapp.AppException.__init__( + self, 'source or target must be defined, '\ + 'got %(source)r and %(target)r respectively.' % locals()) + + +class PushError(cliapp.AppException): + pass + + +class NoRefspecsError(PushError): + + def __init__(self, remote): + self.remote = remote + PushError.__init__( + self, 'Push to remote "%s" was given no refspecs.' % remote) + + +class PushFailureError(PushError): + + def __init__(self, remote, refspecs, exit, results, stderr): + self.remote = remote + self.push_url = push_url = remote.get_push_url() + self.refspecs = refspecs + self.exit = exit + self.results = results + self.stderr = stderr + PushError.__init__(self, 'Push to remote "%(remote)s", '\ + 'push url %(push_url)s '\ + 'failed with exit code %(exit)s' % locals()) + + +class RefSpec(object): + '''Class representing how to push or pull a ref. + + `source` is a reference to the local commit/tag you want to push to + the remote. + `target` is the ref on the remote you want to push to. + `require` is the value that the remote is expected to currently be. + Currently `require` is only used to provide a reverse of the respec, + but future versions of Git will support requiring the value of + `target` on the remote to be at a certain commit, or fail. + `force` defaults to false, and if set adds the flag to push even if + it's non-fast-forward. + + If `source` is not provided, but `target` is, then the refspec will + delete `target` on the remote. + If `source` is provided, but `target` is not, then `source` is used + as the `target`, since if you specify a ref for the `source`, you + can push the same local branch to the same remote branch. + + ''' + + def __init__(self, source=None, target=None, require=None, force=False): + if source is None and target is None: + raise InvalidRefSpecError(source, target) + self.source = source + self.target = target + self.require = require + self.force = force + if target is None: + # Default to source if target not given, source must be a + # branch name, or when this refspec is pushed it will fail. + self.target = target = source + if source is None: # Delete if source not given + self.source = source = '0' * 40 + + @property + def push_args(self): + '''Arguments to pass to push to push this ref. + + Returns an iterable of the arguments that would need to be added + to a push command to push this ref spec. + + This currently returns a single-element tuple, but it may expand + to multiple arguments, e.g. + 1. tags expand to `tag "$name"` + 2. : expands to all the matching refs + 3. When Git 1.8.5 becomes available, + `"--force-with-lease=$target:$required" "$source:$target"`. + + ''' + + # TODO: Use require parameter when Git 1.8.5 is available, + # to allow the push to fail if the target ref is not at + # that commit by using the --force-with-lease option. + return ('%(force)s%(source)s:%(target)s' % { + 'force': '+' if self.force else '', + 'source': self.source, + 'target': self.target + }), + + def revert(self): + '''Create a respec which will undo the effect of pushing this one. + + If `require` was not specified, the revert refspec will delete + the branch. + + ''' + + return self.__class__(source=(self.require or '0' * 40), + target=self.target, require=self.source, + force=self.force) + + +PUSH_FORMAT = re.compile(r''' +# Match flag, this is the eventual result in a nutshell +(?P<flag>[- +*=!])\t +# The refspec is colon separated and separated from the rest by another tab. +(?P<from>[^:]*):(?P<to>[^\t]*)\t +# Two possible formats remain, so separate the two with a capture group +(?: + # Summary is an arbitrary string, separated from the reason by a space + (?P<summary>.*)[ ] + # Reason is enclosed in parenthesis and ends the line + \((?P<reason>.*)\) + # The reason is optional, so we may instead only have the summary + | (?P<summary_only>.*) +) +''', re.VERBOSE) + + +class Remote(object): + '''Represent a remote git repository. + + This can either be nascent or concrete, depending on whether the + name is given. + + Changes to a concrete remote's config are written-through to git's + config files, while a nascent remote keeps changes in-memory. + + ''' + + def __init__(self, gd, name=None): + self.gd = gd + self.name = name + self.push_url = None + self.fetch_url = None + + def __str__(self): + return self.name or '(nascent remote)' + + def set_fetch_url(self, url): + self.fetch_url = url + if self.name is not None: + morphlib.git.gitcmd(self.gd._runcmd, 'remote', 'set-url', + self.name, url) + + def set_push_url(self, url): + self.push_url = url + if self.name is not None: + morphlib.git.gitcmd(self.gd._runcmd, 'remote', 'set-url', + '--push', self.name, url) + + def _get_remote_url(self, remote_name, kind): + # As distasteful as it is to parse the output of porcelain + # commands, this is the best option. + # Git config can be used to get the raw value, but this is + # incorrect when url.*.insteadof rules are involved. + # Re-implementing the rewrite logic in morph is duplicated effort + # and more work to keep it in sync. + # It's possible to get the fetch url with `git ls-remote --get-url + # <remote>`, but this will just print the remote's name if it + # is not defined. + # It is only possible to use git to get the push url by parsing + # `git remote -v` or `git remote show -n <remote>`, and `git + # remote -v` is easier to parse. + output = morphlib.git.gitcmd(self.gd._runcmd, 'remote', '-v') + for line in output.splitlines(): + words = line.split() + if (len(words) == 3 and + words[0] == remote_name and + words[2] == '(%s)' % kind): + return words[1] + + return None + + def get_fetch_url(self): + if self.name is None: + return self.fetch_url + return self._get_remote_url(self.name, 'fetch') + + def get_push_url(self): + if self.name is None: + return self.push_url or self.get_fetch_url() + return self._get_remote_url(self.name, 'push') + + @staticmethod + def _parse_ls_remote_output(output): # pragma: no cover + for line in output.splitlines(): + sha1, refname = line.split(None, 1) + yield sha1, refname + + def ls(self): # pragma: no cover + out = morphlib.git.gitcmd(self.gd._runcmd, 'ls-remote', + self.get_fetch_url()) + return self._parse_ls_remote_output(out) + + @staticmethod + def _parse_push_output(output): + for line in output.splitlines(): + m = PUSH_FORMAT.match(line) + # Push may output lines that are not related to the status, + # so ignore any that don't match the status format. + if m is None: + continue + # Ensure the same number of arguments + ret = list(m.group('flag', 'from', 'to')) + ret.append(m.group('summary') or m.group('summary_only')) + ret.append(m.group('reason')) + yield tuple(ret) + + def push(self, *refspecs): + '''Push given refspecs to the remote and return results. + + If no refspecs are given, an exception is raised. + + Returns an iterable of (flag, from_ref, to_ref, summary, reason) + + If the push fails, a PushFailureError is raised, from which the + result can be retrieved with the `results` field. + + ''' + + if not refspecs: + raise NoRefspecsError(self) + push_name = self.name or self.get_push_url() + cmdline = ['push', '--porcelain', push_name] + cmdline.extend(itertools.chain.from_iterable( + rs.push_args for rs in refspecs)) + exit, out, err = morphlib.git.gitcmd(self.gd._runcmd_unchecked, + *cmdline) + if exit != 0: + raise PushFailureError(self, refspecs, exit, + self._parse_push_output(out), err) + return self._parse_push_output(out) + + def pull(self, branch=None): # pragma: no cover + if branch: + repo = self.get_fetch_url() + ret = morphlib.git.gitcmd(self.gd._runcmd, 'pull', repo, branch) + else: + ret = morphlib.git.gitcmd(self.gd._runcmd, 'pull') + return ret + + +class GitDirectory(object): + + '''Represent a git working tree + .git directory. + + This class represents a directory that is the result of a + "git clone". It includes both the .git subdirectory and + the working tree. It is a thin abstraction, meant to make + it easier to do certain git operations. + + ''' + + def __init__(self, dirname): + self.dirname = morphlib.util.find_root(dirname, '.git') + # if we are in a bare repo, self.dirname will now be None + # so we just use the provided dirname + if not self.dirname: + self.dirname = dirname + self._config = {} + + def _runcmd(self, argv, **kwargs): + '''Run a command at the root of the git directory. + + See cliapp.runcmd for arguments. + + Do NOT use this from outside the class. Add more public + methods for specific git operations instead. + + ''' + + return cliapp.runcmd(argv, cwd=self.dirname, **kwargs) + + def _runcmd_unchecked(self, *args, **kwargs): + return cliapp.runcmd_unchecked(*args, cwd=self.dirname, **kwargs) + + def checkout(self, branch_name): # pragma: no cover + '''Check out a git branch.''' + morphlib.git.gitcmd(self._runcmd, 'checkout', branch_name) + if self.has_fat(): + self.fat_init() + self.fat_pull() + + def branch(self, new_branch_name, base_ref): # pragma: no cover + '''Create a git branch based on an existing ref. + + This does not automatically check out the branch. + + base_ref may be None, in which case the current branch is used. + + ''' + + argv = ['branch', new_branch_name] + if base_ref is not None: + argv.append(base_ref) + morphlib.git.gitcmd(self._runcmd, *argv) + + def is_currently_checked_out(self, ref): # pragma: no cover + '''Is ref currently checked out?''' + + # Try the ref name directly first. If that fails, prepend origin/ + # to it. (FIXME: That's a kludge, and should be fixed.) + try: + parsed_ref = morphlib.git.gitcmd(self._runcmd, 'rev-parse', ref) + except cliapp.AppException: + parsed_ref = morphlib.git.gitcmd(self._runcmd, 'rev-parse', + 'origin/%s' % ref) + parsed_head = morphlib.git.gitcmd(self._runcmd, 'rev-parse', 'HEAD') + return parsed_ref.strip() == parsed_head.strip() + + def get_file_from_ref(self, ref, filename): # pragma: no cover + '''Get file contents from git by ref and filename. + + `ref` should be a tree-ish e.g. HEAD, master, refs/heads/master, + refs/tags/foo, though SHA1 tag, commit or tree IDs are also valid. + + `filename` is the path to the file object from the base of the + git directory. + + Returns the contents of the referred to file as a string. + + ''' + + # Blob ID is left as the git revision, rather than SHA1, since + # we know get_blob_contents will accept it + blob_id = '%s:%s' % (ref, filename) + return self.get_blob_contents(blob_id) + + def get_blob_contents(self, blob_id): # pragma: no cover + '''Get file contents from git by ID''' + return morphlib.git.gitcmd(self._runcmd, 'cat-file', 'blob', + blob_id) + + def get_commit_contents(self, commit_id): # pragma: no cover + '''Get commit contents from git by ID''' + return morphlib.git.gitcmd(self._runcmd, 'cat-file', 'commit', + commit_id) + + def update_submodules(self, app): # pragma: no cover + '''Change .gitmodules URLs, and checkout submodules.''' + morphlib.git.update_submodules(app, self.dirname) + + def set_config(self, key, value): + '''Set a git repository configuration variable. + + The key must have at least one period in it: foo.bar for example, + not just foo. The part before the first period is interpreted + by git as a section name. + + ''' + + morphlib.git.gitcmd(self._runcmd, 'config', key, value) + self._config[key] = value + + def get_config(self, key): + '''Return value for a git repository configuration variable.''' + + if key not in self._config: + value = morphlib.git.gitcmd(self._runcmd, 'config', '-z', key) + self._config[key] = value.rstrip('\0') + return self._config[key] + + def get_remote(self, *args, **kwargs): + '''Get a remote for this Repository. + + Gets a previously configured remote if a remote name is given. + Otherwise a nascent one is created. + + ''' + return Remote(self, *args, **kwargs) + + def update_remotes(self): # pragma: no cover + '''Run "git remote update --prune".''' + morphlib.git.gitcmd(self._runcmd, 'remote', 'update', '--prune') + + def is_bare(self): + '''Determine whether the repository has no work tree (is bare)''' + return self.get_config('core.bare') == 'true' + + def list_files(self, ref=None): + '''Return an iterable of the files in the repository. + + If `ref` is specified, list files at that ref, otherwise + use the working tree. + + If this is a bare repository and no ref is specified, raises + an exception. + + ''' + if ref is None and self.is_bare(): + raise NoWorkingTreeError(self) + if ref is None: + return self._list_files_in_work_tree() + else: + return self._list_files_in_ref(ref) + + def _rev_parse(self, ref): + try: + return morphlib.git.gitcmd(self._runcmd, 'rev-parse', + '--verify', ref).strip() + except cliapp.AppException as e: + raise InvalidRefError(self, ref) + + def disambiguate_ref(self, ref): # pragma: no cover + try: + out = morphlib.git.gitcmd(self._runcmd, 'rev-parse', + '--symbolic-full-name', ref) + return out.strip() + except cliapp.AppException: # ref not found + if ref.startswith('refs/heads/'): + return ref + elif ref.startswith('heads/'): + return 'refs/' + ref + else: + return 'refs/heads/' + ref + + def get_upstream_of_branch(self, branch): # pragma: no cover + try: + out = morphlib.git.gitcmd( + self._runcmd, 'rev-parse', '--abbrev-ref', + '%s@{upstream}' % branch).strip() + return out + except cliapp.AppException as e: + emsg = str(e) + if 'does not point to a branch' in emsg: + # ref wasn't a branch, can't have upstream + # treat it the same as no upstream for convenience + return None + elif 'No upstream configured for branch' in emsg: + return None + raise + + def resolve_ref_to_commit(self, ref): + return self._rev_parse('%s^{commit}' % ref) + + def resolve_ref_to_tree(self, ref): + return self._rev_parse('%s^{tree}' % ref) + + def _list_files_in_work_tree(self): + for dirpath, subdirs, filenames in os.walk(self.dirname): + if dirpath == self.dirname and '.git' in subdirs: + subdirs.remove('.git') + for filename in filenames: + filepath = os.path.join(dirpath, filename) + yield os.path.relpath(filepath, start=self.dirname) + + def _list_files_in_ref(self, ref): + tree = self.resolve_ref_to_tree(ref) + output = morphlib.git.gitcmd(self._runcmd, 'ls-tree', + '--name-only', '-rz', tree) + # ls-tree appends \0 instead of interspersing, so we need to + # strip the trailing \0 before splitting + paths = output.strip('\0').split('\0') + return paths + + def read_file(self, filename, ref=None): + if ref is None and self.is_bare(): + raise NoWorkingTreeError(self) + if ref is None: + with open(os.path.join(self.dirname, filename)) as f: + return f.read() + tree = self.resolve_ref_to_tree(ref) + return self.get_file_from_ref(tree, filename) + + def is_symlink(self, filename, ref=None): + if ref is None and self.is_bare(): + raise NoWorkingTreeError(self) + if ref is None: + filepath = os.path.join(self.dirname, filename.lstrip('/')) + return os.path.islink(filepath) + tree_entry = morphlib.git.gitcmd(self._runcmd, 'ls-tree', ref, + filename) + file_mode = tree_entry.split(' ', 1)[0] + return file_mode == '120000' + + @property + def HEAD(self): + output = morphlib.git.gitcmd(self._runcmd, 'rev-parse', + '--abbrev-ref', 'HEAD') + return output.strip() + + def get_index(self, index_file=None): + return morphlib.gitindex.GitIndex(self, index_file) + + def store_blob(self, blob_contents): + '''Hash `blob_contents`, store it in git and return the sha1. + + `blob_contents` must either be a string or a value suitable to + pass to subprocess.Popen i.e. a file descriptor or file object + with fileno() method. + + ''' + if isinstance(blob_contents, basestring): + kwargs = {'feed_stdin': blob_contents} + else: + kwargs = {'stdin': blob_contents} + return morphlib.git.gitcmd(self._runcmd, 'hash-object', '-t', 'blob', + '-w', '--stdin', **kwargs).strip() + + def commit_tree(self, tree, parent, message, **kwargs): + '''Create a commit''' + # NOTE: Will need extension for 0 or N parents. + env = {} + for who, info in itertools.product(('committer', 'author'), + ('name', 'email')): + argname = '%s_%s' % (who, info) + envname = 'GIT_%s_%s' % (who.upper(), info.upper()) + if argname in kwargs: + env[envname] = kwargs[argname] + for who in ('committer', 'author'): + argname = '%s_date' % who + envname = 'GIT_%s_DATE' % who.upper() + if argname in kwargs: + env[envname] = kwargs[argname].isoformat() + return morphlib.git.gitcmd(self._runcmd, 'commit-tree', tree, + '-p', parent, '-m', message, + env=env).strip() + + @staticmethod + def _check_is_sha1(string): + if not morphlib.git.is_valid_sha1(string): + raise ExpectedSha1Error(string) + + def _update_ref(self, ref_args, message): + args = ['update-ref'] + # No test coverage, since while this functionality is useful, + # morph does not need an API for inspecting the reflog, so + # it existing purely to test ref updates is a tad overkill. + if message is not None: # pragma: no cover + args.extend(('-m', message)) + args.extend(ref_args) + morphlib.git.gitcmd(self._runcmd, *args) + + def add_ref(self, ref, sha1, message=None): + '''Create a ref called `ref` in the repository pointing to `sha1`. + + `message` is a string to add to the reflog about this change + `ref` must not already exist, if it does, use `update_ref` + `sha1` must be a 40 character hexadecimal string representing + the SHA1 of the commit or tag this ref will point to, this is + the result of the commit_tree or resolve_ref_to_commit methods. + + ''' + self._check_is_sha1(sha1) + # 40 '0' characters is code for no previous value + # this ensures it will fail if the branch already exists + try: + return self._update_ref((ref, sha1, '0' * 40), message) + except Exception, e: + raise RefAddError(self, ref, sha1, e) + + def update_ref(self, ref, sha1, old_sha1, message=None): + '''Change the commit the ref `ref` points to, to `sha1`. + + `message` is a string to add to the reflog about this change + `sha1` and `old_sha` must be 40 character hexadecimal strings + representing the SHA1 of the commit or tag this ref will point + to and currently points to respectively. This is the result of + the commit_tree or resolve_ref_to_commit methods. + `ref` must exist, and point to `old_sha1`. + This is to avoid unexpected results when multiple processes + attempt to change refs. + + ''' + self._check_is_sha1(sha1) + self._check_is_sha1(old_sha1) + try: + return self._update_ref((ref, sha1, old_sha1), message) + except Exception, e: + raise RefUpdateError(self, ref, old_sha1, sha1, e) + + def delete_ref(self, ref, old_sha1, message=None): + '''Remove the ref `ref`. + + `message` is a string to add to the reflog about this change + `old_sha1` must be a 40 character hexadecimal string representing + the SHA1 of the commit or tag this ref will point to, this is + the result of the commit_tree or resolve_ref_to_commit methods. + `ref` must exist, and point to `old_sha1`. + This is to avoid unexpected results when multiple processes + attempt to change refs. + + ''' + self._check_is_sha1(old_sha1) + try: + return self._update_ref(('-d', ref, old_sha1), message) + except Exception, e: + raise RefDeleteError(self, ref, old_sha1, e) + + def describe(self): + version = morphlib.git.gitcmd(self._runcmd, 'describe', + '--always', '--dirty=-unreproducible') + return version.strip() + + def fat_init(self): # pragma: no cover + return morphlib.git.gitcmd(self._runcmd, 'fat', 'init') + + def fat_push(self): # pragma: no cover + return morphlib.git.gitcmd(self._runcmd, 'fat', 'push') + + def fat_pull(self): # pragma: no cover + return morphlib.git.gitcmd(self._runcmd, 'fat', 'pull') + + def has_fat(self): # pragma: no cover + return os.path.isfile(self.join_path('.gitfat')) + + def join_path(self, path): # pragma: no cover + return os.path.join(self.dirname, path) + + def get_relpath(self, path): # pragma: no cover + return os.path.relpath(path, self.dirname) + + +def init(dirname): + '''Initialise a new git repository.''' + + morphlib.git.gitcmd(cliapp.runcmd, 'init', cwd=dirname) + gd = GitDirectory(dirname) + return gd + + +def clone_from_cached_repo(cached_repo, dirname, ref): # pragma: no cover + '''Clone a CachedRepo into the desired directory. + + The given ref is checked out (or git's default branch is checked out + if ref is None). + + ''' + + cached_repo.clone_checkout(ref, dirname) + return GitDirectory(dirname) + diff --git a/morphlib/gitdir_tests.py b/morphlib/gitdir_tests.py new file mode 100644 index 00000000..456e3716 --- /dev/null +++ b/morphlib/gitdir_tests.py @@ -0,0 +1,505 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import datetime +import os +import shutil +import tempfile +import unittest + +import morphlib + + +class GitDirectoryTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.dirname = os.path.join(self.tempdir, 'foo') + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def fake_git_clone(self): + os.mkdir(self.dirname) + os.mkdir(os.path.join(self.dirname, '.git')) + + def test_has_dirname_attribute(self): + self.fake_git_clone() + gitdir = morphlib.gitdir.GitDirectory(self.dirname) + self.assertEqual(gitdir.dirname, self.dirname) + + def test_runs_command_in_right_directory(self): + self.fake_git_clone() + gitdir = morphlib.gitdir.GitDirectory(self.dirname) + output = gitdir._runcmd(['pwd']) + self.assertEqual(output.strip(), self.dirname) + + def test_sets_and_gets_configuration(self): + os.mkdir(self.dirname) + gitdir = morphlib.gitdir.init(self.dirname) + gitdir.set_config('foo.bar', 'yoyo') + self.assertEqual(gitdir.get_config('foo.bar'), 'yoyo') + + def test_gets_index(self): + os.mkdir(self.dirname) + gitdir = morphlib.gitdir.init(self.dirname) + self.assertIsInstance(gitdir.get_index(), morphlib.gitindex.GitIndex) + + +class GitDirectoryContentsTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.dirname = os.path.join(self.tempdir, 'foo') + os.mkdir(self.dirname) + gd = morphlib.gitdir.init(self.dirname) + for fn in ('foo', 'bar.morph', 'baz.morph', 'quux'): + with open(os.path.join(self.dirname, fn), "w") as f: + f.write('dummy morphology text') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Initial commit') + os.rename(os.path.join(self.dirname, 'foo'), + os.path.join(self.dirname, 'foo.morph')) + self.mirror = os.path.join(self.tempdir, 'mirror') + morphlib.git.gitcmd(gd._runcmd, 'clone', '--mirror', self.dirname, + self.mirror) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_lists_files_in_work_tree(self): + expected = ['bar.morph', 'baz.morph', 'foo.morph', 'quux'] + + gd = morphlib.gitdir.GitDirectory(self.dirname) + self.assertEqual(sorted(gd.list_files()), expected) + + gd = morphlib.gitdir.GitDirectory(self.dirname + '/') + self.assertEqual(sorted(gd.list_files()), expected) + + def test_read_file_in_work_tree(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + self.assertEqual(gd.read_file('bar.morph'), + 'dummy morphology text') + + def test_list_raises_no_ref_no_work_tree(self): + gd = morphlib.gitdir.GitDirectory(self.mirror) + self.assertRaises(morphlib.gitdir.NoWorkingTreeError, + gd.list_files) + + def test_read_raises_no_ref_no_work_tree(self): + gd = morphlib.gitdir.GitDirectory(self.mirror) + self.assertRaises(morphlib.gitdir.NoWorkingTreeError, + gd.read_file, 'bar.morph') + + def test_lists_files_in_HEAD(self): + for gitdir in (self.dirname, self.mirror): + gd = morphlib.gitdir.GitDirectory(gitdir) + self.assertEqual(sorted(gd.list_files('HEAD')), + ['bar.morph', 'baz.morph', 'foo', 'quux']) + + def test_read_files_in_HEAD(self): + for gitdir in (self.dirname, self.mirror): + gd = morphlib.gitdir.GitDirectory(gitdir) + self.assertEqual(gd.read_file('bar.morph', 'HEAD'), + 'dummy morphology text') + + def test_lists_files_in_named_ref(self): + for gitdir in (self.dirname, self.mirror): + gd = morphlib.gitdir.GitDirectory(gitdir) + self.assertEqual(sorted(gd.list_files('master')), + ['bar.morph', 'baz.morph', 'foo', 'quux']) + + def test_read_file_in_named_ref(self): + for gitdir in (self.dirname, self.mirror): + gd = morphlib.gitdir.GitDirectory(gitdir) + self.assertEqual(gd.read_file('bar.morph', 'master'), + 'dummy morphology text') + + def test_list_raises_invalid_ref(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + self.assertRaises(morphlib.gitdir.InvalidRefError, + gd.list_files, 'no-such-ref') + + def test_read_raises_invalid_ref(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + self.assertRaises(morphlib.gitdir.InvalidRefError, + gd.read_file, 'bar', 'no-such-ref') + + def test_HEAD(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + self.assertEqual(gd.HEAD, 'master') + + gd.branch('foo', 'master') + self.assertEqual(gd.HEAD, 'master') + + gd.checkout('foo') + self.assertEqual(gd.HEAD, 'foo') + + def test_resolve_ref(self): + # Just tests that you get an object IDs back and that the + # commit and tree IDs are different, since checking the actual + # value of the commit requires foreknowledge of the result or + # re-implementing the body in the test. + gd = morphlib.gitdir.GitDirectory(self.dirname) + commit = gd.resolve_ref_to_commit(gd.HEAD) + self.assertEqual(len(commit), 40) + tree = gd.resolve_ref_to_tree(gd.HEAD) + self.assertEqual(len(tree), 40) + self.assertNotEqual(commit, tree) + + def test_store_blob_with_string(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + sha1 = gd.store_blob('test string') + self.assertEqual('test string', gd.get_blob_contents(sha1)) + + def test_store_blob_with_file(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + with open(os.path.join(self.tempdir, 'blob'), 'w') as f: + f.write('test string') + with open(os.path.join(self.tempdir, 'blob'), 'r') as f: + sha1 = gd.store_blob(f) + self.assertEqual('test string', gd.get_blob_contents(sha1)) + + def test_commit_tree(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + parent = gd.resolve_ref_to_commit(gd.HEAD) + tree = gd.resolve_ref_to_tree(parent) + aname = 'Author Name' + aemail = 'author@email' + cname = 'Committer Name' + cemail = 'committer@email' + pseudo_now = datetime.datetime.fromtimestamp(683074800) + + now_str = "683074800" + message= 'MESSAGE' + expected = [ + "tree %(tree)s", + "parent %(parent)s", + "author %(aname)s <%(aemail)s> %(now_str)s +0000", + "committer %(cname)s <%(cemail)s> %(now_str)s +0000", + "", + "%(message)s", + "", + ] + expected = [l % locals() for l in expected] + commit = gd.commit_tree(tree, parent, message=message, + committer_name=cname, + committer_email=cemail, + committer_date=pseudo_now, + author_name=aname, + author_email=aemail, + author_date=pseudo_now, + ) + self.assertEqual(expected, gd.get_commit_contents(commit).split('\n')) + + def test_describe(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + + morphlib.git.gitcmd(gd._runcmd, 'tag', '-a', '-m', 'Example', + 'example', 'HEAD') + self.assertEqual(gd.describe(), 'example-unreproducible') + + morphlib.git.gitcmd(gd._runcmd, 'reset', '--hard') + self.assertEqual(gd.describe(), 'example') + + +class GitDirectoryFileTypeTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.dirname = os.path.join(self.tempdir, 'foo') + os.mkdir(self.dirname) + gd = morphlib.gitdir.init(self.dirname) + with open(os.path.join(self.dirname, 'file'), "w") as f: + f.write('dummy morphology text') + os.symlink('file', os.path.join(self.dirname, 'link')) + os.symlink('no file', os.path.join(self.dirname, 'broken')) + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Initial commit') + self.mirror = os.path.join(self.tempdir, 'mirror') + morphlib.git.gitcmd(gd._runcmd, 'clone', '--mirror', self.dirname, + self.mirror) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_working_tree_symlinks(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + self.assertTrue(gd.is_symlink('link')) + self.assertTrue(gd.is_symlink('broken')) + self.assertFalse(gd.is_symlink('file')) + + def test_bare_symlinks(self): + gd = morphlib.gitdir.GitDirectory(self.mirror) + self.assertTrue(gd.is_symlink('link', 'HEAD')) + self.assertTrue(gd.is_symlink('broken', 'HEAD')) + self.assertFalse(gd.is_symlink('file', 'HEAD')) + + def test_is_symlink_raises_no_ref_no_work_tree(self): + gd = morphlib.gitdir.GitDirectory(self.mirror) + self.assertRaises(morphlib.gitdir.NoWorkingTreeError, + gd.is_symlink, 'file') + + +class GitDirectoryRefTwiddlingTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.dirname = os.path.join(self.tempdir, 'foo') + os.mkdir(self.dirname) + gd = morphlib.gitdir.init(self.dirname) + with open(os.path.join(self.dirname, 'foo'), 'w') as f: + f.write('dummy text\n') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Initial commit') + # Add a second commit for update_ref test, so it has another + # commit to roll back from + with open(os.path.join(self.dirname, 'bar'), 'w') as f: + f.write('dummy text\n') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Second commit') + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_expects_sha1s(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + self.assertRaises(morphlib.gitdir.ExpectedSha1Error, + gd.add_ref, 'refs/heads/foo', 'HEAD') + self.assertRaises(morphlib.gitdir.ExpectedSha1Error, + gd.update_ref, 'refs/heads/foo', 'HEAD', 'HEAD') + self.assertRaises(morphlib.gitdir.ExpectedSha1Error, + gd.update_ref, 'refs/heads/master', + gd._rev_parse(gd.HEAD), 'HEAD') + self.assertRaises(morphlib.gitdir.ExpectedSha1Error, + gd.update_ref, 'refs/heads/master', + 'HEAD', gd._rev_parse(gd.HEAD)) + self.assertRaises(morphlib.gitdir.ExpectedSha1Error, + gd.delete_ref, 'refs/heads/master', 'HEAD') + + def test_add_ref(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + head_commit = gd.resolve_ref_to_commit(gd.HEAD) + gd.add_ref('refs/heads/foo', head_commit) + self.assertEqual(gd.resolve_ref_to_commit('refs/heads/foo'), + head_commit) + + def test_add_ref_fail(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + head_commit = gd.resolve_ref_to_commit('refs/heads/master') + self.assertRaises(morphlib.gitdir.RefAddError, + gd.add_ref, 'refs/heads/master', head_commit) + + def test_update_ref(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + head_commit = gd._rev_parse('refs/heads/master') + prev_commit = gd._rev_parse('refs/heads/master^') + gd.update_ref('refs/heads/master', prev_commit, head_commit) + self.assertEqual(gd._rev_parse('refs/heads/master'), prev_commit) + + def test_update_ref_fail(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + head_commit = gd._rev_parse('refs/heads/master') + prev_commit = gd._rev_parse('refs/heads/master^') + gd.delete_ref('refs/heads/master', head_commit) + with self.assertRaises(morphlib.gitdir.RefUpdateError): + gd.update_ref('refs/heads/master', prev_commit, head_commit) + + def test_delete_ref(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + head_commit = gd._rev_parse('refs/heads/master') + gd.delete_ref('refs/heads/master', head_commit) + self.assertRaises(morphlib.gitdir.InvalidRefError, + gd._rev_parse, 'refs/heads/master') + + def test_delete_ref_fail(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + prev_commit = gd._rev_parse('refs/heads/master^') + with self.assertRaises(morphlib.gitdir.RefDeleteError): + gd.delete_ref('refs/heads/master', prev_commit) + + +class GitDirectoryRemoteConfigTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.dirname = os.path.join(self.tempdir, 'foo') + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_sets_urls(self): + os.mkdir(self.dirname) + gitdir = morphlib.gitdir.init(self.dirname) + remote = gitdir.get_remote('origin') + self.assertEqual(remote.get_fetch_url(), None) + self.assertEqual(remote.get_push_url(), None) + + morphlib.git.gitcmd(gitdir._runcmd, 'remote', 'add', 'origin', + 'foobar') + fetch_url = 'git://git.example.com/foo.git' + push_url = 'ssh://git@git.example.com/foo.git' + remote.set_fetch_url(fetch_url) + remote.set_push_url(push_url) + self.assertEqual(remote.get_fetch_url(), fetch_url) + self.assertEqual(remote.get_push_url(), push_url) + + def test_nascent_remote_fetch(self): + os.mkdir(self.dirname) + gitdir = morphlib.gitdir.init(self.dirname) + remote = gitdir.get_remote(None) + self.assertEqual(remote.get_fetch_url(), None) + self.assertEqual(remote.get_push_url(), None) + + fetch_url = 'git://git.example.com/foo.git' + push_url = 'ssh://git@git.example.com/foo.git' + remote.set_fetch_url(fetch_url) + remote.set_push_url(push_url) + self.assertEqual(remote.get_fetch_url(), fetch_url) + self.assertEqual(remote.get_push_url(), push_url) + + +class RefSpecTests(unittest.TestCase): + + def setUp(self): + pass + + def tearDown(self): + pass + + @staticmethod + def refspec(*args, **kwargs): + return morphlib.gitdir.RefSpec(*args, **kwargs) + + def test_input(self): + with self.assertRaises(morphlib.gitdir.InvalidRefSpecError): + morphlib.gitdir.RefSpec() + + def test_rs_from_source(self): + rs = self.refspec(source='master') + self.assertEqual(rs.push_args, ('master:master',)) + + def test_rs_from_target(self): + rs = self.refspec(target='master') + self.assertEqual(rs.push_args, ('%s:master' % ('0' * 40),)) + + def test_rs_with_target_and_source(self): + rs = self.refspec(source='foo', target='master') + self.assertEqual(rs.push_args, ('foo:master',)) + + def test_rs_with_source_and_force(self): + rs = self.refspec('master', force=True) + self.assertEqual(rs.push_args, ('+master:master',)) + + def test_rs_revert_from_source(self): + revert = self.refspec(source='master').revert() + self.assertEqual(revert.push_args, ('%s:master' % ('0' * 40),)) + + def test_rs_revert_inc_require(self): + revert = self.refspec(source='master', require=('beef'*5)).revert() + self.assertEqual(revert.push_args, ('%s:master' % ('beef' * 5),)) + + def test_rs_double_revert(self): + rs = self.refspec(target='master').revert().revert() + self.assertEqual(rs.push_args, ('%s:master' % ('0' * 40),)) + + +class GitDirectoryRemotePushTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.dirname = os.path.join(self.tempdir, 'foo') + os.mkdir(self.dirname) + gd = morphlib.gitdir.init(self.dirname) + with open(os.path.join(self.dirname, 'foo'), 'w') as f: + f.write('dummy text\n') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Initial commit') + morphlib.git.gitcmd(gd._runcmd, 'checkout', '-b', 'foo') + with open(os.path.join(self.dirname, 'foo'), 'w') as f: + f.write('updated text\n') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Second commit') + self.mirror = os.path.join(self.tempdir, 'mirror') + morphlib.git.gitcmd(gd._runcmd, 'init', '--bare', self.mirror) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_push_needs_refspecs(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + r = gd.get_remote() + r.set_push_url(self.mirror) + self.assertRaises(morphlib.gitdir.NoRefspecsError, r.push) + + def test_push_new(self): + push_master = morphlib.gitdir.RefSpec('master') + gd = morphlib.gitdir.GitDirectory(self.dirname) + r = gd.get_remote() + r.set_push_url(self.mirror) + self.assertEqual(sorted(r.push(push_master)), + [('*', 'refs/heads/master', 'refs/heads/master', + '[new branch]', None)]) + + def test_double_push(self): + push_master = morphlib.gitdir.RefSpec('master') + gd = morphlib.gitdir.GitDirectory(self.dirname) + r = gd.get_remote() + r.set_push_url(self.mirror) + r.push(push_master) + self.assertEqual(sorted(r.push(push_master)), + [('=', 'refs/heads/master', 'refs/heads/master', + '[up to date]', None)]) + + def test_push_update(self): + push_master = morphlib.gitdir.RefSpec('master') + push_foo = morphlib.gitdir.RefSpec(source='foo', target='master') + gd = morphlib.gitdir.GitDirectory(self.dirname) + r = gd.get_remote() + r.set_push_url(self.mirror) + r.push(push_master) + flag, ref_from, ref_to, summary, reason = \ + list(r.push(push_foo))[0] + self.assertEqual((flag, ref_from, ref_to), + (' ', 'refs/heads/foo', 'refs/heads/master')) + + def test_rewind_fail(self): + push_master = morphlib.gitdir.RefSpec('master') + push_foo = morphlib.gitdir.RefSpec(source='foo', target='master') + gd = morphlib.gitdir.GitDirectory(self.dirname) + r = gd.get_remote() + r.set_push_url(self.mirror) + r.push(push_foo) + with self.assertRaises(morphlib.gitdir.PushFailureError) as push_fail: + r.push(push_master) + self.assertEqual(sorted(push_fail.exception.results), + [('!', 'refs/heads/master', 'refs/heads/master', + '[rejected]', 'non-fast-forward')]) + + def test_force_push(self): + push_master = morphlib.gitdir.RefSpec('master', force=True) + push_foo = morphlib.gitdir.RefSpec(source='foo', target='master') + gd = morphlib.gitdir.GitDirectory(self.dirname) + r = gd.get_remote() + r.set_push_url(self.mirror) + r.push(push_foo) + flag, ref_from, ref_to, summary, reason = \ + list(r.push(push_master))[0] + self.assertEqual((flag, ref_from, ref_to, reason), + ('+', 'refs/heads/master', 'refs/heads/master', + 'forced update')) diff --git a/morphlib/gitindex.py b/morphlib/gitindex.py new file mode 100644 index 00000000..e22f6225 --- /dev/null +++ b/morphlib/gitindex.py @@ -0,0 +1,161 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import collections +import os + +import morphlib + + +STATUS_UNTRACKED = '??' +STATUS_IGNORED = '!!' + + +class GitIndex(object): + '''An object that represents operations on the working tree. + + Index objects can be constructed with a different path to the + index file, which can be used to construct commits without + altering the working tree, index or HEAD. + + The file must either be a previously initialised index, or a + non-existant file. + + Git creates a lock file and atomically alters the index by + renaming a temporary file into place, so `index_file` must be + in a writable directory. + + ''' + + def __init__(self, gd, index_file): + self._gd = gd + self._index_file = index_file + + def _run_git(self, *args, **kwargs): + if self._index_file is not None: + kwargs['env'] = kwargs.get('env', dict(os.environ)) + kwargs['env']['GIT_INDEX_FILE'] = self._index_file + return morphlib.git.gitcmd(self._gd._runcmd, *args, **kwargs) + + def _get_status(self): + '''Return git status output in a Python useful format + + This runs git status such that unusual filenames are preserved + and returns its output in a sequence of (status_code, to_path, + from_path). + + from_path is None unless the status_code says there was a + rename, in which case it is the path it was renamed from. + + Untracked and ignored changes are also included in the output, + their status codes are '??' and '!!' respectively. + + ''' + + # git status -z will NUL terminate paths, so we don't have to + # unescape the paths it outputs. Unfortunately each status entry + # can have 1 or 2 paths, so extra parsing is required. + # To handle this, we split it into NUL delimited tokens. + # The first token of an entry is the 2 character status code, + # a space, then the path. + # If our status code starts with R then it's a rename, hence + # has a second path, requiring us to pop an extra token. + status = self._run_git('status', '-z', '--ignored') + tokens = collections.deque(status.split('\0')) + while True: + tok = tokens.popleft() + # Status output is NUL terminated rather than delimited, + # and split is for delimited output. A side effect of this is + # that we get an empty token as the last output. This suits + # us fine, as it gives us a sentinel value to terminate with. + if not tok: + return + + # The first token of an entry is 2 character status, a space, + # then the path + code = tok[:2] + to_path = tok[3:] + + # If the code starts with R then it's a rename, and + # the next token says where the file was renamed from + from_path = tokens.popleft() if code[0] == 'R' else None + yield code, to_path, from_path + + def get_uncommitted_changes(self): + for code, to_path, from_path in self._get_status(): + if (code not in (STATUS_UNTRACKED, STATUS_IGNORED) + or code == (STATUS_UNTRACKED) and to_path.endswith('.morph')): + yield code, to_path, from_path + + def set_to_tree(self, treeish): + '''Modify the index to contain the contents of the treeish.''' + self._run_git('read-tree', treeish) + + def add_files_from_index_info(self, infos): + '''Add files without interacting with the working tree. + + `infos` is an iterable of (file mode string, object sha1, path) + There are no constraints on the size of the iterable + + ''' + + # update-index may take NUL terminated input lines of the entries + # to add so we generate a string for the input, rather than + # having many command line arguments, since for a large amount + # of entries, this can be too many arguments to process and the + # exec will fail. + # Generating the input as a string uses more memory than using + # subprocess.Popen directly and using .communicate, but is much + # less verbose. + feed_stdin = '\0'.join('%o %s\t%s' % (mode, sha1, path) + for mode, sha1, path in infos) + '\0' + self._run_git('update-index', '--add', '-z', '--index-info', + feed_stdin=feed_stdin) + + def add_files_from_working_tree(self, paths): + '''Add existing files to the index. + + Given an iterable of paths to files in the working tree, + relative to the git repository's top-level directory, + add the contents of the files to git's object store, + and the index. + + This is similar to the following: + + gd = GitDirectory(...) + idx = gd.get_index() + for path in paths: + fullpath = os.path.join(gd,dirname, path) + with open(fullpath, 'r') as f: + sha1 = gd.store_blob(f) + idx.add_files_from_index_info([(os.stat(fullpath).st_mode, + sha1, path)]) + + ''' + + if self._gd.is_bare(): + raise morphlib.gitdir.NoWorkingTreeError(self._gd) + # Handle paths in smaller chunks, so that the runcmd + # cannot fail from exceeding command line length + # 50 is an arbitrary limit + for paths in morphlib.util.iter_trickle(paths, 50): + self._run_git('add', *paths) + + def write_tree(self): + '''Transform the index into a tree in the object store.''' + return self._run_git('write-tree').strip() diff --git a/morphlib/gitindex_tests.py b/morphlib/gitindex_tests.py new file mode 100644 index 00000000..32d40a8c --- /dev/null +++ b/morphlib/gitindex_tests.py @@ -0,0 +1,93 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import os +import shutil +import tempfile +import unittest + +import morphlib + + +class GitIndexTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.dirname = os.path.join(self.tempdir, 'foo') + os.mkdir(self.dirname) + gd = morphlib.gitdir.init(self.dirname) + with open(os.path.join(self.dirname, 'foo'), 'w') as f: + f.write('dummy text\n') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Initial commit') + self.mirror = os.path.join(self.tempdir, 'mirror') + morphlib.git.gitcmd(gd._runcmd, 'clone', '--mirror', self.dirname, + self.mirror) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_uncommitted_changes(self): + idx = morphlib.gitdir.GitDirectory(self.dirname).get_index() + self.assertEqual(list(idx.get_uncommitted_changes()), []) + os.unlink(os.path.join(self.dirname, 'foo')) + self.assertEqual(sorted(idx.get_uncommitted_changes()), + [(' D', 'foo', None)]) + + def test_uncommitted_alt_index(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + idx = gd.get_index(os.path.join(self.tempdir, 'index')) + self.assertEqual(sorted(idx.get_uncommitted_changes()), + [('D ', 'foo', None)]) + # 'D ' means not in the index, but in the working tree + + def test_set_to_tree_alt_index(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + idx = gd.get_index(os.path.join(self.tempdir, 'index')) + # Read the HEAD commit into the index, which is the same as the + # working tree, so there are no uncommitted changes reported + # by status + idx.set_to_tree(gd.HEAD) + self.assertEqual(list(idx.get_uncommitted_changes()),[]) + + def test_add_files_from_index_info(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + idx = gd.get_index(os.path.join(self.tempdir, 'index')) + filepath = os.path.join(gd.dirname, 'foo') + with open(filepath, 'r') as f: + sha1 = gd.store_blob(f) + idx.add_files_from_index_info( + [(os.stat(filepath).st_mode, sha1, 'foo')]) + self.assertEqual(list(idx.get_uncommitted_changes()),[]) + + def test_add_files_from_working_tree(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + idx = gd.get_index() + idx.add_files_from_working_tree(['foo']) + self.assertEqual(list(idx.get_uncommitted_changes()),[]) + + def test_add_files_from_working_tree_fails_in_bare(self): + gd = morphlib.gitdir.GitDirectory(self.mirror) + idx = gd.get_index() + self.assertRaises(morphlib.gitdir.NoWorkingTreeError, + idx.add_files_from_working_tree, ['foo']) + + def test_write_tree(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + idx = gd.get_index() + self.assertEqual(idx.write_tree(), gd.resolve_ref_to_tree(gd.HEAD)) diff --git a/morphlib/gitversion.py b/morphlib/gitversion.py new file mode 100644 index 00000000..c593c330 --- /dev/null +++ b/morphlib/gitversion.py @@ -0,0 +1,59 @@ +# Copyright (C) 2013 - 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +'''Version information retrieved either from the package data, or the + git repository the library is being run from. + + It is an error to run morph without this version information, since + it makes it impossible to reproduce any Systems that are built. +''' + + +import subprocess +import os + +import cliapp + + +try: + import pkgutil + version = pkgutil.get_data('morphlib', 'version') + commit = pkgutil.get_data('morphlib', 'commit') + tree = pkgutil.get_data('morphlib', 'tree') + ref = pkgutil.get_data('morphlib', 'ref') +except IOError, e: + from os.path import dirname + def run_git(*args): + command = ['git'] + list(args) + p = subprocess.Popen(command, + cwd=os.path.dirname(__file__), + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + o = p.communicate() + if p.returncode: + raise subprocess.CalledProcessError(p.returncode, + command) + return o[0].strip() + + try: + version = run_git('describe', '--abbrev=40', '--always', + '--dirty=-unreproducible', + '--match=DO-NOT-MATCH-ANY-TAGS') + commit = run_git('rev-parse', 'HEAD^{commit}') + tree = run_git('rev-parse', 'HEAD^{tree}') + ref = run_git('rev-parse', '--symbolic-full-name', 'HEAD') + except cliapp.AppException: + raise cliapp.AppException("morphlib version could not be determined") diff --git a/morphlib/localartifactcache.py b/morphlib/localartifactcache.py new file mode 100644 index 00000000..955ee97f --- /dev/null +++ b/morphlib/localartifactcache.py @@ -0,0 +1,151 @@ +# Copyright (C) 2012, 2013, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import collections +import os +import time + +import morphlib + + +class LocalArtifactCache(object): + '''Abstraction over the local artifact cache + + It provides methods for getting a file handle to cached artifacts + so that the layout of the cache need not be known. + + It also updates modification times of artifacts so that it can track + when they were last used, so it can be requested to clean up if + disk space is low. + + Modification time is updated in both the get and has methods. + + NOTE: Parts of the build assume that every artifact of a source is + available, so all the artifacts of a source need to be removed together. + + This complication needs to be handled either during the fetch logic, by + updating the mtime of every artifact belonging to a source, or at + cleanup time by only removing an artifact if every artifact belonging to + a source is too old, and then remove them all at once. + + Since the cleanup logic will be complicated for other reasons it makes + sense to put the complication there. + ''' + + def __init__(self, cachefs): + self.cachefs = cachefs + + def put(self, artifact): + filename = self.artifact_filename(artifact) + return morphlib.savefile.SaveFile(filename, mode='w') + + def put_artifact_metadata(self, artifact, name): + filename = self._artifact_metadata_filename(artifact, name) + return morphlib.savefile.SaveFile(filename, mode='w') + + def put_source_metadata(self, source, cachekey, name): + filename = self._source_metadata_filename(source, cachekey, name) + return morphlib.savefile.SaveFile(filename, mode='w') + + def _has_file(self, filename): + if os.path.exists(filename): + os.utime(filename, None) + return True + return False + + def has(self, artifact): + filename = self.artifact_filename(artifact) + return self._has_file(filename) + + def has_artifact_metadata(self, artifact, name): + filename = self._artifact_metadata_filename(artifact, name) + return self._has_file(filename) + + def has_source_metadata(self, source, cachekey, name): + filename = self._source_metadata_filename(source, cachekey, name) + return self._has_file(filename) + + def get(self, artifact): + filename = self.artifact_filename(artifact) + os.utime(filename, None) + return open(filename) + + def get_artifact_metadata(self, artifact, name): + filename = self._artifact_metadata_filename(artifact, name) + os.utime(filename, None) + return open(filename) + + def get_source_metadata_filename(self, source, cachekey, name): + return self._source_metadata_filename(source, cachekey, name) + + def get_source_metadata(self, source, cachekey, name): + filename = self._source_metadata_filename(source, cachekey, name) + os.utime(filename, None) + return open(filename) + + def _join(self, basename): + '''Wrapper for pyfilesystem's getsyspath. + + This is required because its API throws us a garbage unicode + string, when file paths are binary data. + ''' + return str(self.cachefs.getsyspath(basename)) + + def artifact_filename(self, artifact): + basename = artifact.basename() + return self._join(basename) + + def _artifact_metadata_filename(self, artifact, name): + return self._join(artifact.metadata_basename(name)) + + def _source_metadata_filename(self, source, cachekey, name): + return self._join('%s.%s' % (cachekey, name)) + + def clear(self): + '''Clear everything from the artifact cache directory. + + After calling this, the artifact cache will be entirely empty. + Caveat caller. + + ''' + for filename in self.cachefs.walkfiles(): + self.cachefs.remove(filename) + + def list_contents(self): + '''Return the set of sources cached and related information. + + returns a [(cache_key, set(artifacts), last_used)] + + ''' + CacheInfo = collections.namedtuple('CacheInfo', ('artifacts', 'mtime')) + contents = collections.defaultdict(lambda: CacheInfo(set(), 0)) + for filename in self.cachefs.walkfiles(): + cachekey = filename[:63] + artifact = filename[65:] + artifacts, max_mtime = contents[cachekey] + artifacts.add(artifact) + art_info = self.cachefs.getinfo(filename) + time_t = art_info['modified_time'].timetuple() + contents[cachekey] = CacheInfo(artifacts, + max(max_mtime, time.mktime(time_t))) + return ((cache_key, info.artifacts, info.mtime) + for cache_key, info in contents.iteritems()) + + def remove(self, cachekey): + '''Remove all artifacts associated with the given cachekey.''' + for filename in (x for x in self.cachefs.walkfiles() + if x.startswith(cachekey)): + self.cachefs.remove(filename) diff --git a/morphlib/localartifactcache_tests.py b/morphlib/localartifactcache_tests.py new file mode 100644 index 00000000..4325cfbe --- /dev/null +++ b/morphlib/localartifactcache_tests.py @@ -0,0 +1,192 @@ +# Copyright (C) 2012,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import unittest +import os + +import fs.tempfs + +import morphlib + + +class LocalArtifactCacheTests(unittest.TestCase): + + def setUp(self): + self.tempfs = fs.tempfs.TempFS() + + loader = morphlib.morphloader.MorphologyLoader() + morph = loader.load_from_string( + ''' + name: chunk + kind: chunk + products: + - artifact: chunk-runtime + include: + - usr/bin + - usr/sbin + - usr/lib + - usr/libexec + - artifact: chunk-devel + include: + - usr/include + ''') + sources = morphlib.source.make_sources('repo', 'ref', + 'chunk.morph', 'sha1', + 'tree', morph) + self.source, = sources + self.source.cache_key = '0'*64 + self.runtime_artifact = morphlib.artifact.Artifact( + self.source, 'chunk-runtime') + self.devel_artifact = morphlib.artifact.Artifact( + self.source, 'chunk-devel') + + def test_artifact_filename(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + filename = cache.artifact_filename(self.devel_artifact) + expected_name = self.tempfs.getsyspath(self.devel_artifact.basename()) + self.assertEqual(filename, expected_name) + + def test_get_source_metadata_filename(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + artifact = self.devel_artifact + source = self.source + name = 'foobar' + + filename = cache.get_source_metadata_filename(artifact.source, + source.cache_key, name) + expected_name = self.tempfs.getsyspath('%s.%s' % + (source.cache_key, name)) + self.assertEqual(filename, expected_name) + + def test_put_artifacts_and_check_whether_the_cache_has_them(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + + handle = cache.put(self.runtime_artifact) + handle.write('runtime') + handle.close() + + self.assertTrue(cache.has(self.runtime_artifact)) + + handle = cache.put(self.devel_artifact) + handle.write('devel') + handle.close() + + self.assertTrue(cache.has(self.runtime_artifact)) + self.assertTrue(cache.has(self.devel_artifact)) + + def test_put_artifacts_and_get_them_afterwards(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + + handle = cache.put(self.runtime_artifact) + handle.write('runtime') + handle.close() + + handle = cache.get(self.runtime_artifact) + stored_data = handle.read() + handle.close() + + self.assertEqual(stored_data, 'runtime') + + handle = cache.put(self.devel_artifact) + handle.write('devel') + handle.close() + + handle = cache.get(self.runtime_artifact) + stored_data = handle.read() + handle.close() + + self.assertEqual(stored_data, 'runtime') + + handle = cache.get(self.devel_artifact) + stored_data = handle.read() + handle.close() + + self.assertEqual(stored_data, 'devel') + + def test_put_check_and_get_artifact_metadata(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + + handle = cache.put_artifact_metadata(self.runtime_artifact, 'log') + handle.write('log line 1\nlog line 2\n') + handle.close() + + self.assertTrue(cache.has_artifact_metadata( + self.runtime_artifact, 'log')) + + handle = cache.get_artifact_metadata(self.runtime_artifact, 'log') + stored_metadata = handle.read() + handle.close() + + self.assertEqual(stored_metadata, 'log line 1\nlog line 2\n') + + def test_put_check_and_get_source_metadata(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + + handle = cache.put_source_metadata(self.source, 'mycachekey', 'log') + handle.write('source log line 1\nsource log line 2\n') + handle.close() + + self.assertTrue(cache.has_source_metadata( + self.source, 'mycachekey', 'log')) + + handle = cache.get_source_metadata(self.source, 'mycachekey', 'log') + stored_metadata = handle.read() + handle.close() + + self.assertEqual(stored_metadata, + 'source log line 1\nsource log line 2\n') + + def test_clears_artifact_cache(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + + handle = cache.put(self.runtime_artifact) + handle.write('runtime') + handle.close() + + self.assertTrue(cache.has(self.runtime_artifact)) + cache.clear() + self.assertFalse(cache.has(self.runtime_artifact)) + + def test_put_artifacts_and_list_them_afterwards(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + + handle = cache.put(self.runtime_artifact) + handle.write('runtime') + handle.close() + + self.assertEqual(len(list(cache.list_contents())), 1) + + handle = cache.put(self.devel_artifact) + handle.write('devel') + handle.close() + + self.assertEqual(len(list(cache.list_contents())), 1) + + def test_put_artifacts_and_remove_them_afterwards(self): + cache = morphlib.localartifactcache.LocalArtifactCache(self.tempfs) + + handle = cache.put(self.runtime_artifact) + handle.write('runtime') + handle.close() + + handle = cache.put(self.devel_artifact) + handle.write('devel') + handle.close() + + key = list(cache.list_contents())[0][0] + cache.remove(key) + + self.assertEqual(len(list(cache.list_contents())), 0) diff --git a/morphlib/localrepocache.py b/morphlib/localrepocache.py new file mode 100644 index 00000000..8d2030c4 --- /dev/null +++ b/morphlib/localrepocache.py @@ -0,0 +1,237 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import logging +import os +import re +import urllib2 +import urlparse +import string +import tempfile + +import cliapp +import fs.osfs + +import morphlib + + +# urlparse.urljoin needs to know details of the URL scheme being used. +# It does not know about git:// by default, so we teach it here. +gitscheme = ['git'] +urlparse.uses_relative.extend(gitscheme) +urlparse.uses_netloc.extend(gitscheme) +urlparse.uses_params.extend(gitscheme) +urlparse.uses_query.extend(gitscheme) +urlparse.uses_fragment.extend(gitscheme) + + +def quote_url(url): + ''' Convert URIs to strings that only contain digits, letters, % and _. + + NOTE: When changing the code of this function, make sure to also apply + the same to the quote_url() function of lorry. Otherwise the git tarballs + generated by lorry may no longer be found by morph. + + ''' + valid_chars = string.digits + string.letters + '%_' + transl = lambda x: x if x in valid_chars else '_' + return ''.join([transl(x) for x in url]) + + +class NoRemote(morphlib.Error): + + def __init__(self, reponame, errors): + self.reponame = reponame + self.errors = errors + + def __str__(self): + return '\n\t'.join(['Cannot find remote git repository: %s' % + self.reponame] + self.errors) + + +class NotCached(morphlib.Error): + def __init__(self, reponame): + self.reponame = reponame + + def __str__(self): # pragma: no cover + return 'Repository %s is not cached yet' % self.reponame + + +class LocalRepoCache(object): + + '''Manage locally cached git repositories. + + When we build stuff, we need a local copy of the git repository. + To avoid having to clone the repositories for every build, we + maintain a local cache of the repositories: we first clone the + remote repository to the cache, and then make a local clone from + the cache to the build environment. This class manages the local + cached repositories. + + Repositories may be specified either using a full URL, in a form + understood by git(1), or as a repository name to which a base url + is prepended. The base urls are given to the class when it is + created. + + Instead of cloning via a normal 'git clone' directly from the + git server, we first try to download a tarball from a url, and + if that works, we unpack the tarball. + + ''' + + def __init__(self, app, cachedir, resolver, tarball_base_url=None): + self._app = app + self.fs = fs.osfs.OSFS('/') + self._cachedir = cachedir + self._resolver = resolver + if tarball_base_url and not tarball_base_url.endswith('/'): + tarball_base_url += '/' # pragma: no cover + self._tarball_base_url = tarball_base_url + self._cached_repo_objects = {} + + def _git(self, args, cwd=None): # pragma: no cover + '''Execute git command. + + This is a method of its own so that unit tests can easily override + all use of the external git command. + + ''' + + morphlib.git.gitcmd(self._app.runcmd, *args, cwd=cwd) + + def _fetch(self, url, path): # pragma: no cover + '''Fetch contents of url into a file. + + This method is meant to be overridden by unit tests. + + ''' + self._app.status(msg="Trying to fetch %(tarball)s to seed the cache", + tarball=url, chatty=True) + self._app.runcmd(['wget', '-q', '-O-', url], + ['tar', 'xf', '-'], cwd=path) + + def _mkdtemp(self, dirname): # pragma: no cover + '''Creates a temporary directory. + + This method is meant to be overridden by unit tests. + + ''' + return tempfile.mkdtemp(dir=dirname) + + def _escape(self, url): + '''Escape a URL so it can be used as a basename in a file.''' + + # FIXME: The following is a nicer way than to do this. + # However, for compatibility, we need to use the same as the + # tarball server (set up by Lorry) uses. + # return urllib.quote(url, safe='') + + return quote_url(url) + + def _cache_name(self, url): + scheme, netloc, path, query, fragment = urlparse.urlsplit(url) + if scheme != 'file': + path = os.path.join(self._cachedir, self._escape(url)) + return path + + def has_repo(self, reponame): + '''Have we already got a cache of a given repo?''' + url = self._resolver.pull_url(reponame) + path = self._cache_name(url) + return self.fs.exists(path) + + def _clone_with_tarball(self, repourl, path): + tarball_url = urlparse.urljoin(self._tarball_base_url, + self._escape(repourl)) + '.tar' + try: + self.fs.makedir(path) + self._fetch(tarball_url, path) + self._git(['config', 'remote.origin.url', repourl], cwd=path) + self._git(['config', 'remote.origin.mirror', 'true'], cwd=path) + self._git(['config', 'remote.origin.fetch', '+refs/*:refs/*'], + cwd=path) + except BaseException, e: # pragma: no cover + if self.fs.exists(path): + self.fs.removedir(path, force=True) + return False, 'Unable to extract tarball %s: %s' % ( + tarball_url, e) + + return True, None + + def cache_repo(self, reponame): + '''Clone the given repo into the cache. + + If the repo is already cloned, do nothing. + + ''' + errors = [] + if not self.fs.exists(self._cachedir): + self.fs.makedir(self._cachedir, recursive=True) + + try: + return self.get_repo(reponame) + except NotCached, e: + pass + + repourl = self._resolver.pull_url(reponame) + path = self._cache_name(repourl) + if self._tarball_base_url: + ok, error = self._clone_with_tarball(repourl, path) + if ok: + return self.get_repo(reponame) + else: + errors.append(error) + self._app.status( + msg='Failed to fetch tarball, falling back to git clone.') + target = self._mkdtemp(self._cachedir) + try: + self._git(['clone', '--mirror', '-n', repourl, target]) + except cliapp.AppException, e: + errors.append('Unable to clone from %s to %s: %s' % + (repourl, target, e)) + if self.fs.exists(target): + self.fs.removedir(target, recursive=True, force=True) + raise NoRemote(reponame, errors) + + self.fs.rename(target, path) + return self.get_repo(reponame) + + def get_repo(self, reponame): + '''Return an object representing a cached repository.''' + + if reponame in self._cached_repo_objects: + return self._cached_repo_objects[reponame] + else: + repourl = self._resolver.pull_url(reponame) + path = self._cache_name(repourl) + if self.fs.exists(path): + repo = morphlib.cachedrepo.CachedRepo(self._app, reponame, + repourl, path) + self._cached_repo_objects[reponame] = repo + return repo + raise NotCached(reponame) + + def get_updated_repo(self, reponame): # pragma: no cover + '''Return object representing cached repository, which is updated.''' + + self._app.status(msg='Updating git repository %s in cache' % reponame) + if not self._app.settings['no-git-update']: + cached_repo = self.cache_repo(reponame) + cached_repo.update() + else: + cached_repo = self.get_repo(reponame) + return cached_repo + diff --git a/morphlib/localrepocache_tests.py b/morphlib/localrepocache_tests.py new file mode 100644 index 00000000..22b5ea54 --- /dev/null +++ b/morphlib/localrepocache_tests.py @@ -0,0 +1,160 @@ +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import unittest +import urllib2 +import os + +import cliapp +import fs.memoryfs + +import morphlib + + +class FakeApplication(object): + + def status(self, msg): + pass + + +class LocalRepoCacheTests(unittest.TestCase): + + def setUp(self): + aliases = ['upstream=git://example.com/#example.com:%s.git'] + repo_resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases) + tarball_base_url = 'http://lorry.example.com/tarballs/' + self.reponame = 'upstream:reponame' + self.repourl = 'git://example.com/reponame' + escaped_url = 'git___example_com_reponame' + self.tarball_url = '%s%s.tar' % (tarball_base_url, escaped_url) + self.cachedir = '/cache/dir' + self.cache_path = '%s/%s' % (self.cachedir, escaped_url) + self.remotes = {} + self.fetched = [] + self.removed = [] + self.lrc = morphlib.localrepocache.LocalRepoCache( + FakeApplication(), self.cachedir, repo_resolver, tarball_base_url) + self.lrc.fs = fs.memoryfs.MemoryFS() + self.lrc._git = self.fake_git + self.lrc._fetch = self.not_found + self.lrc._mkdtemp = self.fake_mkdtemp + self._mkdtemp_count = 0 + + def fake_git(self, args, cwd=None): + if args[0] == 'clone': + self.assertEqual(len(args), 5) + remote = args[3] + local = args[4] + self.remotes['origin'] = {'url': remote, 'updates': 0} + self.lrc.fs.makedir(local, recursive=True) + elif args[0:2] == ['remote', 'set-url']: + remote = args[2] + url = args[3] + self.remotes[remote] = {'url': url} + elif args[0:2] == ['config', 'remote.origin.url']: + remote = 'origin' + url = args[2] + self.remotes[remote] = {'url': url} + elif args[0:2] == ['config', 'remote.origin.mirror']: + remote = 'origin' + elif args[0:2] == ['config', 'remote.origin.fetch']: + remote = 'origin' + else: + raise NotImplementedError() + + def fake_mkdtemp(self, dirname): + thing = "foo"+str(self._mkdtemp_count) + self._mkdtemp_count += 1 + self.lrc.fs.makedir(dirname+"/"+thing) + return thing + + def not_found(self, url, path): + raise cliapp.AppException('Not found') + + def test_has_not_got_shortened_repo_initially(self): + self.assertFalse(self.lrc.has_repo(self.reponame)) + + def test_has_not_got_absolute_repo_initially(self): + self.assertFalse(self.lrc.has_repo(self.repourl)) + + def test_caches_shortened_repository_on_request(self): + self.lrc.cache_repo(self.reponame) + self.assertTrue(self.lrc.has_repo(self.reponame)) + self.assertTrue(self.lrc.has_repo(self.repourl)) + + def test_caches_absolute_repository_on_request(self): + self.lrc.cache_repo(self.repourl) + self.assertTrue(self.lrc.has_repo(self.reponame)) + self.assertTrue(self.lrc.has_repo(self.repourl)) + + def test_cachedir_does_not_exist_initially(self): + self.assertFalse(self.lrc.fs.exists(self.cachedir)) + + def test_creates_cachedir_if_missing(self): + self.lrc.cache_repo(self.repourl) + self.assertTrue(self.lrc.fs.exists(self.cachedir)) + + def test_happily_caches_same_repo_twice(self): + self.lrc.cache_repo(self.repourl) + self.lrc.cache_repo(self.repourl) + + def test_fails_to_cache_when_remote_does_not_exist(self): + def fail(args): + self.lrc.fs.makedir(args[4]) + raise cliapp.AppException('') + self.lrc._git = fail + self.assertRaises(morphlib.localrepocache.NoRemote, + self.lrc.cache_repo, self.repourl) + + def test_does_not_mind_a_missing_tarball(self): + self.lrc.cache_repo(self.repourl) + self.assertEqual(self.fetched, []) + + def test_fetches_tarball_when_it_exists(self): + self.lrc._fetch = lambda url, path: self.fetched.append(url) + self.unpacked_tar = "" + self.mkdir_path = "" + self.lrc.cache_repo(self.repourl) + self.assertEqual(self.fetched, [self.tarball_url]) + self.assertFalse(self.lrc.fs.exists(self.cache_path + '.tar')) + self.assertEqual(self.remotes['origin']['url'], self.repourl) + + def test_gets_cached_shortened_repo(self): + self.lrc.cache_repo(self.reponame) + cached = self.lrc.get_repo(self.reponame) + self.assertTrue(cached is not None) + + def test_gets_cached_absolute_repo(self): + self.lrc.cache_repo(self.repourl) + cached = self.lrc.get_repo(self.repourl) + self.assertTrue(cached is not None) + + def test_get_repo_raises_exception_if_repo_is_not_cached(self): + self.assertRaises(Exception, self.lrc.get_repo, self.repourl) + + def test_escapes_repourl_as_filename(self): + escaped = self.lrc._escape(self.repourl) + self.assertFalse('/' in escaped) + + def test_noremote_error_message_contains_repo_name(self): + e = morphlib.localrepocache.NoRemote(self.repourl, []) + self.assertTrue(self.repourl in str(e)) + + def test_avoids_caching_local_repo(self): + self.lrc.fs.makedir('/local/repo', recursive=True) + self.lrc.cache_repo('file:///local/repo') + cached = self.lrc.get_repo('file:///local/repo') + assert cached.path == '/local/repo' diff --git a/morphlib/morphloader.py b/morphlib/morphloader.py new file mode 100644 index 00000000..8289b01e --- /dev/null +++ b/morphlib/morphloader.py @@ -0,0 +1,789 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import collections +import logging +import warnings +import yaml + +import morphlib + + +class MorphologyObsoleteFieldWarning(UserWarning): + + def __init__(self, morphology, spec, field): + self.kind = morphology['kind'] + self.morphology_name = morphology.get('name', '<unknown>') + self.stratum_name = spec.get('alias', spec['morph']) + self.field = field + + def __str__(self): + format_string = ('%(kind)s morphology %(morphology_name)s refers to ' + 'stratum %(stratum_name)s with the %(field)s field. ' + 'Defaulting to null.') + return format_string % self.__dict__ + + +class MorphologySyntaxError(morphlib.Error): + pass + + +class MorphologyNotYamlError(MorphologySyntaxError): + + def __init__(self, morphology, errmsg): + self.msg = 'Syntax error in morphology %s:\n%s' % (morphology, errmsg) + + +class NotADictionaryError(MorphologySyntaxError): + + def __init__(self, morph_filename): + self.msg = 'Not a dictionary: morphology %s' % morph_filename + + +class MorphologyValidationError(morphlib.Error): + pass + + +class UnknownKindError(MorphologyValidationError): + + def __init__(self, kind, morph_filename): + self.msg = ( + 'Unknown kind %s in morphology %s' % (kind, morph_filename)) + + +class MissingFieldError(MorphologyValidationError): + + def __init__(self, field, morphology_name): + self.field = field + self.morphology_name = morphology_name + self.msg = ( + 'Missing field %s from morphology %s' % (field, morphology_name)) + + +class InvalidFieldError(MorphologyValidationError): + + def __init__(self, field, morphology_name): + self.field = field + self.morphology_name = morphology_name + self.msg = ( + 'Field %s not allowed in morphology %s' % (field, morphology_name)) + + +class InvalidTypeError(MorphologyValidationError): + + def __init__(self, field, expected, actual, morphology_name): + self.field = field + self.expected = expected + self.actual = actual + self.morphology_name = morphology_name + self.msg = ( + 'Field %s expected type %s, got %s in morphology %s' % + (field, expected, actual, morphology_name)) + + +class ObsoleteFieldsError(MorphologyValidationError): + + def __init__(self, fields, morph_filename): + self.msg = ( + 'Morphology %s uses obsolete fields: %s' % + (morph_filename, ' '.join(fields))) + + +class UnknownArchitectureError(MorphologyValidationError): + + def __init__(self, arch, morph_filename): + self.msg = ('Unknown architecture %s in morphology %s' + % (arch, morph_filename)) + + +class NoBuildDependenciesError(MorphologyValidationError): + + def __init__(self, stratum_name, chunk_name, morph_filename): + self.msg = ( + 'Stratum %s has no build dependencies for chunk %s in %s' % + (stratum_name, chunk_name, morph_filename)) + + +class NoStratumBuildDependenciesError(MorphologyValidationError): + + def __init__(self, stratum_name, morph_filename): + self.msg = ( + 'Stratum %s has no build dependencies in %s' % + (stratum_name, morph_filename)) + + +class EmptyStratumError(MorphologyValidationError): + + def __init__(self, stratum_name, morph_filename): + self.msg = ( + 'Stratum %s has no chunks in %s' % + (stratum_name, morph_filename)) + + +class DuplicateChunkError(MorphologyValidationError): + + def __init__(self, stratum_name, chunk_name): + self.stratum_name = stratum_name + self.chunk_name = chunk_name + MorphologyValidationError.__init__( + self, 'Duplicate chunk %(chunk_name)s '\ + 'in stratum %(stratum_name)s' % locals()) + + +class EmptyRefError(MorphologyValidationError): + + def __init__(self, ref_location, morph_filename): + self.ref_location = ref_location + self.morph_filename = morph_filename + MorphologyValidationError.__init__( + self, 'Empty ref found for %(ref_location)s '\ + 'in %(morph_filename)s' % locals()) + + +class ChunkSpecRefNotStringError(MorphologyValidationError): + + def __init__(self, ref_value, chunk_name, stratum_name): + self.ref_value = ref_value + self.chunk_name = chunk_name + self.stratum_name = stratum_name + MorphologyValidationError.__init__( + self, 'Ref %(ref_value)s for %(chunk_name)s '\ + 'in stratum %(stratum_name)s is not a string' % locals()) + + +class SystemStrataNotListError(MorphologyValidationError): + + def __init__(self, system_name, strata_type): + self.system_name = system_name + self.strata_type = strata_type + typename = strata_type.__name__ + MorphologyValidationError.__init__( + self, 'System %(system_name)s has the wrong type for its strata: '\ + '%(typename)s, expected list' % locals()) + + +class DuplicateStratumError(MorphologyValidationError): + + def __init__(self, system_name, stratum_name): + self.system_name = system_name + self.stratum_name = stratum_name + MorphologyValidationError.__init__( + self, 'Duplicate stratum %(stratum_name)s '\ + 'in system %(system_name)s' % locals()) + + +class SystemStratumSpecsNotMappingError(MorphologyValidationError): + + def __init__(self, system_name, strata): + self.system_name = system_name + self.strata = strata + MorphologyValidationError.__init__( + self, 'System %(system_name)s has stratum specs '\ + 'that are not mappings.' % locals()) + + +class EmptySystemError(MorphologyValidationError): + + def __init__(self, system_name): + MorphologyValidationError.__init__( + self, 'System %(system_name)s has no strata.' % locals()) + + +class MultipleValidationErrors(MorphologyValidationError): + + def __init__(self, name, errors): + self.name = name + self.errors = errors + self.msg = 'Multiple errors when validating %(name)s:' + for error in errors: + self.msg += ('\n' + str(error)) + + +class DuplicateDeploymentNameError(MorphologyValidationError): + + def __init__(self, cluster_filename, duplicates): + self.duplicates = duplicates + self.cluster_filename = cluster_filename + morphlib.Error.__init__(self, + 'Cluster %s contains the following duplicate deployment names:%s' + % (cluster_filename, '\n ' + '\n '.join(duplicates))) + + +class MorphologyDumper(yaml.SafeDumper): + keyorder = ( + 'name', + 'kind', + 'description', + 'arch', + 'strata', + 'configuration-extensions', + 'morph', + 'repo', + 'ref', + 'unpetrify-ref', + 'build-depends', + 'build-mode', + 'artifacts', + 'max-jobs', + 'products', + 'chunks', + 'build-system', + 'pre-configure-commands', + 'configure-commands', + 'post-configure-commands', + 'pre-build-commands', + 'build-commands', + 'post-build-commands', + 'pre-install-commands', + 'install-commands', + 'post-install-commands', + 'artifact', + 'include', + 'systems', + 'deploy-defaults', + 'deploy', + 'type', + 'location', + ) + + @classmethod + def _iter_in_global_order(cls, mapping): + for key in cls.keyorder: + if key in mapping: + yield key, mapping[key] + for key in sorted(mapping.iterkeys()): + if key not in cls.keyorder: + yield key, mapping[key] + + @classmethod + def _represent_dict(cls, dumper, mapping): + return dumper.represent_mapping('tag:yaml.org,2002:map', + cls._iter_in_global_order(mapping)) + + @classmethod + def _represent_str(cls, dumper, orig_data): + fallback_representer = yaml.representer.SafeRepresenter.represent_str + try: + data = unicode(orig_data, 'ascii') + if data.count('\n') == 0: + return fallback_representer(dumper, orig_data) + except UnicodeDecodeError: + try: + data = unicode(orig_data, 'utf-8') + if data.count('\n') == 0: + return fallback_representer(dumper, orig_data) + except UnicodeDecodeError: + return fallback_representer(dumper, orig_data) + return dumper.represent_scalar(u'tag:yaml.org,2002:str', + data, style='|') + + @classmethod + def _represent_unicode(cls, dumper, data): + if data.count('\n') == 0: + return yaml.representer.SafeRepresenter.represent_unicode(dumper, + data) + return dumper.represent_scalar(u'tag:yaml.org,2002:str', + data, style='|') + + def __init__(self, *args, **kwargs): + yaml.SafeDumper.__init__(self, *args, **kwargs) + self.add_representer(dict, self._represent_dict) + self.add_representer(str, self._represent_str) + self.add_representer(unicode, self._represent_unicode) + + +class MorphologyLoader(object): + + '''Load morphologies from disk, or save them back to disk.''' + + _required_fields = { + 'chunk': [ + 'name', + ], + 'stratum': [ + 'name', + ], + 'system': [ + 'name', + 'arch', + 'strata', + ], + 'cluster': [ + 'name', + 'systems', + ], + } + + _obsolete_fields = { + 'system': [ + 'system-kind', + 'disk-size', + ], + } + + _static_defaults = { + 'chunk': { + 'description': '', + 'pre-configure-commands': None, + 'configure-commands': None, + 'post-configure-commands': None, + 'pre-build-commands': None, + 'build-commands': None, + 'post-build-commands': None, + 'pre-test-commands': None, + 'test-commands': None, + 'post-test-commands': None, + 'pre-install-commands': None, + 'install-commands': None, + 'post-install-commands': None, + 'devices': [], + 'products': [], + 'max-jobs': None, + 'build-system': 'manual', + 'build-mode': 'staging', + 'prefix': '/usr', + 'system-integration': [], + }, + 'stratum': { + 'chunks': [], + 'description': '', + 'build-depends': [], + 'products': [], + }, + 'system': { + 'description': '', + 'arch': None, + 'configuration-extensions': [], + }, + 'cluster': { + 'description': '', + }, + } + + def parse_morphology_text(self, text, morph_filename): + '''Parse a textual morphology. + + The text may be a string, or an open file handle. + + Return the new Morphology object, or raise an error indicating + the problem. This method does minimal validation: a syntactically + correct morphology is fine, even if none of the fields are + valid. It also does not set any default values for any of the + fields. See validate and set_defaults. + + ''' + + try: + obj = yaml.safe_load(text) + except yaml.error.YAMLError as e: + raise MorphologyNotYamlError(morph_filename, e) + + if not isinstance(obj, dict): + raise NotADictionaryError(morph_filename) + + return morphlib.morphology.Morphology(obj) + + def load_from_string(self, string, filename='string'): + '''Load a morphology from a string. + + Return the Morphology object. + + ''' + + m = self.parse_morphology_text(string, filename) + m.filename = filename + self.validate(m) + self.set_commands(m) + self.set_defaults(m) + return m + + def load_from_file(self, filename): + '''Load a morphology from a named file. + + Return the Morphology object. + + ''' + + with open(filename) as f: + text = f.read() + return self.load_from_string(text, filename=filename) + + def save_to_string(self, morphology): + '''Return normalised textual form of morphology.''' + + return yaml.dump(morphology.data, Dumper=MorphologyDumper, + default_flow_style=False) + + def save_to_file(self, filename, morphology): + '''Save a morphology object to a named file.''' + + text = self.save_to_string(morphology) + with morphlib.savefile.SaveFile(filename, 'w') as f: + f.write(text) + + def validate(self, morph): + '''Validate a morphology.''' + + # Validate that the kind field is there. + self._require_field('kind', morph) + + # The rest of the validation is dependent on the kind. + kind = morph['kind'] + if kind not in ('system', 'stratum', 'chunk', 'cluster'): + raise UnknownKindError(morph['kind'], morph.filename) + + required = ['kind'] + self._required_fields[kind] + obsolete = self._obsolete_fields.get(kind, []) + allowed = self._static_defaults[kind].keys() + self._require_fields(required, morph) + self._deny_obsolete_fields(obsolete, morph) + self._deny_unknown_fields(required + allowed, morph) + + getattr(self, '_validate_%s' % kind)(morph) + + def _validate_cluster(self, morph): + # Deployment names must be unique within a cluster + deployments = collections.Counter() + for system in morph['systems']: + deployments.update(system['deploy'].iterkeys()) + if 'subsystems' in system: + deployments.update(self._get_subsystem_names(system)) + duplicates = set(deployment for deployment, count + in deployments.iteritems() if count > 1) + if duplicates: + raise DuplicateDeploymentNameError(morph.filename, duplicates) + + def _get_subsystem_names(self, system): # pragma: no cover + for subsystem in system.get('subsystems', []): + for name in subsystem['deploy'].iterkeys(): + yield name + for name in self._get_subsystem_names(subsystem): + yield name + + def _validate_system(self, morph): + # A system must contain at least one stratum + strata = morph['strata'] + if (not isinstance(strata, collections.Iterable) + or isinstance(strata, collections.Mapping)): + + raise SystemStrataNotListError(morph['name'], + type(strata)) + + if not strata: + raise EmptySystemError(morph['name']) + + if not all(isinstance(o, collections.Mapping) for o in strata): + raise SystemStratumSpecsNotMappingError(morph['name'], strata) + + # All stratum names should be unique within a system. + names = set() + for spec in strata: + name = spec.get('alias', spec['morph']) + if name in names: + raise DuplicateStratumError(morph['name'], name) + names.add(name) + + # Validate stratum spec fields + self._validate_stratum_specs_fields(morph, 'strata') + + # We allow the ARMv7 little-endian architecture to be specified + # as armv7 and armv7l. Normalise. + if morph['arch'] == 'armv7': + morph['arch'] = 'armv7l' + + # Architecture name must be known. + if morph['arch'] not in morphlib.valid_archs: + raise UnknownArchitectureError(morph['arch'], morph.filename) + + def _validate_stratum(self, morph): + # Require at least one chunk. + if len(morph.get('chunks', [])) == 0: + raise EmptyStratumError(morph['name'], morph.filename) + + # All chunk names must be unique within a stratum. + names = set() + for spec in morph['chunks']: + name = spec.get('alias', spec['name']) + if name in names: + raise DuplicateChunkError(morph['name'], name) + names.add(name) + + # All chunk refs must be strings. + for spec in morph['chunks']: + if 'ref' in spec: + ref = spec['ref'] + if ref == None: + raise EmptyRefError( + spec.get('alias', spec['name']), morph.filename) + elif not isinstance(ref, basestring): + raise ChunkSpecRefNotStringError( + ref, spec.get('alias', spec['name']), morph.filename) + + # Require build-dependencies for the stratum itself, unless + # it has chunks built in bootstrap mode. + if 'build-depends' in morph: + if not isinstance(morph['build-depends'], list): + raise InvalidTypeError( + 'build-depends', list, type(morph['build-depends']), + morph['name']) + else: + for spec in morph['chunks']: + if spec.get('build-mode') in ['bootstrap', 'test']: + break + else: + raise NoStratumBuildDependenciesError( + morph['name'], morph.filename) + + # Validate build-dependencies if specified + self._validate_stratum_specs_fields(morph, 'build-depends') + + # Require build-dependencies for each chunk. + for spec in morph['chunks']: + chunk_name = spec.get('alias', spec['name']) + if 'build-depends' in spec: + if not isinstance(spec['build-depends'], list): + raise InvalidTypeError( + '%s.build-depends' % chunk_name, list, + type(spec['build-depends']), morph['name']) + else: + raise NoBuildDependenciesError( + morph['name'], chunk_name, morph.filename) + + @classmethod + def _validate_chunk(cls, morphology): + errors = [] + + if 'products' in morphology: + cls._validate_products(morphology['name'], + morphology['products'], errors) + + if len(errors) == 1: + raise errors[0] + elif errors: + raise MultipleValidationErrors(morphology['name'], errors) + + @classmethod + def _validate_products(cls, morphology_name, products, errors): + '''Validate the products field is of the correct type.''' + if (not isinstance(products, collections.Iterable) + or isinstance(products, collections.Mapping)): + raise InvalidTypeError('products', list, + type(products), morphology_name) + + for spec_index, spec in enumerate(products): + + if not isinstance(spec, collections.Mapping): + e = InvalidTypeError('products[%d]' % spec_index, + dict, type(spec), morphology_name) + errors.append(e) + continue + + cls._validate_products_spec_fields_exist(morphology_name, + spec_index, spec, errors) + + if 'include' in spec: + cls._validate_products_specs_include( + morphology_name, spec_index, spec['include'], errors) + + product_spec_required_fields = ('artifact', 'include') + @classmethod + def _validate_products_spec_fields_exist( + cls, morphology_name, spec_index, spec, errors): + + given_fields = sorted(spec.iterkeys()) + missing = (field for field in cls.product_spec_required_fields + if field not in given_fields) + for field in missing: + e = MissingFieldError('products[%d].%s' % (spec_index, field), + morphology_name) + errors.append(e) + unexpected = (field for field in given_fields + if field not in cls.product_spec_required_fields) + for field in unexpected: + e = InvalidFieldError('products[%d].%s' % (spec_index, field), + morphology_name) + errors.append(e) + + @classmethod + def _validate_products_specs_include(cls, morphology_name, spec_index, + include_patterns, errors): + '''Validate that products' include field is a list of strings.''' + # Allow include to be most iterables, but not a mapping + # or a string, since iter of a mapping is just the keys, + # and the iter of a string is a 1 character length string, + # which would also validate as an iterable of strings. + if (not isinstance(include_patterns, collections.Iterable) + or isinstance(include_patterns, collections.Mapping) + or isinstance(include_patterns, basestring)): + + e = InvalidTypeError('products[%d].include' % spec_index, list, + type(include_patterns), morphology_name) + errors.append(e) + else: + for pattern_index, pattern in enumerate(include_patterns): + pattern_path = ('products[%d].include[%d]' % + (spec_index, pattern_index)) + if not isinstance(pattern, basestring): + e = InvalidTypeError(pattern_path, str, + type(pattern), morphology_name) + errors.append(e) + + @classmethod + def _warn_obsolete_field(cls, morphology, spec, field): + warnings.warn(MorphologyObsoleteFieldWarning(morphology, spec, field), + stacklevel=2) + + @classmethod + def _validate_stratum_specs_fields(cls, morphology, specs_field): + for spec in morphology.get(specs_field, None) or []: + for obsolete_field in ('repo', 'ref'): + if obsolete_field in spec: + cls._warn_obsolete_field(morphology, spec, obsolete_field) + + def _require_field(self, field, morphology): + if field not in morphology: + raise MissingFieldError(field, morphology.filename) + + def _require_fields(self, fields, morphology): + for field in fields: + self._require_field(field, morphology) + + def _deny_obsolete_fields(self, fields, morphology): + obsolete_ones = [x for x in morphology if x in fields] + if obsolete_ones: + raise ObsoleteFieldsError(obsolete_ones, morphology.filename) + + def _deny_unknown_fields(self, allowed, morphology): + for field in morphology: + if field not in allowed: + raise InvalidFieldError(field, morphology.filename) + + def set_defaults(self, morphology): + '''Set all missing fields in the morpholoy to their defaults. + + The morphology is assumed to be valid. + + ''' + + kind = morphology['kind'] + defaults = self._static_defaults[kind] + for key in defaults: + if key not in morphology: + morphology[key] = defaults[key] + + getattr(self, '_set_%s_defaults' % kind)(morphology) + + def unset_defaults(self, morphology): + '''If a field is equal to its default, delete it. + + The morphology is assumed to be valid. + + ''' + + kind = morphology['kind'] + defaults = self._static_defaults[kind] + for key in defaults: + if key in morphology and morphology[key] == defaults[key]: + del morphology[key] + + getattr(self, '_unset_%s_defaults' % kind)(morphology) + + @classmethod + def _set_stratum_specs_defaults(cls, morphology, specs_field): + for spec in morphology.get(specs_field, None) or []: + for obsolete_field in ('repo', 'ref'): + if obsolete_field in spec: + del spec[obsolete_field] + + @classmethod + def _unset_stratum_specs_defaults(cls, morphology, specs_field): + for spec in morphology.get(specs_field, []): + for obsolete_field in ('repo', 'ref'): + if obsolete_field in spec: + del spec[obsolete_field] + + def _set_cluster_defaults(self, morph): + for system in morph.get('systems', []): + if 'deploy-defaults' not in system: + system['deploy-defaults'] = {} + if 'deploy' not in system: + system['deploy'] = {} + + def _unset_cluster_defaults(self, morph): + for system in morph.get('systems', []): + if 'deploy-defaults' in system and system['deploy-defaults'] == {}: + del system['deploy-defaults'] + if 'deploy' in system and system['deploy'] == {}: + del system['deploy'] + + def _set_system_defaults(self, morph): + self._set_stratum_specs_defaults(morph, 'strata') + + def _unset_system_defaults(self, morph): + self._unset_stratum_specs_defaults(morph, 'strata') + + def _set_stratum_defaults(self, morph): + for spec in morph['chunks']: + if 'repo' not in spec: + spec['repo'] = spec['name'] + if 'build-mode' not in spec: + spec['build-mode'] = \ + self._static_defaults['chunk']['build-mode'] + if 'prefix' not in spec: + spec['prefix'] = \ + self._static_defaults['chunk']['prefix'] + self._set_stratum_specs_defaults(morph, 'build-depends') + + def _unset_stratum_defaults(self, morph): + for spec in morph['chunks']: + if 'repo' in spec and spec['repo'] == spec['name']: + del spec['repo'] + if 'build-mode' in spec and spec['build-mode'] == \ + self._static_defaults['chunk']['build-mode']: + del spec['build-mode'] + if 'prefix' in spec and spec['prefix'] == \ + self._static_defaults['chunk']['prefix']: + del spec['prefix'] + self._unset_stratum_specs_defaults(morph, 'strata') + + def _set_chunk_defaults(self, morph): + if morph['max-jobs'] is not None: + morph['max-jobs'] = int(morph['max-jobs']) + + def _unset_chunk_defaults(self, morph): # pragma: no cover + for key in self._static_defaults['chunk']: + if key not in morph: continue + if 'commands' not in key: continue + attr = key.replace('-', '_') + default_bs = self._static_defaults['chunk']['build-system'] + bs = morphlib.buildsystem.lookup_build_system( + morph.get('build-system', default_bs)) + default_value = getattr(bs, attr) + if morph[key] == default_value: + del morph[key] + + def set_commands(self, morph): + if morph['kind'] == 'chunk': + for key in self._static_defaults['chunk']: + if 'commands' not in key: continue + if key not in morph: + attr = '_'.join(key.split('-')) + default = self._static_defaults['chunk']['build-system'] + bs = morphlib.buildsystem.lookup_build_system( + morph.get('build-system', default)) + morph[key] = getattr(bs, attr) diff --git a/morphlib/morphloader_tests.py b/morphlib/morphloader_tests.py new file mode 100644 index 00000000..dd70c824 --- /dev/null +++ b/morphlib/morphloader_tests.py @@ -0,0 +1,989 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import contextlib +import os +import shutil +import tempfile +import unittest +import warnings + +import morphlib +from morphlib.morphloader import MorphologyObsoleteFieldWarning + + +class MorphologyLoaderTests(unittest.TestCase): + + def setUp(self): + self.loader = morphlib.morphloader.MorphologyLoader() + self.tempdir = tempfile.mkdtemp() + self.filename = os.path.join(self.tempdir, 'foo.morph') + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_parses_yaml_from_string(self): + string = '''\ +name: foo +kind: chunk +build-system: dummy +''' + morph = self.loader.parse_morphology_text(string, 'test') + self.assertEqual(morph['kind'], 'chunk') + self.assertEqual(morph['name'], 'foo') + self.assertEqual(morph['build-system'], 'dummy') + + def test_fails_to_parse_utter_garbage(self): + self.assertRaises( + morphlib.morphloader.MorphologySyntaxError, + self.loader.parse_morphology_text, ',,,', 'test') + + def test_fails_to_parse_non_dict(self): + self.assertRaises( + morphlib.morphloader.NotADictionaryError, + self.loader.parse_morphology_text, '- item1\n- item2\n', 'test') + + def test_fails_to_validate_dict_without_kind(self): + m = morphlib.morphology.Morphology({ + 'invalid': 'field', + }) + self.assertRaises( + morphlib.morphloader.MissingFieldError, self.loader.validate, m) + + def test_fails_to_validate_chunk_with_no_fields(self): + m = morphlib.morphology.Morphology({ + 'kind': 'chunk', + }) + self.assertRaises( + morphlib.morphloader.MissingFieldError, self.loader.validate, m) + + def test_fails_to_validate_chunk_with_invalid_field(self): + m = morphlib.morphology.Morphology({ + 'kind': 'chunk', + 'name': 'foo', + 'invalid': 'field', + }) + self.assertRaises( + morphlib.morphloader.InvalidFieldError, self.loader.validate, m) + + def test_validate_requires_products_list(self): + m = morphlib.morphology.Morphology( + kind='chunk', + name='foo', + products={ + 'foo-runtime': ['.'], + 'foo-devel': ['.'], + }) + with self.assertRaises(morphlib.morphloader.InvalidTypeError) as cm: + self.loader.validate(m) + e = cm.exception + self.assertEqual(e.field, 'products') + self.assertEqual(e.expected, list) + self.assertEqual(e.actual, dict) + self.assertEqual(e.morphology_name, 'foo') + + def test_validate_requires_products_list_of_mappings(self): + m = morphlib.morphology.Morphology( + kind='chunk', + name='foo', + products=[ + 'foo-runtime', + ]) + with self.assertRaises(morphlib.morphloader.InvalidTypeError) as cm: + self.loader.validate(m) + e = cm.exception + self.assertEqual(e.field, 'products[0]') + self.assertEqual(e.expected, dict) + self.assertEqual(e.actual, str) + self.assertEqual(e.morphology_name, 'foo') + + def test_validate_requires_products_list_required_fields(self): + m = morphlib.morphology.Morphology( + kind='chunk', + name='foo', + products=[ + { + 'factiart': 'foo-runtime', + 'cludein': [], + } + ]) + with self.assertRaises(morphlib.morphloader.MultipleValidationErrors) \ + as cm: + self.loader.validate(m) + exs = cm.exception.errors + self.assertEqual(type(exs[0]), morphlib.morphloader.MissingFieldError) + self.assertEqual(exs[0].field, 'products[0].artifact') + self.assertEqual(type(exs[1]), morphlib.morphloader.MissingFieldError) + self.assertEqual(exs[1].field, 'products[0].include') + self.assertEqual(type(exs[2]), morphlib.morphloader.InvalidFieldError) + self.assertEqual(exs[2].field, 'products[0].cludein') + self.assertEqual(type(exs[3]), morphlib.morphloader.InvalidFieldError) + self.assertEqual(exs[3].field, 'products[0].factiart') + + def test_validate_requires_products_list_include_is_list(self): + m = morphlib.morphology.Morphology( + kind='chunk', + name='foo', + products=[ + { + 'artifact': 'foo-runtime', + 'include': '.*', + } + ]) + with self.assertRaises(morphlib.morphloader.InvalidTypeError) as cm: + self.loader.validate(m) + ex = cm.exception + self.assertEqual(ex.field, 'products[0].include') + self.assertEqual(ex.expected, list) + self.assertEqual(ex.actual, str) + self.assertEqual(ex.morphology_name, 'foo') + + def test_validate_requires_products_list_include_is_list_of_strings(self): + m = morphlib.morphology.Morphology( + kind='chunk', + name='foo', + products=[ + { + 'artifact': 'foo-runtime', + 'include': [ + 123, + ] + } + ]) + with self.assertRaises(morphlib.morphloader.InvalidTypeError) as cm: + self.loader.validate(m) + ex = cm.exception + self.assertEqual(ex.field, 'products[0].include[0]') + self.assertEqual(ex.expected, str) + self.assertEqual(ex.actual, int) + self.assertEqual(ex.morphology_name, 'foo') + + + def test_fails_to_validate_stratum_with_no_fields(self): + m = morphlib.morphology.Morphology({ + 'kind': 'stratum', + }) + self.assertRaises( + morphlib.morphloader.MissingFieldError, self.loader.validate, m) + + def test_fails_to_validate_stratum_with_invalid_field(self): + m = morphlib.morphology.Morphology({ + 'kind': 'stratum', + 'name': 'foo', + 'invalid': 'field', + }) + self.assertRaises( + morphlib.morphloader.InvalidFieldError, self.loader.validate, m) + + def test_validate_requires_chunk_refs_in_stratum_to_be_strings(self): + m = morphlib.morphology.Morphology({ + 'kind': 'stratum', + 'name': 'foo', + 'build-depends': [], + 'chunks': [ + { + 'name': 'chunk', + 'repo': 'test:repo', + 'ref': 1, + 'build-depends': [] + } + ] + }) + with self.assertRaises( + morphlib.morphloader.ChunkSpecRefNotStringError): + self.loader.validate(m) + + def test_fails_to_validate_stratum_with_empty_refs_for_a_chunk(self): + m = morphlib.morphology.Morphology({ + 'kind': 'stratum', + 'name': 'foo', + 'build-depends': [], + 'chunks' : [ + { + 'name': 'chunk', + 'repo': 'test:repo', + 'ref': None, + 'build-depends': [] + } + ] + }) + with self.assertRaises( + morphlib.morphloader.EmptyRefError): + self.loader.validate(m) + + def test_fails_to_validate_system_with_obsolete_system_kind_field(self): + m = morphlib.morphology.Morphology({ + 'kind': 'system', + 'name': 'foo', + 'arch': 'x86_64', + 'strata': [ + {'morph': 'bar'}, + ], + 'system-kind': 'foo', + }) + self.assertRaises( + morphlib.morphloader.ObsoleteFieldsError, self.loader.validate, m) + + def test_fails_to_validate_system_with_obsolete_disk_size_field(self): + m = morphlib.morphology.Morphology({ + 'kind': 'system', + 'name': 'foo', + 'arch': 'x86_64', + 'strata': [ + {'morph': 'bar'}, + ], + 'disk-size': 'over 9000', + }) + self.assertRaises( + morphlib.morphloader.ObsoleteFieldsError, self.loader.validate, m) + + def test_fails_to_validate_system_with_no_fields(self): + m = morphlib.morphology.Morphology({ + 'kind': 'system', + }) + self.assertRaises( + morphlib.morphloader.MissingFieldError, self.loader.validate, m) + + def test_fails_to_validate_system_with_invalid_field(self): + m = morphlib.morphology.Morphology( + kind="system", + name="foo", + arch="blah", + strata=[ + {'morph': 'bar'}, + ], + invalid='field') + self.assertRaises( + morphlib.morphloader.InvalidFieldError, self.loader.validate, m) + + def test_fails_to_validate_morphology_with_unknown_kind(self): + m = morphlib.morphology.Morphology({ + 'kind': 'invalid', + }) + self.assertRaises( + morphlib.morphloader.UnknownKindError, self.loader.validate, m) + + def test_validate_requires_unique_stratum_names_within_a_system(self): + m = morphlib.morphology.Morphology( + { + "kind": "system", + "name": "foo", + "arch": "x86-64", + "strata": [ + { + "morph": "stratum", + "repo": "test1", + "ref": "ref" + }, + { + "morph": "stratum", + "repo": "test2", + "ref": "ref" + } + ] + }) + self.assertRaises(morphlib.morphloader.DuplicateStratumError, + self.loader.validate, m) + + def test_validate_requires_unique_chunk_names_within_a_stratum(self): + m = morphlib.morphology.Morphology( + { + "kind": "stratum", + "name": "foo", + "chunks": [ + { + "name": "chunk", + "repo": "test1", + "ref": "ref" + }, + { + "name": "chunk", + "repo": "test2", + "ref": "ref" + } + ] + }) + self.assertRaises(morphlib.morphloader.DuplicateChunkError, + self.loader.validate, m) + + def test_validate_requires_a_valid_architecture(self): + m = morphlib.morphology.Morphology( + kind="system", + name="foo", + arch="blah", + strata=[ + {'morph': 'bar'}, + ]) + self.assertRaises( + morphlib.morphloader.UnknownArchitectureError, + self.loader.validate, m) + + def test_validate_normalises_architecture_armv7_to_armv7l(self): + m = morphlib.morphology.Morphology( + kind="system", + name="foo", + arch="armv7", + strata=[ + {'morph': 'bar'}, + ]) + self.loader.validate(m) + self.assertEqual(m['arch'], 'armv7l') + + def test_validate_requires_build_deps_for_chunks_in_strata(self): + m = morphlib.morphology.Morphology( + { + "kind": "stratum", + "name": "foo", + "chunks": [ + { + "name": "foo", + "repo": "foo", + "ref": "foo", + "morph": "foo", + "build-mode": "bootstrap", + } + ], + }) + + self.assertRaises( + morphlib.morphloader.NoBuildDependenciesError, + self.loader.validate, m) + + def test_validate_requires_build_deps_or_bootstrap_mode_for_strata(self): + m = morphlib.morphology.Morphology( + { + "name": "stratum-no-bdeps-no-bootstrap", + "kind": "stratum", + "chunks": [ + { + "name": "chunk", + "repo": "test:repo", + "ref": "sha1", + "build-depends": [] + } + ] + }) + + self.assertRaises( + morphlib.morphloader.NoStratumBuildDependenciesError, + self.loader.validate, m) + + m['build-depends'] = [ + { + "morph": "foo", + }, + ] + self.loader.validate(m) + + del m['build-depends'] + m['chunks'][0]['build-mode'] = 'bootstrap' + self.loader.validate(m) + + def test_validate_stratum_build_deps_are_list(self): + m = morphlib.morphology.Morphology( + { + "name": "stratum-invalid-bdeps", + "kind": "stratum", + "build-depends": 0.1, + "chunks": [ + { + "name": "chunk", + "repo": "test:repo", + "ref": "sha1", + "build-depends": [] + } + ] + }) + + self.assertRaises( + morphlib.morphloader.InvalidTypeError, + self.loader.validate, m) + + def test_validate_chunk_build_deps_are_list(self): + m = morphlib.morphology.Morphology( + { + "name": "stratum-invalid-bdeps", + "kind": "stratum", + "build-depends": [ + { "morph": "foo" }, + ], + "chunks": [ + { + "name": "chunk", + "repo": "test:repo", + "ref": "sha1", + "build-depends": 0.1 + } + ] + }) + + self.assertRaises( + morphlib.morphloader.InvalidTypeError, + self.loader.validate, m) + + def test_validate_requires_chunks_in_strata(self): + m = morphlib.morphology.Morphology( + { + "name": "stratum", + "kind": "stratum", + "chunks": [ + ], + "build-depends": [ + { + "repo": "foo", + "ref": "foo", + "morph": "foo", + }, + ], + }) + + self.assertRaises( + morphlib.morphloader.EmptyStratumError, + self.loader.validate, m) + + def test_validate_requires_strata_in_system(self): + m = morphlib.morphology.Morphology( + name='system', + kind='system', + arch='testarch') + self.assertRaises( + morphlib.morphloader.MissingFieldError, + self.loader.validate, m) + + def test_validate_requires_list_of_strata_in_system(self): + for v in (None, {}): + m = morphlib.morphology.Morphology( + name='system', + kind='system', + arch='testarch', + strata=v) + with self.assertRaises( + morphlib.morphloader.SystemStrataNotListError) as cm: + + self.loader.validate(m) + self.assertEqual(cm.exception.strata_type, type(v)) + + def test_validate_requires_non_empty_strata_in_system(self): + m = morphlib.morphology.Morphology( + name='system', + kind='system', + arch='testarch', + strata=[]) + self.assertRaises( + morphlib.morphloader.EmptySystemError, + self.loader.validate, m) + + def test_validate_requires_stratum_specs_in_system(self): + m = morphlib.morphology.Morphology( + name='system', + kind='system', + arch='testarch', + strata=["foo"]) + with self.assertRaises( + morphlib.morphloader.SystemStratumSpecsNotMappingError) as cm: + + self.loader.validate(m) + self.assertEqual(cm.exception.strata, ["foo"]) + + def test_validate_requires_unique_deployment_names_in_cluster(self): + subsystem = [{'morph': 'baz', 'deploy': {'foobar': None}}] + m = morphlib.morphology.Morphology( + name='cluster', + kind='cluster', + systems=[{'morph': 'foo', + 'deploy': {'deployment': {}}, + 'subsystems': subsystem}, + {'morph': 'bar', + 'deploy': {'deployment': {}}, + 'subsystems': subsystem}]) + with self.assertRaises( + morphlib.morphloader.DuplicateDeploymentNameError) as cm: + self.loader.validate(m) + ex = cm.exception + self.assertIn('foobar', ex.duplicates) + self.assertIn('deployment', ex.duplicates) + + def test_loads_yaml_from_string(self): + string = '''\ +name: foo +kind: chunk +build-system: dummy +''' + morph = self.loader.load_from_string(string) + self.assertEqual(morph['kind'], 'chunk') + self.assertEqual(morph['name'], 'foo') + self.assertEqual(morph['build-system'], 'dummy') + + def test_loads_json_from_string(self): + string = '''\ +{ + "name": "foo", + "kind": "chunk", + "build-system": "dummy" +} +''' + morph = self.loader.load_from_string(string) + self.assertEqual(morph['kind'], 'chunk') + self.assertEqual(morph['name'], 'foo') + self.assertEqual(morph['build-system'], 'dummy') + + def test_loads_from_file(self): + with open(self.filename, 'w') as f: + f.write('''\ +name: foo +kind: chunk +build-system: dummy +''') + morph = self.loader.load_from_file(self.filename) + self.assertEqual(morph['kind'], 'chunk') + self.assertEqual(morph['name'], 'foo') + self.assertEqual(morph['build-system'], 'dummy') + + def test_saves_to_string(self): + morph = morphlib.morphology.Morphology({ + 'name': 'foo', + 'kind': 'chunk', + 'build-system': 'dummy', + }) + text = self.loader.save_to_string(morph) + + # The following verifies that the YAML is written in a normalised + # fashion. + self.assertEqual(text, '''\ +name: foo +kind: chunk +build-system: dummy +''') + + def test_saves_to_file(self): + morph = morphlib.morphology.Morphology({ + 'name': 'foo', + 'kind': 'chunk', + 'build-system': 'dummy', + }) + self.loader.save_to_file(self.filename, morph) + + with open(self.filename) as f: + text = f.read() + + # The following verifies that the YAML is written in a normalised + # fashion. + self.assertEqual(text, '''\ +name: foo +kind: chunk +build-system: dummy +''') + + def test_validate_does_not_set_defaults(self): + m = morphlib.morphology.Morphology({ + 'kind': 'chunk', + 'name': 'foo', + }) + self.loader.validate(m) + self.assertEqual(sorted(m.keys()), sorted(['kind', 'name'])) + + def test_sets_defaults_for_chunks(self): + m = morphlib.morphology.Morphology({ + 'kind': 'chunk', + 'name': 'foo', + }) + self.loader.set_defaults(m) + self.loader.validate(m) + self.assertEqual( + dict(m), + { + 'kind': 'chunk', + 'name': 'foo', + 'description': '', + 'build-system': 'manual', + 'build-mode': 'staging', + + 'configure-commands': None, + 'pre-configure-commands': None, + 'post-configure-commands': None, + + 'build-commands': None, + 'pre-build-commands': None, + 'post-build-commands': None, + + 'test-commands': None, + 'pre-test-commands': None, + 'post-test-commands': None, + + 'install-commands': None, + 'pre-install-commands': None, + 'post-install-commands': None, + + 'products': [], + 'system-integration': [], + 'devices': [], + 'max-jobs': None, + 'prefix': '/usr', + }) + + def test_unsets_defaults_for_chunks(self): + m = morphlib.morphology.Morphology({ + 'kind': 'chunk', + 'name': 'foo', + 'build-system': 'manual', + }) + self.loader.unset_defaults(m) + self.assertEqual( + dict(m), + { + 'kind': 'chunk', + 'name': 'foo', + }) + + def test_sets_defaults_for_strata(self): + m = morphlib.morphology.Morphology({ + 'kind': 'stratum', + 'name': 'foo', + 'chunks': [ + { + 'name': 'bar', + 'repo': 'bar', + 'ref': 'bar', + 'morph': 'bar', + 'build-mode': 'bootstrap', + 'build-depends': [], + }, + ], + }) + self.loader.set_defaults(m) + self.loader.validate(m) + self.assertEqual( + dict(m), + { + 'kind': 'stratum', + 'name': 'foo', + 'description': '', + 'build-depends': [], + 'chunks': [ + { + 'name': 'bar', + "repo": "bar", + "ref": "bar", + "morph": "bar", + 'build-mode': 'bootstrap', + 'build-depends': [], + 'prefix': '/usr', + }, + ], + 'products': [], + }) + + def test_unsets_defaults_for_strata(self): + test_dict = { + 'kind': 'stratum', + 'name': 'foo', + 'chunks': [ + { + 'name': 'bar', + "ref": "bar", + 'build-mode': 'staging', + 'build-depends': [], + 'prefix': '/usr', + }, + ], + } + test_dict_with_build_depends = dict(test_dict) + test_dict_with_build_depends["build-depends"] = [] + m = morphlib.morphology.Morphology(test_dict_with_build_depends) + self.loader.unset_defaults(m) + self.assertEqual( + dict(m), + test_dict) + + def test_sets_defaults_for_system(self): + m = morphlib.morphology.Morphology( + kind='system', + name='foo', + arch='testarch', + strata=[ + { + 'morph': 'bar', + 'repo': 'obsolete', + 'ref': 'obsolete', + }, + ]) + self.loader.set_defaults(m) + self.assertEqual( + { + 'kind': 'system', + 'name': 'foo', + 'description': '', + 'arch': 'testarch', + 'strata': [ + { + 'morph': 'bar', + }, + ], + 'configuration-extensions': [], + }, + dict(m)) + + def test_unsets_defaults_for_system(self): + m = morphlib.morphology.Morphology( + { + 'description': '', + 'kind': 'system', + 'name': 'foo', + 'arch': 'testarch', + 'strata': [ + { + 'morph': 'bar', + 'repo': None, + 'ref': None, + }, + ], + 'configuration-extensions': [], + }) + self.loader.unset_defaults(m) + self.assertEqual( + dict(m), + { + 'kind': 'system', + 'name': 'foo', + 'arch': 'testarch', + 'strata': [ + {'morph': 'bar'}, + ], + }) + + def test_sets_defaults_for_cluster(self): + m = morphlib.morphology.Morphology( + name='foo', + kind='cluster', + systems=[ + {'morph': 'foo'}, + {'morph': 'bar'}]) + self.loader.set_defaults(m) + self.loader.validate(m) + self.assertEqual(m['systems'], + [{'morph': 'foo', + 'deploy-defaults': {}, + 'deploy': {}}, + {'morph': 'bar', + 'deploy-defaults': {}, + 'deploy': {}}]) + + def test_unsets_defaults_for_cluster(self): + m = morphlib.morphology.Morphology( + name='foo', + kind='cluster', + description='', + systems=[ + {'morph': 'foo', + 'deploy-defaults': {}, + 'deploy': {}}, + {'morph': 'bar', + 'deploy-defaults': {}, + 'deploy': {}}]) + self.loader.unset_defaults(m) + self.assertNotIn('description', m) + self.assertEqual(m['systems'], + [{'morph': 'foo'}, + {'morph': 'bar'}]) + + def test_sets_stratum_chunks_repo_from_name(self): + m = morphlib.morphology.Morphology( + { + "name": "foo", + "kind": "stratum", + "chunks": [ + { + "name": "le-chunk", + "ref": "ref", + "build-depends": [], + } + ] + }) + + self.loader.set_defaults(m) + self.loader.validate(m) + self.assertEqual(m['chunks'][0]['repo'], 'le-chunk') + + def test_collapses_stratum_chunks_repo_from_name(self): + m = morphlib.morphology.Morphology( + { + "name": "foo", + "kind": "stratum", + "chunks": [ + { + "name": "le-chunk", + "repo": "le-chunk", + "morph": "le-chunk", + "ref": "ref", + "build-depends": [], + } + ] + }) + + self.loader.unset_defaults(m) + self.assertTrue('repo' not in m['chunks'][0]) + + def test_convertes_max_jobs_to_an_integer(self): + m = morphlib.morphology.Morphology( + { + "name": "foo", + "kind": "chunk", + "max-jobs": "42" + }) + self.loader.set_defaults(m) + self.assertEqual(m['max-jobs'], 42) + + def test_parses_simple_cluster_morph(self): + string = ''' + name: foo + kind: cluster + systems: + - morph: bar + ''' + m = self.loader.parse_morphology_text(string, 'test') + self.loader.set_defaults(m) + self.loader.validate(m) + self.assertEqual(m['name'], 'foo') + self.assertEqual(m['kind'], 'cluster') + self.assertEqual(m['systems'][0]['morph'], 'bar') + + @contextlib.contextmanager + def catch_warnings(*warning_classes): + with warnings.catch_warnings(record=True) as caught_warnings: + warnings.resetwarnings() + for warning_class in warning_classes: + warnings.simplefilter("always", warning_class) + yield caught_warnings + + def test_warns_when_systems_refer_to_strata_with_repo_or_ref(self): + for obsolete_field in ('repo', 'ref'): + m = morphlib.morphology.Morphology( + name="foo", + kind="system", + arch="testarch", + strata=[ + { + 'morph': 'bar', + obsolete_field: 'obsolete', + }]) + + with self.catch_warnings(MorphologyObsoleteFieldWarning) \ + as caught_warnings: + + self.loader.validate(m) + self.assertEqual(len(caught_warnings), 1) + warning = caught_warnings[0].message + self.assertEqual(warning.kind, 'system') + self.assertEqual(warning.morphology_name, 'foo') + self.assertEqual(warning.stratum_name, 'bar') + self.assertEqual(warning.field, obsolete_field) + + def test_warns_when_strata_refer_to_build_depends_with_repo_or_ref(self): + for obsolete_field in ('repo', 'ref'): + m = morphlib.morphology.Morphology( + { + 'name': 'foo', + 'kind': 'stratum', + 'build-depends': [ + { + 'morph': 'bar', + obsolete_field: 'obsolete' + }, + ], + 'chunks': [ + { + 'morph': 'chunk', + 'name': 'chunk', + 'build-mode': 'test', + 'build-depends': [], + }, + ], + }) + + with self.catch_warnings(MorphologyObsoleteFieldWarning) \ + as caught_warnings: + + self.loader.validate(m) + self.assertEqual(len(caught_warnings), 1) + warning = caught_warnings[0].message + self.assertEqual(warning.kind, 'stratum') + self.assertEqual(warning.morphology_name, 'foo') + self.assertEqual(warning.stratum_name, 'bar') + self.assertEqual(warning.field, obsolete_field) + + def test_unordered_asciibetically_after_ordered(self): + # We only get morphologies with arbitrary keys in clusters + m = morphlib.morphology.Morphology( + name='foo', + kind='cluster', + systems=[ + { + 'morph': 'system-name', + 'repo': 'test:morphs', + 'ref': 'master', + 'deploy': { + 'deployment-foo': { + 'type': 'tarball', + 'location': '/tmp/path.tar', + 'HOSTNAME': 'aasdf', + } + } + } + ] + ) + s = self.loader.save_to_string(m) + # root field order + self.assertLess(s.find('name'), s.find('kind')) + self.assertLess(s.find('kind'), s.find('systems')) + # systems field order + self.assertLess(s.find('morph'), s.find('repo')) + self.assertLess(s.find('repo'), s.find('ref')) + self.assertLess(s.find('ref'), s.find('deploy')) + # deployment keys field order + self.assertLess(s.find('type'), s.find('location')) + self.assertLess(s.find('location'), s.find('HOSTNAME')) + + def test_multi_line_round_trip(self): + s = ('name: foo\n' + 'kind: bar\n' + 'description: |\n' + ' 1 2 3\n' + ' 4 5 6\n' + ' 7 8 9\n') + m = self.loader.parse_morphology_text(s, 'string') + self.assertEqual(s, self.loader.save_to_string(m)) + + def test_smoketest_multi_line_unicode(self): + m = morphlib.morphology.Morphology( + name=u'foo', + description=u'1 2 3\n4 5 6\n7 8 9\n', + ) + s = self.loader.save_to_string(m) + + def test_smoketest_multi_line_unicode_encoded(self): + m = morphlib.morphology.Morphology( + name=u'foo \u263A'.encode('utf-8'), + description=u'1 \u263A\n2 \u263A\n3 \u263A\n'.encode('utf-8'), + ) + s = self.loader.save_to_string(m) + + def test_smoketest_binary_garbage(self): + m = morphlib.morphology.Morphology( + description='\x92', + ) + s = self.loader.save_to_string(m) diff --git a/morphlib/morphology.py b/morphlib/morphology.py new file mode 100644 index 00000000..009ed044 --- /dev/null +++ b/morphlib/morphology.py @@ -0,0 +1,51 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import UserDict + + +class Morphology(UserDict.IterableUserDict): + + '''A container for a morphology, plus its metadata. + + A morphology is, basically, a dict. This class acts as that dict, + plus stores additional metadata about the morphology, such as where + it came from, and the ref that was used for it. It also has a dirty + attribute, to indicate whether the morphology has had changes done + to it, but does not itself set that attribute: the caller has to + maintain the flag themselves. + + This class does NO validation of the data, nor does it parse the + morphology text, or produce a textual form of itself. For those + things, see MorphologyLoader. + + ''' + + def __init__(self, *args, **kwargs): + UserDict.IterableUserDict.__init__(self, *args, **kwargs) + self.repo_url = None + self.ref = None + self.filename = None + self.dirty = None + + @property + def needs_artifact_metadata_cached(self): # pragma: no cover + return self.get('kind') == 'stratum' + + def __hash__(self): # pragma: no cover + return id(self) diff --git a/morphlib/morphology_tests.py b/morphlib/morphology_tests.py new file mode 100644 index 00000000..385f62ee --- /dev/null +++ b/morphlib/morphology_tests.py @@ -0,0 +1,48 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import unittest + +import morphlib + + +class MorphologyTests(unittest.TestCase): + + def setUp(self): + self.morph = morphlib.morphology.Morphology() + + def test_has_repo_url_attribute(self): + self.assertEqual(self.morph.repo_url, None) + self.morph.repo_url = 'foo' + self.assertEqual(self.morph.repo_url, 'foo') + + def test_has_ref_attribute(self): + self.assertEqual(self.morph.ref, None) + self.morph.ref = 'foo' + self.assertEqual(self.morph.ref, 'foo') + + def test_has_filename_attribute(self): + self.assertEqual(self.morph.filename, None) + self.morph.filename = 'foo' + self.assertEqual(self.morph.filename, 'foo') + + def test_has_dirty_attribute(self): + self.assertEqual(self.morph.dirty, None) + self.morph.dirty = True + self.assertEqual(self.morph.dirty, True) + diff --git a/morphlib/morphologyfactory.py b/morphlib/morphologyfactory.py new file mode 100644 index 00000000..b0a0528d --- /dev/null +++ b/morphlib/morphologyfactory.py @@ -0,0 +1,90 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import os + +import morphlib +import cliapp + + +class MorphologyFactoryError(cliapp.AppException): + pass + + +class MorphologyNotFoundError(MorphologyFactoryError): + def __init__(self, filename): + MorphologyFactoryError.__init__( + self, "Couldn't find morphology: %s" % filename) + + +class NotcachedError(MorphologyFactoryError): + def __init__(self, repo_name): + MorphologyFactoryError.__init__( + self, "Repository %s is not cached locally and there is no " + "remote cache specified" % repo_name) + + +class MorphologyFactory(object): + + '''A way of creating morphologies which will provide a default''' + + def __init__(self, local_repo_cache, remote_repo_cache=None, app=None): + self._lrc = local_repo_cache + self._rrc = remote_repo_cache + self._app = app + + def status(self, *args, **kwargs): # pragma: no cover + if self._app is not None: + self._app.status(*args, **kwargs) + + def get_morphology(self, reponame, sha1, filename): + morph_name = os.path.splitext(os.path.basename(filename))[0] + loader = morphlib.morphloader.MorphologyLoader() + if self._lrc.has_repo(reponame): + self.status(msg="Looking for %s in local repo cache" % filename, + chatty=True) + try: + repo = self._lrc.get_repo(reponame) + morph = loader.load_from_string(repo.cat(sha1, filename)) + except IOError: + morph = None + file_list = repo.ls_tree(sha1) + elif self._rrc is not None: + self.status(msg="Retrieving %(reponame)s %(sha1)s %(filename)s" + " from the remote git cache.", + reponame=reponame, sha1=sha1, filename=filename, + chatty=True) + try: + text = self._rrc.cat_file(reponame, sha1, filename) + morph = loader.load_from_string(text) + except morphlib.remoterepocache.CatFileError: + morph = None + file_list = self._rrc.ls_tree(reponame, sha1) + else: + raise NotcachedError(reponame) + + if morph is None: + self.status(msg="File %s doesn't exist: attempting to infer " + "chunk morph from repo's build system" + % filename, chatty=True) + bs = morphlib.buildsystem.detect_build_system(file_list) + if bs is None: + raise MorphologyNotFoundError(filename) + morph = bs.get_morphology(morph_name) + loader.validate(morph) + loader.set_commands(morph) + loader.set_defaults(morph) + return morph diff --git a/morphlib/morphologyfactory_tests.py b/morphlib/morphologyfactory_tests.py new file mode 100644 index 00000000..52d5f598 --- /dev/null +++ b/morphlib/morphologyfactory_tests.py @@ -0,0 +1,285 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import unittest + +import morphlib +from morphlib.morphologyfactory import (MorphologyFactory, + MorphologyNotFoundError, + NotcachedError) +from morphlib.remoterepocache import CatFileError + + +class FakeRemoteRepoCache(object): + + def cat_file(self, reponame, sha1, filename): + if filename.endswith('.morph'): + return '''{ + "name": "%s", + "kind": "chunk", + "build-system": "dummy" + }''' % filename[:-len('.morph')] + return 'text' + + def ls_tree(self, reponame, sha1): + return [] + +class FakeLocalRepo(object): + + morphologies = { + 'chunk.morph': ''' + name: chunk + kind: chunk + build-system: dummy + ''', + 'chunk-split.morph': ''' + name: chunk-split + kind: chunk + build-system: dummy + products: + - artifact: chunk-split-runtime + include: [] + - artifact: chunk-split-devel + include: [] + ''', + 'stratum.morph': ''' + name: stratum + kind: stratum + chunks: + - name: chunk + repo: test:repo + ref: sha1 + build-mode: bootstrap + build-depends: [] + ''', + 'stratum-no-chunk-bdeps.morph': ''' + name: stratum-no-chunk-bdeps + kind: stratum + chunks: + - name: chunk + repo: test:repo + ref: sha1 + build-mode: bootstrap + ''', + 'stratum-no-bdeps-no-bootstrap.morph': ''' + name: stratum-no-bdeps-no-bootstrap + kind: stratum + chunks: + - name: chunk + repo: test:repo + ref: sha1 + build-depends: [] + ''', + 'stratum-bdeps-no-bootstrap.morph': ''' + name: stratum-bdeps-no-bootstrap + kind: stratum + build-depends: + - morph: stratum + chunks: + - name: chunk + repo: test:repo + ref: sha1 + build-depends: [] + ''', + 'stratum-empty.morph': ''' + name: stratum-empty + kind: stratum + ''', + 'system.morph': ''' + name: system + kind: system + arch: %(arch)s + strata: + - morph: stratum + ''', + 'parse-error.morph': ''' name''', + 'name-mismatch.morph': ''' + name: fred + kind: stratum + ''', + } + + def __init__(self): + self.arch = 'x86_64' + + def cat(self, sha1, filename): + if filename in self.morphologies: + values = { + 'arch': self.arch, + } + return self.morphologies[filename] % values + elif filename.endswith('.morph'): + return '''{ + "name": "%s", + "kind": "chunk", + "build-system": "dummy" + }''' % filename[:-len('.morph')] + return 'text' + + def ls_tree(self, sha1): + return self.morphologies.keys() + +class FakeLocalRepoCache(object): + + def __init__(self, lr): + self.lr = lr + + def has_repo(self, reponame): + return True + + def get_repo(self, reponame): + return self.lr + + +class FakeApp(object): + + def status(self, **kwargs): + pass + + +class MorphologyFactoryTests(unittest.TestCase): + + def setUp(self): + self.lr = FakeLocalRepo() + self.lrc = FakeLocalRepoCache(self.lr) + self.rrc = FakeRemoteRepoCache() + self.mf = MorphologyFactory(self.lrc, self.rrc, app=FakeApp()) + self.lmf = MorphologyFactory(self.lrc, None) + + def nolocalfile(self, *args): + raise IOError('File not found') + + def noremotefile(self, *args): + raise CatFileError('reponame', 'ref', 'filename') + + def localmorph(self, *args): + return ['chunk.morph'] + + def nolocalmorph(self, *args): + if args[-1].endswith('.morph'): + raise IOError('File not found') + return 'text' + + def autotoolsbuildsystem(self, *args): + return ['configure.in'] + + def remotemorph(self, *args): + return ['remote-chunk.morph'] + + def noremotemorph(self, *args): + if args[-1].endswith('.morph'): + raise CatFileError('reponame', 'ref', 'filename') + return 'text' + + def doesnothaverepo(self, reponame): + return False + + def test_gets_morph_from_local_repo(self): + self.lr.ls_tree = self.localmorph + morph = self.mf.get_morphology('reponame', 'sha1', + 'chunk.morph') + self.assertEqual('chunk', morph['name']) + + def test_gets_morph_from_remote_repo(self): + self.rrc.ls_tree = self.remotemorph + self.lrc.has_repo = self.doesnothaverepo + morph = self.mf.get_morphology('reponame', 'sha1', + 'remote-chunk.morph') + self.assertEqual('remote-chunk', morph['name']) + + def test_autodetects_local_morphology(self): + self.lr.cat = self.nolocalmorph + self.lr.ls_tree = self.autotoolsbuildsystem + morph = self.mf.get_morphology('reponame', 'sha1', + 'assumed-local.morph') + self.assertEqual('assumed-local', morph['name']) + + def test_autodetects_remote_morphology(self): + self.lrc.has_repo = self.doesnothaverepo + self.rrc.cat_file = self.noremotemorph + self.rrc.ls_tree = self.autotoolsbuildsystem + morph = self.mf.get_morphology('reponame', 'sha1', + 'assumed-remote.morph') + self.assertEqual('assumed-remote', morph['name']) + + def test_raises_error_when_no_local_morph(self): + self.lr.cat = self.nolocalfile + self.assertRaises(MorphologyNotFoundError, self.mf.get_morphology, + 'reponame', 'sha1', 'unreached.morph') + + def test_raises_error_when_fails_no_remote_morph(self): + self.lrc.has_repo = self.doesnothaverepo + self.rrc.cat_file = self.noremotefile + self.assertRaises(MorphologyNotFoundError, self.mf.get_morphology, + 'reponame', 'sha1', 'unreached.morph') + + def test_raises_error_when_name_mismatches(self): + self.assertRaises(morphlib.Error, self.mf.get_morphology, + 'reponame', 'sha1', 'name-mismatch.morph') + + def test_looks_locally_with_no_remote(self): + self.lr.ls_tree = self.localmorph + morph = self.lmf.get_morphology('reponame', 'sha1', + 'chunk.morph') + self.assertEqual('chunk', morph['name']) + + def test_autodetects_locally_with_no_remote(self): + self.lr.cat = self.nolocalmorph + self.lr.ls_tree = self.autotoolsbuildsystem + morph = self.mf.get_morphology('reponame', 'sha1', + 'assumed-local.morph') + self.assertEqual('assumed-local', morph['name']) + + def test_fails_when_local_not_cached_and_no_remote(self): + self.lrc.has_repo = self.doesnothaverepo + self.assertRaises(NotcachedError, self.lmf.get_morphology, + 'reponame', 'sha1', 'unreached.morph') + + def test_arch_is_validated(self): + self.lr.arch = 'unknown' + self.assertRaises(morphlib.Error, self.mf.get_morphology, + 'reponame', 'sha1', 'system.morph') + + def test_arch_arm_defaults_to_le(self): + self.lr.arch = 'armv7' + morph = self.mf.get_morphology('reponame', 'sha1', 'system.morph') + self.assertEqual(morph['arch'], 'armv7l') + + def test_fails_on_parse_error(self): + self.assertRaises(morphlib.Error, self.mf.get_morphology, + 'reponame', 'sha1', 'parse-error.morph') + + def test_fails_on_no_chunk_bdeps(self): + self.assertRaises(morphlib.morphloader.NoBuildDependenciesError, + self.mf.get_morphology, 'reponame', 'sha1', + 'stratum-no-chunk-bdeps.morph') + + def test_fails_on_no_bdeps_or_bootstrap(self): + self.assertRaises( + morphlib.morphloader.NoStratumBuildDependenciesError, + self.mf.get_morphology, 'reponame', 'sha1', + 'stratum-no-bdeps-no-bootstrap.morph') + + def test_succeeds_on_bdeps_no_bootstrap(self): + self.mf.get_morphology( + 'reponame', 'sha1', + 'stratum-bdeps-no-bootstrap.morph') + + def test_fails_on_empty_stratum(self): + self.assertRaises( + morphlib.morphloader.EmptyStratumError, + self.mf.get_morphology, 'reponame', 'sha1', 'stratum-empty.morph') + diff --git a/morphlib/morphologyfinder.py b/morphlib/morphologyfinder.py new file mode 100644 index 00000000..87c0de1a --- /dev/null +++ b/morphlib/morphologyfinder.py @@ -0,0 +1,62 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import cliapp + +import morphlib + + +class MorphologyFinder(object): + + '''Abstract away finding morphologies in a git repository. + + This class provides an abstraction layer between a git repository + and the morphologies contained in it. + + ''' + + def __init__(self, gitdir, ref=None): + self.gitdir = gitdir + self.ref = ref + + def read_morphology(self, filename): + '''Return the un-parsed text of a morphology. + + For the given morphology name, locate and return the contents + of the morphology as a string. + + Parsing of this morphology into a form useful for manipulating + is handled by the MorphologyLoader class. + + ''' + return self.gitdir.read_file(filename, self.ref) + + def list_morphologies(self): + '''Return the filenames of all morphologies in the (repo, ref). + + Finds all morphologies in the git directory at the specified + ref. + + ''' + + def is_morphology_path(path): + return path.endswith('.morph') + + return (path + for path in self.gitdir.list_files(self.ref) + if is_morphology_path(path)) diff --git a/morphlib/morphologyfinder_tests.py b/morphlib/morphologyfinder_tests.py new file mode 100644 index 00000000..67161f9b --- /dev/null +++ b/morphlib/morphologyfinder_tests.py @@ -0,0 +1,112 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import os +import shutil +import tempfile +import unittest + +import morphlib + + +class MorphologyFinderTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.dirname = os.path.join(self.tempdir, 'repo') + os.mkdir(self.dirname) + gd = morphlib.gitdir.init(self.dirname) + for fn in ('foo', 'bar.morph', 'baz.morph', 'quux'): + with open(os.path.join(self.dirname, fn), "w") as f: + f.write('dummy morphology text') + morphlib.git.gitcmd(gd._runcmd, 'add', '.') + morphlib.git.gitcmd(gd._runcmd, 'commit', '-m', 'Initial commit') + + # Changes for difference between commited and work tree + newmorphpath = os.path.join(self.dirname, 'foo.morph') + os.unlink(os.path.join(self.dirname, 'foo')) + with open(newmorphpath, 'w') as f: + f.write("altered morphology text") + + # Changes for bare repository + self.mirror = os.path.join(self.tempdir, 'mirror') + morphlib.git.gitcmd(gd._runcmd, 'clone', '--mirror', self.dirname, + self.mirror) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_list_morphs_in_HEAD(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + mf = morphlib.morphologyfinder.MorphologyFinder(gd, 'HEAD') + self.assertEqual(sorted(mf.list_morphologies()), + ['bar.morph', 'baz.morph']) + + def test_list_morphs_in_master(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + mf = morphlib.morphologyfinder.MorphologyFinder(gd, 'master') + self.assertEqual(sorted(mf.list_morphologies()), + ['bar.morph', 'baz.morph']) + + def test_list_morphs_raises_with_invalid_ref(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + mf = morphlib.morphologyfinder.MorphologyFinder(gd, 'invalid_ref') + self.assertRaises(morphlib.gitdir.InvalidRefError, + mf.list_morphologies) + + def test_list_morphs_in_work_tree(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + mf = morphlib.morphologyfinder.MorphologyFinder(gd) + self.assertEqual(sorted(mf.list_morphologies()), + ['bar.morph', 'baz.morph', 'foo.morph']) + + def test_list_morphs_raises_no_worktree_no_ref(self): + gd = morphlib.gitdir.GitDirectory(self.mirror) + mf = morphlib.morphologyfinder.MorphologyFinder(gd) + self.assertRaises(morphlib.gitdir.NoWorkingTreeError, + mf.list_morphologies) + + def test_read_morph_in_HEAD(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + mf = morphlib.morphologyfinder.MorphologyFinder(gd, 'HEAD') + self.assertEqual(mf.read_morphology('bar.morph'), + "dummy morphology text") + + def test_read_morph_in_master(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + mf = morphlib.morphologyfinder.MorphologyFinder(gd, 'master') + self.assertEqual(mf.read_morphology('bar.morph'), + "dummy morphology text") + + def test_read_morph_raises_with_invalid_ref(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + mf = morphlib.morphologyfinder.MorphologyFinder(gd, 'invalid_ref') + self.assertRaises(morphlib.gitdir.InvalidRefError, + mf.read_morphology, 'bar') + + def test_read_morph_in_work_tree(self): + gd = morphlib.gitdir.GitDirectory(self.dirname) + mf = morphlib.morphologyfinder.MorphologyFinder(gd) + self.assertEqual(mf.read_morphology('foo.morph'), + "altered morphology text") + + def test_read_morph_raises_no_worktree_no_ref(self): + gd = morphlib.gitdir.GitDirectory(self.mirror) + mf = morphlib.morphologyfinder.MorphologyFinder(gd) + self.assertRaises(morphlib.gitdir.NoWorkingTreeError, + mf.read_morphology, 'bar.morph') diff --git a/morphlib/morphset.py b/morphlib/morphset.py new file mode 100644 index 00000000..bf061f94 --- /dev/null +++ b/morphlib/morphset.py @@ -0,0 +1,247 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import morphlib + + +class ChunkNotInStratumError(morphlib.Error): + + def __init__(self, stratum_name, chunk_name): + self.msg = ( + 'Stratum %s does not contain %s' % (stratum_name, chunk_name)) + + +class MorphologySet(object): + + '''Store and manipulate a set of Morphology objects.''' + + def __init__(self): + self.morphologies = [] + + def add_morphology(self, morphology): + '''Add a morphology object to the set, unless it's there already.''' + + triplet = ( + morphology.repo_url, + morphology.ref, + morphology.filename + ) + for existing in self.morphologies: + existing_triplet = ( + existing.repo_url, + existing.ref, + existing.filename + ) + if existing_triplet == triplet: + return + + self.morphologies.append(morphology) + + def has(self, repo_url, ref, filename): + '''Does the set have a morphology for the given triplet?''' + return self._get_morphology(repo_url, ref, filename) is not None + + def _get_morphology(self, repo_url, ref, filename): + for m in self.morphologies: + if (m.repo_url == repo_url and + m.ref == ref and + m.filename == filename): + return m + return None + + def _find_spec(self, specs, wanted_name): + for spec in specs: + name = spec.get('name', spec.get('morph')) + if name == wanted_name: + return spec.get('repo'), spec.get('ref'), name + return None, None, None + + def get_chunk_triplet(self, stratum_morph, chunk_name): + '''Return the repo url, ref, morph name triplet for a chunk. + + Given a stratum morphology, find the triplet used to refer to + a given chunk. Note that because of how the chunk may be + referred to using either name or morph fields in the morphology, + the morph field (or its computed value) is always returned. + Note also that the morph field, not the filename, is returned. + + Raise ChunkNotInStratumError if the chunk is not found in the + stratum. + + ''' + + repo_url, ref, morph = self._find_spec( + stratum_morph['chunks'], chunk_name) + if (repo_url, ref, morph) == (None, None, None): + raise ChunkNotInStratumError(stratum_morph['name'], chunk_name) + return repo_url, ref, morph + + def traverse_specs(self, cb_process, cb_filter=lambda s: True): + '''Higher-order function for processing every spec. + + This traverses every spec in all the morphologies, so all chunk, + stratum and stratum-build-depend specs are visited. + + It is to be passed one or two callbacks. `cb_process` is given + a spec, which it may alter, but if it does, it must return True. + + `cb_filter` is given the morphology, the kind of spec it is + working on in addition to the spec itself. + + `cb_filter` is expected to decide whether to run `cb_process` + on the spec. + + Arguably this could be checked in `cb_process`, but it can be less + logic over all since `cb_process` need not conditionally return. + + If any specs have been altered, at the end of iteration, any + morphologies in the MorphologySet that are referred to by an + altered spec are also changed. + + This requires a full iteration of the MorphologySet, so it is not a + cheap operation. + + A coroutine was attempted, but it required the same amount of + code at the call site as doing it by hand. + + ''' + + altered_references = {} + + def process_spec_list(m, kind): + specs = m[kind] + for spec in specs: + if cb_filter(m, kind, spec): + fn = morphlib.util.sanitise_morphology_path( + spec['morph'] if 'morph' in spec else spec['name']) + orig_spec = (spec.get('repo'), spec.get('ref'), fn) + dirtied = cb_process(m, kind, spec) + if dirtied: + m.dirty = True + altered_references[orig_spec] = spec + + for m in self.morphologies: + if m['kind'] == 'system': + process_spec_list(m, 'strata') + elif m['kind'] == 'stratum': + process_spec_list(m, 'build-depends') + process_spec_list(m, 'chunks') + + for m in self.morphologies: + tup = (m.repo_url, m.ref, m.filename) + if tup in altered_references: + spec = altered_references[tup] + if m.ref != spec.get('ref'): + m.ref = spec.get('ref') + m.dirty = True + file = morphlib.util.sanitise_morphology_path( + spec['morph'] if 'morph' in spec else spec['name']) + assert (m.filename == file + or m.repo_url == spec.get('repo')), \ + 'Moving morphologies is not supported.' + + def change_ref(self, repo_url, orig_ref, morph_name, new_ref): + '''Change a triplet's ref to a new one in all morphologies in a ref. + + Change orig_ref to new_ref in any morphology that references the + original triplet. This includes stratum build-dependencies. + + ''' + + def wanted_spec(m, kind, spec): + spec_name = spec['name'] if 'name' in spec else spec['morph'] + return (spec.get('repo') == repo_url and + spec.get('ref') == orig_ref and + spec_name == morph_name) + + def process_spec(m, kind, spec): + spec['unpetrify-ref'] = spec.get('ref') + spec['ref'] = new_ref + return True + + self.traverse_specs(process_spec, wanted_spec) + + def list_refs(self): + '''Return a set of all the (repo, ref) pairs in the MorphologySet. + + This does not dirty the morphologies so they do not need to be + written back to the disk. + + ''' + known = set() + + def wanted_spec(m, kind, spec): + return (spec.get('repo'), spec.get('ref')) not in known + + def process_spec(m, kind, spec): + known.add((spec.get('repo'), spec.get('ref'))) + return False + + self.traverse_specs(process_spec, wanted_spec) + + return known + + def repoint_refs(self, repo_url, new_ref): + '''Change all specs which refer to (repo, *) to (repo, new_ref). + + This is stunningly similar to change_ref, with the exception of + ignoring the morphology name and ref fields. + + It is intended to be used before chunks are petrified + + ''' + def wanted_spec(m, kind, spec): + return spec.get('repo') == repo_url + + def process_spec(m, kind, spec): + if 'unpetrify-ref' not in spec: + spec['unpetrify-ref'] = spec.get('ref') + spec['ref'] = new_ref + return True + + self.traverse_specs(process_spec, wanted_spec) + + def petrify_chunks(self, resolutions): + '''Update _every_ chunk's ref to the value resolved in resolutions. + + `resolutions` must be a {(repo, ref): resolved_ref} + + This is subtly different to change_ref, since that works on + changing a single spec including its filename, and the morphology + those specs refer to, while petrify_chunks is interested in changing + _all_ the refs. + + ''' + + def wanted_chunk_spec(m, kind, spec): + # Do not attempt to petrify non-chunk specs. + # This is not handled by previous implementations, and + # the details are tricky. + if not (m['kind'] == 'stratum' and kind == 'chunks'): + return + ref = spec.get('ref') + return (not morphlib.git.is_valid_sha1(ref) + and (spec.get('repo'), ref) in resolutions) + + def process_chunk_spec(m, kind, spec): + tup = (spec.get('repo'), spec.get('ref')) + spec['unpetrify-ref'] = spec.get('ref') + spec['ref'] = resolutions[tup] + return True + + self.traverse_specs(process_chunk_spec, wanted_chunk_spec) diff --git a/morphlib/morphset_tests.py b/morphlib/morphset_tests.py new file mode 100644 index 00000000..81b5810f --- /dev/null +++ b/morphlib/morphset_tests.py @@ -0,0 +1,202 @@ +# Copyright (C) 2013, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import unittest + +import morphlib + + +class MorphologySetTests(unittest.TestCase): + + def setUp(self): + self.morphs = morphlib.morphset.MorphologySet() + + self.system = morphlib.morphology.Morphology({ + 'kind': 'system', + 'name': 'foo-system', + 'strata': [ + { + 'repo': 'test:morphs', + 'ref': 'master', + 'morph': 'foo-stratum', + }, + ], + }) + self.system.repo_url = 'test:morphs' + self.system.ref = 'master' + self.system.filename = 'foo-system.morph' + + self.stratum = morphlib.morphology.Morphology({ + 'kind': 'stratum', + 'name': 'foo-stratum', + 'chunks': [ + { + 'repo': 'test:foo-chunk', + 'ref': 'master', + 'morph': 'foo-chunk', + }, + ], + 'build-depends': [], + }) + self.stratum.repo_url = 'test:morphs' + self.stratum.ref = 'master' + self.stratum.filename = 'foo-stratum.morph' + + def test_is_empty_initially(self): + self.assertEqual(self.morphs.morphologies, []) + self.assertFalse( + self.morphs.has( + self.system.repo_url, self.system.ref, self.system.filename)) + + def test_adds_morphology(self): + self.morphs.add_morphology(self.system) + self.assertEqual(self.morphs.morphologies, [self.system]) + self.assertTrue( + self.morphs.has( + self.system.repo_url, self.system.ref, self.system.filename)) + + self.morphs.add_morphology(self.stratum) + self.assertEqual( + self.morphs.morphologies, + [self.system, self.stratum]) + + def test_does_not_add_morphology_twice(self): + self.morphs.add_morphology(self.system) + self.morphs.add_morphology(self.system) + self.assertEqual(self.morphs.morphologies, [self.system]) + + def test_get_chunk_triplet(self): + self.morphs.add_morphology(self.system) + self.morphs.add_morphology(self.stratum) + self.assertEqual( + self.morphs.get_chunk_triplet(self.stratum, 'foo-chunk'), + ('test:foo-chunk', 'master', 'foo-chunk')) + + def test_raises_chunk_not_in_stratum_error(self): + self.assertRaises( + morphlib.morphset.ChunkNotInStratumError, + self.morphs.get_chunk_triplet, self.stratum, 'wrong') + + def test_changes_stratum_ref(self): + self.morphs.add_morphology(self.system) + self.morphs.add_morphology(self.stratum) + self.morphs.change_ref( + self.stratum.repo_url, + self.stratum.ref, + self.stratum['name'], + 'new-ref') + self.assertEqual(self.stratum.ref, 'new-ref') + self.assertEqual( + self.system['strata'][0], + { + 'repo': 'test:morphs', + 'ref': 'new-ref', + 'morph': 'foo-stratum', + 'unpetrify-ref': 'master', + }) + + def test_changes_stratum_ref_in_build_depends(self): + other_stratum = morphlib.morphology.Morphology({ + 'name': 'other-stratum', + 'kind': 'stratum', + 'chunks': [], + 'build-depends': [ + { + 'repo': self.stratum.repo_url, + 'ref': self.stratum.ref, + 'morph': self.stratum['name'], + 'unpetrify-ref': 'master', + }, + ] + }) + other_stratum.repo_url = 'test:morphs' + other_stratum.ref = 'master' + other_stratum.filename = 'other-stratum.morph' + + self.morphs.add_morphology(self.system) + self.morphs.add_morphology(self.stratum) + self.morphs.add_morphology(other_stratum) + self.morphs.change_ref( + self.stratum.repo_url, + self.stratum.ref, + self.stratum['name'], + 'new-ref') + self.assertEqual( + other_stratum['build-depends'][0], + { + 'repo': 'test:morphs', + 'ref': 'new-ref', + 'morph': 'foo-stratum', + 'unpetrify-ref': 'master', + }) + + def test_changes_chunk_ref(self): + self.morphs.add_morphology(self.system) + self.morphs.add_morphology(self.stratum) + self.morphs.change_ref( + 'test:foo-chunk', + 'master', + 'foo-chunk', + 'new-ref') + self.assertEqual( + self.stratum['chunks'], + [ + { + 'repo': 'test:foo-chunk', + 'ref': 'new-ref', + 'morph': 'foo-chunk', + 'unpetrify-ref': 'master', + } + ]) + + def test_list_refs(self): + self.morphs.add_morphology(self.system) + self.morphs.add_morphology(self.stratum) + self.assertEqual(sorted(self.morphs.list_refs()), + [('test:foo-chunk', 'master'), + ('test:morphs', 'master')]) + + def test_repoint_refs(self): + self.morphs.add_morphology(self.system) + self.morphs.add_morphology(self.stratum) + self.morphs.repoint_refs('test:morphs', 'test') + self.assertEqual(self.system['strata'], + [ + { + 'morph': 'foo-stratum', + 'ref': 'test', + 'repo': 'test:morphs', + 'unpetrify-ref': 'master', + } + ]) + + def test_petrify_chunks(self): + # TODO: test petrifying a larger morphset + self.morphs.add_morphology(self.system) + self.morphs.add_morphology(self.stratum) + self.morphs.petrify_chunks({('test:foo-chunk', 'master'): '0'*40}) + self.assertEqual( + self.stratum['chunks'], + [ + { + 'repo': 'test:foo-chunk', + 'ref': '0'*40, + 'morph': 'foo-chunk', + 'unpetrify-ref': 'master', + } + ]) diff --git a/morphlib/mountableimage.py b/morphlib/mountableimage.py new file mode 100644 index 00000000..f767228a --- /dev/null +++ b/morphlib/mountableimage.py @@ -0,0 +1,86 @@ +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import logging +import os +import tempfile +import gzip + +import morphlib + + +class MountableImage(object): # pragma: no cover + + '''Mountable image (deals with decompression). + + Note, this is a read-only mount in the sense that the decompressed + image is not then recompressed after, instead any changes are discarded. + + ''' + def __init__(self, app, artifact_path): + self.app = app + self.artifact_path = artifact_path + + def setup(self, path): + self.app.status(msg='Preparing image %(path)s', path=path, chatty=True) + self.app.status(msg=' Decompressing...', chatty=True) + (tempfd, self.temp_path) = \ + tempfile.mkstemp(dir=self.app.settings['tempdir']) + + try: + with os.fdopen(tempfd, "wb") as outfh: + infh = gzip.open(path, "rb") + morphlib.util.copyfileobj(infh, outfh) + infh.close() + except BaseException, e: + logging.error('Caught exception: %s' % str(e)) + logging.info('Removing temporary file %s' % self.temp_path) + os.unlink(self.temp_path) + raise + self.app.status(msg=' Mounting image at %(path)s', + path=self.temp_path, chatty=True) + part = morphlib.fsutils.setup_device_mapping(self.app.runcmd, + self.temp_path) + mount_point = tempfile.mkdtemp(dir=self.app.settings['tempdir']) + morphlib.fsutils.mount(self.app.runcmd, part, mount_point) + self.mount_point = mount_point + return mount_point + + def cleanup(self, path, mount_point): + self.app.status(msg='Clearing down image at %(path)s', path=path, + chatty=True) + try: + morphlib.fsutils.unmount(self.app.runcmd, mount_point) + except BaseException, e: + logging.info('Ignoring error when unmounting: %s' % str(e)) + try: + morphlib.fsutils.undo_device_mapping(self.app.runcmd, path) + except BaseException, e: + logging.info( + 'Ignoring error when undoing device mapping: %s' % str(e)) + try: + os.rmdir(mount_point) + os.unlink(path) + except BaseException, e: + logging.info( + 'Ignoring error when removing temporary files: %s' % str(e)) + + def __enter__(self): + return self.setup(self.artifact_path) + + def __exit__(self, exctype, excvalue, exctraceback): + self.cleanup(self.temp_path, self.mount_point) diff --git a/morphlib/plugins/__init__.py b/morphlib/plugins/__init__.py new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/morphlib/plugins/__init__.py diff --git a/morphlib/plugins/add_binary_plugin.py b/morphlib/plugins/add_binary_plugin.py new file mode 100644 index 00000000..a192f792 --- /dev/null +++ b/morphlib/plugins/add_binary_plugin.py @@ -0,0 +1,130 @@ +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import logging +import os +import re +import urlparse + +import morphlib + + +class AddBinaryPlugin(cliapp.Plugin): + + '''Add a subcommand for dealing with large binary files.''' + + def enable(self): + self.app.add_subcommand( + 'add-binary', self.add_binary, arg_synopsis='FILENAME...') + + def disable(self): + pass + + def add_binary(self, binaries): + '''Add a binary file to the current repository. + + Command line argument: + + * `FILENAME...` is the binaries to be added to the repository. + + This checks for the existence of a .gitfat file in the repository. If + there is one then a line is added to .gitattributes telling it that + the given binary should be handled by git-fat. If there is no .gitfat + file then it is created, with the rsync remote pointing at the correct + directory on the Trove host. A line is then added to .gitattributes to + say that the given binary should be handled by git-fat. + + Example: + + morph add-binary big_binary.tar.gz + + ''' + if not binaries: + raise morphlib.Error('add-binary must get at least one argument') + + gd = morphlib.gitdir.GitDirectory(os.getcwd()) + gd.fat_init() + if not gd.has_fat(): + self._make_gitfat(gd) + self._handle_binaries(binaries, gd) + logging.info('Staged binaries for commit') + + def _handle_binaries(self, binaries, gd): + '''Add a filter for the given file, and then add it to the repo.''' + # begin by ensuring all paths given are relative to the root directory + files = [gd.get_relpath(os.path.realpath(binary)) + for binary in binaries] + + # escape special characters and whitespace + escaped = [] + for path in files: + path = self.escape_glob(path) + path = self.escape_whitespace(path) + escaped.append(path) + + # now add any files that aren't already mentioned in .gitattributes to + # the file so that git fat knows what to do + attr_path = gd.join_path('.gitattributes') + if '.gitattributes' in gd.list_files(): + with open(attr_path, 'r') as attributes: + current = set(f.split()[0] for f in attributes) + else: + current = set() + to_add = set(escaped) - current + + # if we don't need to change .gitattributes then we can just do + # `git add <binaries>` + if not to_add: + gd.get_index().add_files_from_working_tree(files) + return + + with open(attr_path, 'a') as attributes: + for path in to_add: + attributes.write('%s filter=fat -crlf\n' % path) + + # we changed .gitattributes, so need to stage it for committing + files.append(attr_path) + gd.get_index().add_files_from_working_tree(files) + + def _make_gitfat(self, gd): + '''Make .gitfat point to the rsync directory for the repo.''' + remote = gd.get_remote('origin') + if not remote.get_push_url(): + raise Exception( + 'Remote `origin` does not have a push URL defined.') + url = urlparse.urlparse(remote.get_push_url()) + if url.scheme != 'ssh': + raise Exception( + 'Push URL for `origin` is not an SSH URL: %s' % url.geturl()) + fat_store = '%s:%s' % (url.netloc, url.path) + fat_path = gd.join_path('.gitfat') + with open(fat_path, 'w+') as gitfat: + gitfat.write('[rsync]\n') + gitfat.write('remote = %s' % fat_store) + gd.get_index().add_files_from_working_tree([fat_path]) + + def escape_glob(self, path): + '''Escape glob metacharacters in a path and return the result.''' + metachars = re.compile('([*?[])') + path = metachars.sub(r'[\1]', path) + return path + + def escape_whitespace(self, path): + '''Substitute whitespace with [[:space:]] and return the result.''' + whitespace = re.compile('([ \n\r\t])') + path = whitespace.sub(r'[[:space:]]', path) + return path diff --git a/morphlib/plugins/artifact_inspection_plugin.py b/morphlib/plugins/artifact_inspection_plugin.py new file mode 100644 index 00000000..74645f41 --- /dev/null +++ b/morphlib/plugins/artifact_inspection_plugin.py @@ -0,0 +1,307 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import glob +import json +import os +import re +import contextlib + +import fs.tempfs + +import morphlib + +from morphlib.bins import call_in_artifact_directory +from morphlib.extractedtarball import ExtractedTarball +from morphlib.mountableimage import MountableImage + + +class NotASystemArtifactError(cliapp.AppException): + + def __init__(self, artifact): + cliapp.AppException.__init__( + self, '%s is not a system artifact' % artifact) + + +class ProjectVersionGuesser(object): + + def __init__(self, app, lrc, rrc, interesting_files): + self.app = app + self.lrc = lrc + self.rrc = rrc + self.interesting_files = interesting_files + + def file_contents(self, repo, ref, tree): + filenames = [x for x in self.interesting_files if x in tree] + if filenames: + if self.lrc.has_repo(repo): + repository = self.lrc.get_repo(repo) + for filename in filenames: + yield filename, repository.cat(ref, filename) + elif self.rrc: + for filename in filenames: + yield filename, self.rrc.cat_file(repo, ref, filename) + + +class AutotoolsVersionGuesser(ProjectVersionGuesser): + + def __init__(self, app, lrc, rrc): + ProjectVersionGuesser.__init__(self, app, lrc, rrc, [ + 'configure.ac', + 'configure.in', + 'configure.ac.in', + 'configure.in.in', + ]) + + def guess_version(self, repo, ref, tree): + version = None + for filename, data in self.file_contents(repo, ref, tree): + # First, try to grep for AC_INIT() + version = self._check_ac_init(data) + if version: + self.app.status( + msg='%(repo)s: Version of %(ref)s detected ' + 'via %(filename)s:AC_INIT: %(version)s', + repo=repo, ref=ref, filename=filename, + version=version, chatty=True) + break + + # Then, try running autoconf against the configure script + version = self._check_autoconf_package_version( + repo, ref, filename, data) + if version: + self.app.status( + msg='%(repo)s: Version of %(ref)s detected ' + 'by processing %(filename)s: %(version)s', + repo=repo, ref=ref, filename=filename, + version=version, chatty=True) + break + return version + + def _check_ac_init(self, data): + data = data.replace('\n', ' ') + for macro in ['AC_INIT', 'AM_INIT_AUTOMAKE']: + pattern = r'.*%s\((.*?)\).*' % macro + if not re.match(pattern, data): + continue + acinit = re.sub(pattern, r'\1', data) + if acinit: + version = acinit.split(',') + if macro == 'AM_INIT_AUTOMAKE' and len(version) == 1: + continue + version = version[0] if len(version) == 1 else version[1] + version = re.sub('[\[\]]', '', version).strip() + version = version.split()[0] + if version: + if version and version[0].isdigit(): + return version + return None + + def _check_autoconf_package_version(self, repo, ref, filename, data): + with contextlib.closing(fs.tempfs.TempFS( + temp_dir=self.app.settings['tempdir'])) as tempdir: + with open(tempdir.getsyspath(filename), 'w') as f: + f.write(data) + exit_code, output, errors = self.app.runcmd_unchecked( + ['autoconf', filename], + ['grep', '^PACKAGE_VERSION='], + ['cut', '-d=', '-f2'], + ['sed', "s/'//g"], + cwd=tempdir.root_path) + version = None + if output: + output = output.strip() + if output and output[0].isdigit(): + version = output + if exit_code != 0: + self.app.status( + msg='%(repo)s: Failed to detect version from ' + '%(ref)s:%(filename)s', + repo=repo, ref=ref, filename=filename, chatty=True) + return version + + +class VersionGuesser(object): + + def __init__(self, app): + self.app = app + self.lrc, self.rrc = morphlib.util.new_repo_caches(app) + self.guessers = [ + AutotoolsVersionGuesser(app, self.lrc, self.rrc) + ] + + def guess_version(self, repo, ref): + self.app.status(msg='%(repo)s: Guessing version of %(ref)s', + repo=repo, ref=ref, chatty=True) + version = None + try: + if self.lrc.has_repo(repo): + repository = self.lrc.get_repo(repo) + if not self.app.settings['no-git-update']: + repository.update() + tree = repository.ls_tree(ref) + elif self.rrc: + repository = None + tree = self.rrc.ls_tree(repo, ref) + else: + return None + for guesser in self.guessers: + version = guesser.guess_version(repo, ref, tree) + if version: + break + except cliapp.AppException, err: + self.app.status(msg='%(repo)s: Failed to list files in %(ref)s', + repo=repo, ref=ref, chatty=True) + return version + + +class ManifestGenerator(object): + + def __init__(self, app): + self.app = app + self.version_guesser = VersionGuesser(app) + + def generate(self, artifact, dirname): + # Try to find a directory with baserock metadata files. + metadirs = [ + os.path.join(dirname, 'factory', 'baserock'), + os.path.join(dirname, 'baserock') + ] + existing_metadirs = [x for x in metadirs if os.path.isdir(x)] + if not existing_metadirs: + raise NotASystemArtifactError(artifact) + metadir = existing_metadirs[0] + + # Collect all meta information about the system, its strata + # and its chunks that we are interested in. + artifacts = [] + for basename in glob.glob(os.path.join(metadir, '*.meta')): + metafile = os.path.join(metadir, basename) + metadata = json.load(open(metafile)) + + # Try to guess the version of this artifact + version = self.version_guesser.guess_version( + metadata['repo'], metadata['sha1']) + if version is None: + version = '' + else: + version = '-%s' % version + + fst_col = '%s.%s.%s%s' % (metadata['cache-key'][:7], + metadata['kind'], + metadata['artifact-name'], + version) + + original_ref = metadata['original_ref'] + if (metadata['kind'] in ('system', 'stratum') and + 'baserock/builds/' in original_ref): + original_ref = original_ref[: len('baserock/builds/') + 7] + + artifacts.append({ + 'kind': metadata['kind'], + 'name': metadata['artifact-name'], + 'fst_col': fst_col, + 'repo': metadata['repo'], + 'original_ref': original_ref, + 'sha1': metadata['sha1'][:7] + }) + + # Generate a format string for dumping the information. + fmt = self._generate_output_format(artifacts) + self.app.output.write(fmt % ('ARTIFACT', 'REPOSITORY', + 'REF', 'COMMIT')) + + # Print information about system, strata and chunks. + self._print_artifacts(fmt, artifacts, 'system') + self._print_artifacts(fmt, artifacts, 'stratum') + self._print_artifacts(fmt, artifacts, 'chunk') + + def _generate_output_format(self, artifacts): + colwidths = {} + for artifact in artifacts: + for key, value in artifact.iteritems(): + colwidths[key] = max(colwidths.get(key, 0), len(value)) + + return '%%-%is\t' \ + '%%-%is\t' \ + '%%-%is\t' \ + '%%-%is\n' % ( + colwidths['fst_col'], + colwidths['repo'], + colwidths['original_ref'], + colwidths['sha1']) + + def _print_artifacts(self, fmt, artifacts, kind): + for artifact in sorted(artifacts, key=lambda x: x['name']): + if artifact['kind'] == kind: + self.app.output.write(fmt % (artifact['fst_col'], + artifact['repo'], + artifact['original_ref'], + artifact['sha1'])) + + +class ArtifactInspectionPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand('generate-manifest', + self.generate_manifest, + arg_synopsis='SYSTEM-ARTIFACT') + + def disable(self): + pass + + def generate_manifest(self, args): + '''Generate a content manifest for a system image. + + Command line arguments: + + * `SYSTEM-ARTIFACT` is a filename to the system artifact + (root filesystem) for the built system. + + This command generates a manifest for a built system image. + The manifest includes the constituent artifacts, + a guess at the component version, the exact commit for + the component (commit SHA1, repository URL, git symbolic + ref), and the morphology filename. + + + The manifest includes each constituent artifact, with several + columns of data: + + * 7-char cache key with the artifact kind (system, stratum, chunk), + artifact name, and version (if guessable) added + * the git repository + * the symbolic reference + * a 7-char commit id + + Example: + + morph generate-manifest /src/cache/artifacts/foo-rootfs + + ''' + + if len(args) != 1: + raise cliapp.AppException('morph generate-manifest expects ' + 'a system image as its input') + + artifact = args[0] + + def generate_manifest(dirname): + generator = ManifestGenerator(self.app) + generator.generate(artifact, dirname) + + call_in_artifact_directory(self.app, artifact, generate_manifest) diff --git a/morphlib/plugins/branch_and_merge_plugin.py b/morphlib/plugins/branch_and_merge_plugin.py new file mode 100644 index 00000000..d816fb90 --- /dev/null +++ b/morphlib/plugins/branch_and_merge_plugin.py @@ -0,0 +1,656 @@ +# Copyright (C) 2012,2013,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import contextlib +import glob +import logging +import os +import shutil + +import morphlib + + +class BranchAndMergePlugin(cliapp.Plugin): + + '''Add subcommands for handling workspaces and system branches.''' + + def enable(self): + self.app.add_subcommand('init', self.init, arg_synopsis='[DIR]') + self.app.add_subcommand('workspace', self.workspace, arg_synopsis='') + self.app.add_subcommand( + 'checkout', self.checkout, arg_synopsis='REPO BRANCH') + self.app.add_subcommand( + 'branch', self.branch, arg_synopsis='REPO NEW [OLD]') + self.app.add_subcommand( + 'edit', self.edit, arg_synopsis='SYSTEM STRATUM [CHUNK]') + self.app.add_subcommand( + 'show-system-branch', self.show_system_branch, arg_synopsis='') + self.app.add_subcommand( + 'show-branch-root', self.show_branch_root, arg_synopsis='') + self.app.add_subcommand('foreach', self.foreach, + arg_synopsis='-- COMMAND [ARGS...]') + self.app.add_subcommand('status', self.status, + arg_synopsis='') + self.app.add_subcommand('branch-from-image', self.branch_from_image, + arg_synopsis='BRANCH') + group_branch = 'Branching Options' + self.app.settings.string(['metadata-dir'], + 'Set metadata location for branch-from-image' + ' (default: /baserock)', + metavar='DIR', + default='/baserock', + group=group_branch) + + def disable(self): + pass + + def init(self, args): + '''Initialize a workspace directory. + + Command line argument: + + * `DIR` is the directory to use as a workspace, and defaults to + the current directory. + + This creates a workspace, either in the current working directory, + or if `DIR` is given, in that directory. If the directory doesn't + exist, it is created. If it does exist, it must be empty. + + You need to run `morph init` to initialise a workspace, or none + of the other system branching tools will work: they all assume + an existing workspace. Note that a workspace only exists on your + machine, not on the git server. + + Example: + + morph init /src/workspace + cd /src/workspace + + ''' + + if not args: + args = ['.'] + elif len(args) > 1: + raise morphlib.Error('init must get at most one argument') + + ws = morphlib.workspace.create(args[0]) + self.app.status(msg='Initialized morph workspace', chatty=True) + + def workspace(self, args): + '''Show the toplevel directory of the current workspace.''' + + ws = morphlib.workspace.open('.') + self.app.output.write('%s\n' % ws.root) + + # TODO: Move this somewhere nicer + @contextlib.contextmanager + def _initializing_system_branch(self, ws, root_url, system_branch, + cached_repo, base_ref): + '''A context manager for system branches under construction. + + The purpose of this context manager is to factor out the branch + cleanup code for if an exception occurs while a branch is being + constructed. + + This could be handled by a higher order function which takes + a function to initialize the branch as a parameter, but with + statements look nicer and are more obviously about resource + cleanup. + + ''' + root_dir = ws.get_default_system_branch_directory_name(system_branch) + try: + sb = morphlib.sysbranchdir.create( + root_dir, root_url, system_branch) + gd = sb.clone_cached_repo(cached_repo, base_ref) + + yield (sb, gd) + + gd.update_submodules(self.app) + gd.update_remotes() + + except morphlib.sysbranchdir.SystemBranchDirectoryAlreadyExists as e: + logging.error('Caught exception: %s' % str(e)) + raise + except BaseException as e: + # Oops. Clean up. + logging.error('Caught exception: %s' % str(e)) + logging.info('Removing half-finished branch %s' % system_branch) + self._remove_branch_dir_safe(ws.root, root_dir) + raise + + def checkout(self, args): + '''Check out an existing system branch. + + Command line arguments: + + * `REPO` is the URL to the repository to the root repository of + a system branch. + * `BRANCH` is the name of the system branch. + + This will check out an existing system branch to an existing + workspace. You must create the workspace first. This only checks + out the root repository, not the repositories for individual + components. You need to use `morph edit` to check out those. + + Example: + + cd /src/workspace + morph checkout baserock:baserock/morphs master + + ''' + + if len(args) != 2: + raise cliapp.AppException('morph checkout needs a repo and the ' + 'name of a branch as parameters') + + root_url = args[0] + system_branch = args[1] + base_ref = system_branch + + self._require_git_user_config() + + # Open the workspace first thing, so user gets a quick error if + # we're not inside a workspace. + ws = morphlib.workspace.open('.') + + # Make sure the root repository is in the local git repository + # cache, and is up to date. + lrc, rrc = morphlib.util.new_repo_caches(self.app) + cached_repo = lrc.get_updated_repo(root_url) + + # Check the git branch exists. + cached_repo.resolve_ref(system_branch) + + with self._initializing_system_branch( + ws, root_url, system_branch, cached_repo, base_ref) as (sb, gd): + + if gd.has_fat(): + gd.fat_init() + gd.fat_pull() + + + def branch(self, args): + '''Create a new system branch. + + Command line arguments: + + * `REPO` is a repository URL. + * `NEW` is the name of the new system branch. + * `OLD` is the point from which to branch, and defaults to `master`. + + This creates a new system branch. It needs to be run in an + existing workspace (see `morph workspace`). It creates a new + git branch in the clone of the repository in the workspace. The + system branch will not be visible on the git server until you + push your changes to the repository. + + Example: + + cd /src/workspace + morph branch baserock:baserock/morphs jrandom/new-feature + + ''' + + if len(args) not in [2, 3]: + raise cliapp.AppException( + 'morph branch needs name of branch as parameter') + + root_url = args[0] + system_branch = args[1] + base_ref = 'master' if len(args) == 2 else args[2] + origin_base_ref = 'origin/%s' % base_ref + + self._require_git_user_config() + + # Open the workspace first thing, so user gets a quick error if + # we're not inside a workspace. + ws = morphlib.workspace.open('.') + + # Make sure the root repository is in the local git repository + # cache, and is up to date. + lrc, rrc = morphlib.util.new_repo_caches(self.app) + cached_repo = lrc.get_updated_repo(root_url) + + # Make sure the system branch doesn't exist yet. + if cached_repo.ref_exists(system_branch): + raise cliapp.AppException( + 'branch %s already exists in repository %s' % + (system_branch, root_url)) + + # Make sure the base_ref exists. + cached_repo.resolve_ref(base_ref) + + with self._initializing_system_branch( + ws, root_url, system_branch, cached_repo, base_ref) as (sb, gd): + + gd.branch(system_branch, base_ref) + gd.checkout(system_branch) + if gd.has_fat(): + gd.fat_init() + gd.fat_pull() + + def _save_dirty_morphologies(self, loader, sb, morphs): + logging.debug('Saving dirty morphologies: start') + for morph in morphs: + if morph.dirty: + logging.debug( + 'Saving morphology: %s %s %s' % + (morph.repo_url, morph.ref, morph.filename)) + loader.unset_defaults(morph) + loader.save_to_file( + sb.get_filename(morph.repo_url, morph.filename), morph) + morph.dirty = False + logging.debug('Saving dirty morphologies: done') + + def _checkout(self, lrc, sb, repo_url, ref): + logging.debug( + 'Checking out %s (%s) into %s' % + (repo_url, ref, sb.root_directory)) + cached_repo = lrc.get_updated_repo(repo_url) + gd = sb.clone_cached_repo(cached_repo, ref) + gd.update_submodules(self.app) + gd.update_remotes() + + def _load_morphology_from_file(self, loader, dirname, filename): + full_filename = os.path.join(dirname, filename) + return loader.load_from_file(full_filename) + + def _load_morphology_from_git(self, loader, gd, ref, filename): + try: + text = gd.get_file_from_ref(ref, filename) + except cliapp.AppException: + text = gd.get_file_from_ref('origin/%s' % ref, filename) + return loader.load_from_string(text, filename) + + def edit(self, args): + '''Edit or checkout a component in a system branch. + + Command line arguments: + + * `CHUNK` is the name of a chunk + + This makes a local checkout of CHUNK in the current system branch + and edits any stratum morphology file(s) containing the chunk + + ''' + + if len(args) != 1: + raise cliapp.AppException('morph edit needs a chunk ' + 'as parameter') + + ws = morphlib.workspace.open('.') + sb = morphlib.sysbranchdir.open_from_within('.') + loader = morphlib.morphloader.MorphologyLoader() + morphs = self._load_all_sysbranch_morphologies(sb, loader) + + def edit_chunk(morph, chunk_name): + chunk_url, chunk_ref, chunk_morph = ( + morphs.get_chunk_triplet(morph, chunk_name)) + + chunk_dirname = sb.get_git_directory_name(chunk_url) + + if not os.path.exists(chunk_dirname): + lrc, rrc = morphlib.util.new_repo_caches(self.app) + cached_repo = lrc.get_updated_repo(chunk_url) + + gd = sb.clone_cached_repo(cached_repo, chunk_ref) + system_branch_ref = gd.disambiguate_ref(sb.system_branch_name) + sha1 = gd.resolve_ref_to_commit(chunk_ref) + + try: + old_sha1 = gd.resolve_ref_to_commit(system_branch_ref) + except morphlib.gitdir.InvalidRefError as e: + pass + else: + gd.delete_ref(system_branch_ref, old_sha1) + gd.branch(sb.system_branch_name, sha1) + gd.checkout(sb.system_branch_name) + gd.update_submodules(self.app) + gd.update_remotes() + if gd.has_fat(): + gd.fat_init() + gd.fat_pull() + + # Change the refs to the chunk. + if chunk_ref != sb.system_branch_name: + morphs.change_ref( + chunk_url, chunk_ref, + chunk_morph, + sb.system_branch_name) + + return chunk_dirname + + chunk_name = args[0] + dirs = set() + found = 0 + + for morph in morphs.morphologies: + if morph['kind'] == 'stratum': + for chunk in morph['chunks']: + if chunk['name'] == chunk_name: + self.app.status( + msg='Editing %(chunk)s in %(stratum)s stratum', + chunk=chunk_name, stratum=morph['name']) + chunk_dirname = edit_chunk(morph, chunk_name) + dirs.add(chunk_dirname) + found = found + 1 + + # Save any modified strata. + + self._save_dirty_morphologies(loader, sb, morphs.morphologies) + + if found == 0: + self.app.status( + msg="No chunk %(chunk)s found. If you want to create one, add " + "an entry to a stratum morph file.", chunk=chunk_name) + + if found >= 1: + dirs_list = ', '.join(sorted(dirs)) + self.app.status( + msg="Chunk %(chunk)s source is available at %(dirs)s", + chunk=chunk_name, dirs=dirs_list) + + if found > 1: + self.app.status( + msg="Notice that this chunk appears in more than one stratum") + + def show_system_branch(self, args): + '''Show the name of the current system branch.''' + + ws = morphlib.workspace.open('.') + sb = morphlib.sysbranchdir.open_from_within('.') + self.app.output.write('%s\n' % sb.system_branch_name) + + def show_branch_root(self, args): + '''Show the name of the repository holding the system morphologies. + + This would, for example, write out something like: + + /src/ws/master/baserock/baserock/definitions + + when the master branch of the `baserock/baserock/definitions` + repository is checked out. + + ''' + + ws = morphlib.workspace.open('.') + sb = morphlib.sysbranchdir.open_from_within('.') + repo_url = sb.get_config('branch.root') + self.app.output.write('%s\n' % sb.get_git_directory_name(repo_url)) + + def _remove_branch_dir_safe(self, workspace_root, system_branch_root): + # This function avoids throwing any exceptions, so it is safe to call + # inside an 'except' block without altering the backtrace. + + def handle_error(function, path, excinfo): + logging.warning ("Error while trying to clean up %s: %s" % + (path, excinfo)) + + shutil.rmtree(system_branch_root, onerror=handle_error) + + # Remove parent directories that are empty too, avoiding exceptions + parent = os.path.dirname(system_branch_root) + while parent != os.path.abspath(workspace_root): + if len(os.listdir(parent)) > 0 or os.path.islink(parent): + break + os.rmdir(parent) + parent = os.path.dirname(parent) + + def _require_git_user_config(self): + '''Warn if the git user.name and user.email variables are not set.''' + + keys = { + 'user.name': 'My Name', + 'user.email': 'me@example.com', + } + + try: + morphlib.git.check_config_set(self.app.runcmd, keys) + except morphlib.git.ConfigNotSetException as e: + self.app.status( + msg="WARNING: %(message)s", + message=str(e), error=True) + + def foreach(self, args): + '''Run a command in each repository checked out in a system branch. + + Use -- before specifying the command to separate its arguments from + Morph's own arguments. + + Command line arguments: + + * `--` indicates the end of option processing for Morph. + * `COMMAND` is a command to run. + * `ARGS` is a list of arguments or options to be passed onto + `COMMAND`. + + This runs the given `COMMAND` in each git repository belonging + to the current system branch that exists locally in the current + workspace. This can be a handy way to do the same thing in all + the local git repositories. + + For example: + + morph foreach -- git push + + The above command would push any committed changes in each + repository to the git server. + + ''' + + if not args: + raise cliapp.AppException('morph foreach expects a command to run') + + ws = morphlib.workspace.open('.') + sb = morphlib.sysbranchdir.open_from_within('.') + + for gd in sorted(sb.list_git_directories(), key=lambda gd: gd.dirname): + # Get the repository's original name + # Continue in the case of error, since the previous iteration + # worked in the case of the user cloning a repository in the + # system branch's directory. + try: + repo = gd.get_config('morph.repository') + except cliapp.AppException: + continue + + self.app.output.write('%s\n' % repo) + status, output, error = self.app.runcmd_unchecked( + args, cwd=gd.dirname) + self.app.output.write(output) + if status != 0: + self.app.output.write(error) + pretty_command = ' '.join(cliapp.shell_quote(arg) + for arg in args) + raise cliapp.AppException( + 'Command failed at repo %s: %s' + % (repo, pretty_command)) + self.app.output.write('\n') + self.app.output.flush() + + def _load_all_sysbranch_morphologies(self, sb, loader): + '''Read in all the morphologies in the root repository.''' + self.app.status(msg='Loading in all morphologies') + morphs = morphlib.morphset.MorphologySet() + for morph in sb.load_all_morphologies(loader): + morphs.add_morphology(morph) + return morphs + + def status(self, args): + '''Show information about the current system branch or workspace + + This shows the status of every local git repository of the + current system branch. This is similar to running `git status` + in each repository separately. + + If run in a Morph workspace, but not in a system branch checkout, + it lists all checked out system branches in the workspace. + + ''' + + if args: + raise cliapp.AppException('morph status takes no arguments') + + ws = morphlib.workspace.open('.') + try: + sb = morphlib.sysbranchdir.open_from_within('.') + except morphlib.sysbranchdir.NotInSystemBranch: + self._workspace_status(ws) + else: + self._branch_status(ws, sb) + + def _workspace_status(self, ws): + '''Show information about the current workspace + + This lists all checked out system branches in the workspace. + + ''' + self.app.output.write("System branches in current workspace:\n") + branches = sorted(ws.list_system_branches(), + key=lambda x: x.root_directory) + for sb in branches: + self.app.output.write(" %s\n" % sb.get_config('branch.name')) + + def _branch_status(self, ws, sb): + '''Show information about the current branch + + This shows the status of every local git repository of the + current system branch. This is similar to running `git status` + in each repository separately. + + ''' + branch = sb.get_config('branch.name') + root = sb.get_config('branch.root') + + self.app.output.write("On branch %s, root %s\n" % (branch, root)) + + has_uncommitted_changes = False + for gd in sorted(sb.list_git_directories(), key=lambda x: x.dirname): + try: + repo = gd.get_config('morph.repository') + except cliapp.AppException: + self.app.output.write( + ' %s: not part of system branch\n' % gd.dirname) + # TODO: make this less vulnerable to a branch using + # refs/heads/foo instead of foo + head = gd.HEAD + if head != branch: + self.app.output.write( + ' %s: unexpected ref checked out %r\n' % (repo, head)) + if any(gd.get_index().get_uncommitted_changes()): + has_uncommitted_changes = True + self.app.output.write(' %s: uncommitted changes\n' % repo) + + if not has_uncommitted_changes: + self.app.output.write("\nNo repos have outstanding changes.\n") + + def branch_from_image(self, args): + '''Produce a branch of an existing system image. + + Given the metadata specified by --metadata-dir, create a new + branch then petrify it to the state of the commits at the time + the system was built. + + If --metadata-dir is not specified, it defaults to your currently + running system. + + ''' + if len(args) != 1: + raise cliapp.AppException( + "branch-from-image needs exactly 1 argument " + "of the new system branch's name") + system_branch = args[0] + metadata_path = self.app.settings['metadata-dir'] + alias_resolver = morphlib.repoaliasresolver.RepoAliasResolver( + self.app.settings['repo-alias']) + + self._require_git_user_config() + + ws = morphlib.workspace.open('.') + + system, metadata = self._load_system_metadata(metadata_path) + resolved_refs = dict(self._resolve_refs_from_metadata(alias_resolver, + metadata)) + self.app.status(msg='Resolved refs: %r' % resolved_refs) + base_ref = system['sha1'] + # The previous version would fall back to deducing this from the repo + # url and the repo alias resolver, but this does not always work, and + # new systems always have repo-alias in the metadata + root_url = system['repo-alias'] + + lrc, rrc = morphlib.util.new_repo_caches(self.app) + cached_repo = lrc.get_updated_repo(root_url) + + + with self._initializing_system_branch( + ws, root_url, system_branch, cached_repo, base_ref) as (sb, gd): + + # TODO: It's nasty to clone to a sha1 then create a branch + # of that sha1 then check it out, a nicer API may be the + # initial clone not checking out a branch at all, then + # the user creates and checks out their own branches + gd.branch(system_branch, base_ref) + gd.checkout(system_branch) + + loader = morphlib.morphloader.MorphologyLoader() + morphs = self._load_all_sysbranch_morphologies(sb, loader) + + morphs.repoint_refs(sb.root_repository_url, + sb.system_branch_name) + + morphs.petrify_chunks(resolved_refs) + + self._save_dirty_morphologies(loader, sb, morphs.morphologies) + + @staticmethod + def _load_system_metadata(path): + '''Load all metadata in `path` corresponding to a single System. + ''' + + smd = morphlib.systemmetadatadir.SystemMetadataDir(path) + metadata = smd.values() + systems = [md for md in metadata + if 'kind' in md and md['kind'] == 'system'] + + if not systems: + raise cliapp.AppException( + 'Metadata directory does not contain any systems.') + if len(systems) > 1: + raise cliapp.AppException( + 'Metadata directory contains multiple systems.') + system_metadatum = systems[0] + + metadata_cache_id_lookup = dict((md['cache-key'], md) + for md in metadata + if 'cache-key' in md) + + return system_metadatum, metadata_cache_id_lookup + + @staticmethod + def _resolve_refs_from_metadata(alias_resolver, metadata): + '''Pre-resolve a set of refs from existing metadata. + + Given the metadata, generate a mapping of all the (repo, ref) + pairs defined in the metadata and the commit id they resolved to. + + ''' + for md in metadata.itervalues(): + repourls = set((md['repo-alias'], md['repo'])) + repourls.update(alias_resolver.aliases_from_url(md['repo'])) + for repourl in repourls: + yield ((repourl, md['original_ref']), md['sha1']) diff --git a/morphlib/plugins/build_plugin.py b/morphlib/plugins/build_plugin.py new file mode 100644 index 00000000..64630c2b --- /dev/null +++ b/morphlib/plugins/build_plugin.py @@ -0,0 +1,193 @@ +# Copyright (C) 2012,2013,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import contextlib +import uuid + +import morphlib + + +class BuildPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand('build-morphology', self.build_morphology, + arg_synopsis='(REPO REF FILENAME)...') + self.app.add_subcommand('build', self.build, + arg_synopsis='SYSTEM') + self.app.add_subcommand('distbuild-morphology', + self.distbuild_morphology, + arg_synopsis='SYSTEM') + self.app.add_subcommand('distbuild', self.distbuild, + arg_synopsis='SYSTEM') + self.use_distbuild = False + + def disable(self): + self.use_distbuild = False + + def distbuild_morphology(self, args): + '''Distbuild a system, outside of a system branch. + + Command line arguments: + + * `REPO` is a git repository URL. + * `REF` is a branch or other commit reference in that repository. + * `FILENAME` is a morphology filename at that ref. + + See 'help distbuild' and 'help build-morphology' for more information. + + ''' + + addr = self.app.settings['controller-initiator-address'] + port = self.app.settings['controller-initiator-port'] + + build_command = morphlib.buildcommand.InitiatorBuildCommand( + self.app, addr, port) + build_command.build(args) + + def distbuild(self, args): + '''Distbuild a system image in the current system branch + + Command line arguments: + + * `SYSTEM` is the name of the system to build. + + This command launches a distributed build, to use this command + you must first set up a distbuild cluster. + + Artifacts produced during the build will be stored on your trove. + + Once the build completes you can use morph deploy to the deploy + your system, the system artifact will be copied from your trove + and cached locally. + + Example: + + morph distbuild devel-system-x86_64-generic.morph + + ''' + + self.use_distbuild = True + self.build(args) + + def build_morphology(self, args): + '''Build a system, outside of a system branch. + + Command line arguments: + + * `REPO` is a git repository URL. + * `REF` is a branch or other commit reference in that repository. + * `FILENAME` is a morphology filename at that ref. + + You probably want `morph build` instead. However, in some + cases it is more convenient to not have to create a Morph + workspace and check out the relevant system branch, and only + just run the build. For those times, this command exists. + + This subcommand does not automatically commit changes to a + temporary branch, so you can only build from properly committed + sources that have been pushed to the git server. + + Example: + + morph build-morphology baserock:baserock/definitions \ + master devel-system-x86_64-generic.morph + + ''' + + # Raise an exception if there is not enough space + morphlib.util.check_disk_available( + self.app.settings['tempdir'], + self.app.settings['tempdir-min-space'], + self.app.settings['cachedir'], + self.app.settings['cachedir-min-space']) + + build_command = morphlib.buildcommand.BuildCommand(self.app) + build_command.build(args) + + def build(self, args): + '''Build a system image in the current system branch + + Command line arguments: + + * `SYSTEM` is the name of the system to build. + + This builds a system image, and any of its components that + need building. The system name is the basename of the system + morphology, in the root repository of the current system branch, + without the `.morph` suffix in the filename. + + The location of the resulting system image artifact is printed + at the end of the build output. + + You do not need to commit your changes before building, Morph + does that for you, in a temporary branch for each build. However, + note that Morph does not untracked files to the temporary branch, + only uncommitted changes to files git already knows about. You + need to `git add` and commit each new file yourself. + + Example: + + morph build devel-system-x86_64-generic.morph + + ''' + + if len(args) != 1: + raise cliapp.AppException('morph build expects exactly one ' + 'parameter: the system to build') + + # Raise an exception if there is not enough space + morphlib.util.check_disk_available( + self.app.settings['tempdir'], + self.app.settings['tempdir-min-space'], + self.app.settings['cachedir'], + self.app.settings['cachedir-min-space']) + + system_filename = morphlib.util.sanitise_morphology_path(args[0]) + + ws = morphlib.workspace.open('.') + sb = morphlib.sysbranchdir.open_from_within('.') + + build_uuid = uuid.uuid4().hex + + if self.use_distbuild: + addr = self.app.settings['controller-initiator-address'] + port = self.app.settings['controller-initiator-port'] + + build_command = morphlib.buildcommand.InitiatorBuildCommand( + self.app, addr, port) + else: + build_command = morphlib.buildcommand.BuildCommand(self.app) + + loader = morphlib.morphloader.MorphologyLoader() + push = self.app.settings['push-build-branches'] + name = morphlib.git.get_user_name(self.app.runcmd) + email = morphlib.git.get_user_email(self.app.runcmd) + build_ref_prefix = self.app.settings['build-ref-prefix'] + + self.app.status(msg='Starting build %(uuid)s', uuid=build_uuid) + self.app.status(msg='Collecting morphologies involved in ' + 'building %(system)s from %(branch)s', + system=system_filename, + branch=sb.system_branch_name) + + bb = morphlib.buildbranch.BuildBranch(sb, build_ref_prefix) + pbb = morphlib.buildbranch.pushed_build_branch( + bb, loader=loader, changes_need_pushing=push, + name=name, email=email, build_uuid=build_uuid, + status=self.app.status) + with pbb as (repo, ref): + build_command.build([repo, ref, system_filename]) diff --git a/morphlib/plugins/cross-bootstrap_plugin.py b/morphlib/plugins/cross-bootstrap_plugin.py new file mode 100644 index 00000000..7b53a4a5 --- /dev/null +++ b/morphlib/plugins/cross-bootstrap_plugin.py @@ -0,0 +1,306 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import cliapp +import logging +import os.path +import re +import tarfile +import traceback + +import morphlib + +driver_header = '''#!/bin/sh +echo "Morph native bootstrap script" +echo "Generated by Morph version %s\n" + +set -eu + +export PATH=$PATH:/tools/bin:/tools/sbin +export SRCDIR=/src + +''' % morphlib.__version__ + +driver_footer = ''' + +echo "Complete!" +''' + +def escape_source_name(source): + repo_name = source.repo.original_name + ref = source.original_ref + source_name = '%s__%s' % (repo_name, ref) + return re.sub('[:/]', '_', source_name) + +# Most of this is ripped from RootfsTarballBuilder, and should be reconciled +# with it. +class BootstrapSystemBuilder(morphlib.builder2.BuilderBase): + '''Build a bootstrap system tarball + + The bootstrap system image contains a minimal cross-compiled toolchain + and a set of extracted sources for the rest of the system, with shell + scripts to run the required morphology commands. This allows new + architectures to be bootstrapped without needing to build Python, Git, + Perl and all of Morph's other dependencies first. + ''' + + def build_and_cache(self): + with self.build_watch('overall-build'): + for system_name, artifact in self.source.artifacts.iteritems(): + handle = self.local_artifact_cache.put(artifact) + fs_root = self.staging_area.destdir(self.source) + try: + self.unpack_binary_chunks(fs_root) + self.unpack_sources(fs_root) + self.write_build_script(fs_root) + self.create_tarball(handle, fs_root, system_name) + except BaseException, e: + logging.error(traceback.format_exc()) + self.app.status(msg='Error while building bootstrap image', + error=True) + handle.abort() + raise + + handle.close() + + self.save_build_times() + return self.source.artifacts.items() + + def unpack_binary_chunks(self, dest): + cache = self.local_artifact_cache + for chunk_source in self.source.cross_sources: + for chunk_artifact in chunk_source.artifacts.itervalues(): + with cache.get(chunk_artifact) as chunk_file: + try: + morphlib.bins.unpack_binary_from_file(chunk_file, dest) + except BaseException, e: + self.app.status( + msg='Error unpacking binary chunk %(name)s', + name=chunk_artifact.name, + error=True) + raise + + def unpack_sources(self, path): + # Multiple chunks sources may be built from the same repo ('linux' + # and 'linux-api-headers' are a good example), so we check out the + # sources once per repository. + # + # It might be neater to build these as "source artifacts" individually, + # but that would waste huge amounts of space in the artifact cache. + for s in self.source.native_sources: + escaped_source = escape_source_name(s) + source_dir = os.path.join(path, 'src', escaped_source) + if not os.path.exists(source_dir): + os.makedirs(source_dir) + morphlib.builder2.extract_sources( + self.app, self.repo_cache, s.repo, s.sha1, source_dir) + + name = s.name + chunk_script = os.path.join(path, 'src', 'build-%s' % name) + with morphlib.savefile.SaveFile(chunk_script, 'w') as f: + self.write_chunk_build_script(s, f) + os.chmod(chunk_script, 0777) + + def write_build_script(self, path): + '''Output a script to run build on the bootstrap target''' + + driver_script = os.path.join(path, 'native-bootstrap') + with morphlib.savefile.SaveFile(driver_script, 'w') as f: + f.write(driver_header) + + f.write('echo Setting up build environment...\n') + for k,v in self.staging_area.env.iteritems(): + if k != 'PATH': + f.write('export %s="%s"\n' % (k, v)) + + for s in self.source.native_sources: + name = s.name + f.write('\necho Building %s\n' % name) + f.write('mkdir /%s.inst\n' % name) + f.write('env DESTDIR=/%s.inst $SRCDIR/build-%s\n' + % (name, name)) + f.write('echo Installing %s\n' % name) + f.write('(cd /%s.inst; find . | cpio -umdp /)\n' % name) + f.write('if [ -e /sbin/ldconfig ]; then /sbin/ldconfig; fi\n') + + f.write(driver_footer) + os.chmod(driver_script, 0777) + + def write_chunk_build_script(self, source, f): + m = source.morphology + f.write('#!/bin/sh\n') + f.write('# Build script generated by morph\n') + f.write('set -e\n') + f.write('chunk_name=%s\n' % m['name']) + + repo = escape_source_name(source) + f.write('cp -a $SRCDIR/%s $DESTDIR/$chunk_name.build\n' % repo) + f.write('cd $DESTDIR/$chunk_name.build\n') + f.write('export PREFIX=%s\n' % source.prefix) + + bs = morphlib.buildsystem.lookup_build_system(m['build-system']) + + # FIXME: merge some of this with Morphology + steps = [ + ('pre-configure', False), + ('configure', False), + ('post-configure', False), + ('pre-build', True), + ('build', True), + ('post-build', True), + ('pre-test', False), + ('test', False), + ('post-test', False), + ('pre-install', False), + ('install', False), + ('post-install', False), + ] + + for step, in_parallel in steps: + key = '%s-commands' % step + cmds = m[key] + for cmd in cmds: + f.write('(') + if in_parallel: + max_jobs = m['max-jobs'] + if max_jobs is None: + max_jobs = self.max_jobs + f.write('export MAKEFLAGS=-j%s; ' % max_jobs) + f.write('set -e; %s) || exit 1\n' % cmd) + + f.write('rm -Rf $DESTDIR/$chunk_name.build') + + def create_tarball(self, handle, fs_root, system_name): + unslashy_root = fs_root[1:] + def uproot_info(info): + info.name = os.path.relpath(info.name, unslashy_root) + if info.islnk(): + info.linkname = os.path.relpath(info.linkname, unslashy_root) + return info + + tar = tarfile.TarFile.gzopen(fileobj=handle, mode="w", + compresslevel=1, + name=system_name) + self.app.status(msg='Constructing tarball of root filesystem', + chatty=True) + tar.add(fs_root, recursive=True, filter=uproot_info) + tar.close() + + +class CrossBootstrapPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand('cross-bootstrap', + self.cross_bootstrap, + arg_synopsis='TARGET REPO REF SYSTEM-MORPH') + + def disable(self): + pass + + def cross_bootstrap(self, args): + '''Cross-bootstrap a system from a different architecture.''' + + # A brief overview of this process: the goal is to native build as much + # of the system as possible because that's easier, but in order to do + # so we need at least 'build-essential'. 'morph cross-bootstrap' will + # cross-build any bootstrap-mode chunks in the given system and + # will then prepare a large rootfs tarball which, when booted, will + # build the rest of the chunks in the system using the cross-built + # build-essential. + # + # This approach saves us from having to run Morph, Git, Python, Perl, + # or anything else complex and difficult to cross-build on the target + # until it is bootstrapped. The user of this command needs to provide + # a kernel and handle booting the system themselves (the generated + # tarball contains a /bin/sh that can be used as 'init'. + # + # This function is a variant of the BuildCommand() class in morphlib. + + # To do: make it work on a system branch instead of repo/ref/morph + # triplet. + + if len(args) < 4: + raise cliapp.AppException( + 'cross-bootstrap requires 4 arguments: target archicture, and ' + 'repo, ref and and name of the system morphology') + + arch = args[0] + root_repo, ref, system_name = args[1:4] + + if arch not in morphlib.valid_archs: + raise morphlib.Error('Unsupported architecture "%s"' % arch) + + # Get system artifact + + build_env = morphlib.buildenvironment.BuildEnvironment( + self.app.settings, arch) + build_command = morphlib.buildcommand.BuildCommand(self.app, build_env) + + morph_name = morphlib.util.sanitise_morphology_path(system_name) + srcpool = build_command.create_source_pool(root_repo, ref, morph_name) + + # FIXME: this is a quick fix in order to get it working for + # Baserock 13 release, it is not a reasonable fix + def validate(self, root_artifact): + root_arch = root_artifact.source.morphology['arch'] + target_arch = arch + if root_arch != target_arch: + raise morphlib.Error( + 'Target architecture is %s ' + 'but the system architecture is %s' + % (target_arch, root_arch)) + + morphlib.buildcommand.BuildCommand._validate_architecture = validate + + system_artifact = build_command.resolve_artifacts(srcpool) + + # Calculate build order + # This is basically a hacked version of BuildCommand.build_in_order() + sources = build_command.get_ordered_sources(system_artifact.walk()) + cross_sources = [] + native_sources = [] + for s in sources: + if s.morphology['kind'] == 'chunk': + if s.build_mode == 'bootstrap': + cross_sources.append(s) + else: + native_sources.append(s) + + if len(cross_sources) == 0: + raise morphlib.Error( + 'Nothing to cross-compile. Only chunks built in \'bootstrap\' ' + 'mode can be cross-compiled.') + + for s in cross_sources: + build_command.cache_or_build_source(s, build_env) + + for s in native_sources: + build_command.fetch_sources(s) + + # Install those to the output tarball ... + self.app.status(msg='Building final bootstrap system image') + system_artifact.source.cross_sources = cross_sources + system_artifact.source.native_sources = native_sources + staging_area = build_command.create_staging_area( + build_env, use_chroot=False) + builder = BootstrapSystemBuilder( + self.app, staging_area, build_command.lac, build_command.rac, + system_artifact.source, build_command.lrc, 1, False) + builder.build_and_cache() + + self.app.status( + msg='Bootstrap tarball for %(name)s is cached at %(cachepath)s', + name=system_artifact.name, + cachepath=build_command.lac.artifact_filename(system_artifact)) diff --git a/morphlib/plugins/deploy_plugin.py b/morphlib/plugins/deploy_plugin.py new file mode 100644 index 00000000..2bc53a0d --- /dev/null +++ b/morphlib/plugins/deploy_plugin.py @@ -0,0 +1,613 @@ +# Copyright (C) 2013, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import json +import logging +import os +import shutil +import sys +import tarfile +import tempfile +import uuid + +import cliapp +import morphlib + + +class DeployPlugin(cliapp.Plugin): + + def enable(self): + group_deploy = 'Deploy Options' + self.app.settings.boolean(['upgrade'], + 'specify that you want to upgrade an ' + 'existing cluster. Deprecated: use the ' + '`morph upgrade` command instead', + group=group_deploy) + self.app.add_subcommand( + 'deploy', self.deploy, + arg_synopsis='CLUSTER [DEPLOYMENT...] [SYSTEM.KEY=VALUE]') + self.app.add_subcommand( + 'upgrade', self.upgrade, + arg_synopsis='CLUSTER [DEPLOYMENT...] [SYSTEM.KEY=VALUE]') + + def disable(self): + pass + + def deploy(self, args): + '''Deploy a built system image or a set of images. + + Command line arguments: + + * `CLUSTER` is the name of the cluster to deploy. + + * `DEPLOYMENT...` is the name of zero or more deployments in the + morphology to deploy. If none are specified then all deployments + in the morphology are deployed. + + * `SYSTEM.KEY=VALUE` can be used to assign `VALUE` to a parameter + named `KEY` for the system identified by `SYSTEM` in the cluster + morphology (see below). This will override parameters defined + in the morphology. + + Morph deploys a set of systems listed in a cluster morphology. + "Deployment" here is quite a general concept: it covers anything + where a system image is taken, configured, and then put somewhere + where it can be run. The deployment mechanism is quite flexible, + and can be extended by the user. + + A cluster morphology defines a list of systems to deploy, and + for each system a list of ways to deploy them. It contains the + following fields: + + * **name**: MUST be the same as the basename of the morphology + filename, sans .morph suffix. + + * **kind**: MUST be `cluster`. + + * **systems**: a list of systems to deploy; + the value is a list of mappings, where each mapping has the + following keys: + + * **morph**: the system morphology to use in the specified + commit. + + * **deploy**: a mapping where each key identifies a + system and each system has at least the following keys: + + * **type**: identifies the type of development e.g. (kvm, + nfsboot) (see below). + * **location**: where the deployed system should end up + at. The syntax depends on the deployment type (see below). + Any additional item on the dictionary will be added to the + environment as `KEY=VALUE`. + + * **deploy-defaults**: allows multiple deployments of the same + system to share some settings, when they can. Default settings + will be overridden by those defined inside the deploy mapping. + + # Example + + name: cluster-foo + kind: cluster + systems: + - morph: devel-system-x86_64-generic.morph + deploy: + cluster-foo-x86_64-1: + type: kvm + location: kvm+ssh://user@host/x86_64-1/x86_64-1.img + HOSTNAME: cluster-foo-x86_64-1 + DISK_SIZE: 4G + RAM_SIZE: 4G + VCPUS: 2 + - morph: devel-system-armv7-highbank + deploy-defaults: + type: nfsboot + location: cluster-foo-nfsboot-server + deploy: + cluster-foo-armv7-1: + HOSTNAME: cluster-foo-armv7-1 + cluster-foo-armv7-2: + HOSTNAME: cluster-foo-armv7-2 + + Each system defined in a cluster morphology can be deployed in + multiple ways (`type` in a cluster morphology). Morph provides + five types of deployment: + + * `tar` where Morph builds a tar archive of the root file system. + + * `rawdisk` where Morph builds a raw disk image and sets up the + image with a bootloader and configuration so that it can be + booted. Disk size is set with `DISK_SIZE` (see below). + + * `virtualbox-ssh` where Morph creates a VirtualBox disk image, + and creates a new virtual machine on a remote host, accessed + over ssh. Disk and RAM size are set with `DISK_SIZE` and + `RAM_SIZE` (see below). + + * `kvm`, which is similar to `virtualbox-ssh`, but uses libvirt + and KVM instead of VirtualBox. Disk and RAM size are set with + `DISK_SIZE` and `RAM_SIZE` (see below). + + * `nfsboot` where Morph creates a system to be booted over + a network. + + In addition to the deployment type, the user must also give + a value for `location`. Its syntax depends on the deployment + types. The deployment types provided by Morph use the + following syntaxes: + + * `tar`: pathname to the tar archive to be created; for + example, `/home/alice/testsystem.tar` + + * `rawdisk`: pathname to the disk image to be created; for + example, `/home/alice/testsystem.img` + + * `virtualbox-ssh` and `kvm`: a custom URL scheme that + provides the target host machine (the one that runs + VirtualBox or `kvm`), the name of the new virtual machine, + and the location on the target host of the virtual disk + file. The target host is accessed over ssh. For example, + `vbox+ssh://alice@192.168.122.1/testsys/home/alice/testsys.vdi` + or `kvm+ssh://alice@192.168.122.1/testsys/home/alice/testys.img` + where + + * `alice@192.168.122.1` is the target as given to ssh, + **from within the development host** (which may be + different from the target host's normal address); + + * `testsys` is the new VM's name; + + * `/home/alice/testsys.vdi` and `/home/alice/testys.img` are + the pathnames of the disk image files on the target host. + + * `nfsboot`: the address of the nfsboot server. (Note this is just + the _address_ of the trove, _not_ `user@...`, since `root@` will + automatically be prepended to the server address.) + + The following `KEY=VALUE` parameters are supported for `rawdisk`, + `virtualbox-ssh` and `kvm` and deployment types: + + * `DISK_SIZE=X` to set the size of the disk image. `X` should use a + suffix of `K`, `M`, or `G` (in upper or lower case) to indicate + kilo-, mega-, or gigabytes. For example, `DISK_SIZE=100G` would + create a 100 gigabyte disk image. **This parameter is mandatory**. + + The `kvm` and `virtualbox-ssh` deployment types support an additional + parameter: + + * `RAM_SIZE=X` to set the size of virtual RAM for the virtual + machine. `X` is interpreted in the same was as `DISK_SIZE`, + and defaults to `1G`. + + * `AUTOSTART=<VALUE>` - allowed values are `yes` and `no` + (default) + + For the `nfsboot` write extension, + + * the following `KEY=VALUE` pairs are mandatory + + * `NFSBOOT_CONFIGURE=yes` (or any non-empty value). This + enables the `nfsboot` configuration extension (see + below) which MUST be used when using the `nfsboot` + write extension. + + * `HOSTNAME=<STRING>` a unique identifier for that system's + `nfs` root when it's deployed on the nfsboot server - the + extension creates a directory with that name for the `nfs` + root, and stores kernels by that name for the tftp server. + + * the following `KEY=VALUE` pairs are optional + + * `VERSION_LABEL=<STRING>` - set the name of the system + version being deployed, when upgrading. Defaults to + "factory". + + Each deployment type is implemented by a **write extension**. The + ones provided by Morph are listed above, but users may also + create their own by adding them in the same git repository + and branch as the system morphology. A write extension is a + script that does whatever is needed for the deployment. A write + extension is passed two command line parameters: the name of an + unpacked directory tree that contains the system files (after + configuration, see below), and the `location` parameter. + + Regardless of the type of deployment, the image may be + configured for a specific deployment by using **configuration + extensions**. The extensions are listed in the system morphology + file: + + ... + configuration-extensions: + - set-hostname + + The above specifies that the extension `set-hostname` is to + be run. Morph will run all the configuration extensions listed + in the system morphology, and no others. (This way, configuration + is more easily tracked in git.) + + Configuration extensions are scripts that get the unpacked + directory tree of the system as their parameter, and do whatever + is needed to configure the tree. + + Morph provides the following configuration extension built in: + + * `set-hostname` sets the hostname of the system to the value + of the `HOSTNAME` variable. + * `nfsboot` configures the system for nfsbooting. This MUST + be used when deploying with the `nfsboot` write extension. + + Any `KEY=VALUE` parameters given in `deploy` or `deploy-defaults` + sections of the cluster morphology, or given through the command line + are set as environment variables when either the configuration or the + write extension runs (except `type` and `location`). + + Deployment configuration is stored in the deployed system as + /baserock/deployment.meta. THIS CONTAINS ALL ENVIRONMENT VARIABLES SET + DURING DEPLOYMENT, so make sure you have no sensitive information in + your environment that is being leaked. As a special case, any + environment/deployment variable that contains 'PASSWORD' in its name is + stripped out and not stored in the final system. + + ''' + + # Nasty hack to allow deploying things of a different architecture + def validate(self, root_artifact): + pass + morphlib.buildcommand.BuildCommand._validate_architecture = validate + + if not args: + raise cliapp.AppException( + 'Too few arguments to deploy command (see help)') + + # Raise an exception if there is not enough space in tempdir + # / for the path and 0 for the minimum size is a no-op + # it exists because it is complicated to check the available + # disk space given dirs may be on the same device + morphlib.util.check_disk_available( + self.app.settings['tempdir'], + self.app.settings['tempdir-min-space'], + '/', 0) + + self.app.settings['no-git-update'] = True + cluster_filename = morphlib.util.sanitise_morphology_path(args[0]) + + ws = morphlib.workspace.open('.') + sb = morphlib.sysbranchdir.open_from_within('.') + + build_uuid = uuid.uuid4().hex + + build_command = morphlib.buildcommand.BuildCommand(self.app) + build_command = self.app.hookmgr.call('new-build-command', + build_command) + loader = morphlib.morphloader.MorphologyLoader() + name = morphlib.git.get_user_name(self.app.runcmd) + email = morphlib.git.get_user_email(self.app.runcmd) + build_ref_prefix = self.app.settings['build-ref-prefix'] + root_repo_dir = morphlib.gitdir.GitDirectory( + sb.get_git_directory_name(sb.root_repository_url)) + cluster_text = root_repo_dir.read_file(cluster_filename) + cluster_morphology = loader.load_from_string(cluster_text, + filename=cluster_filename) + + if cluster_morphology['kind'] != 'cluster': + raise cliapp.AppException( + "Error: morph deployment commands are only supported for " + "cluster morphologies.") + + # parse the rest of the args + all_subsystems = set() + all_deployments = set() + deployments = set() + for system in cluster_morphology['systems']: + all_deployments.update(system['deploy'].iterkeys()) + if 'subsystems' in system: + all_subsystems.update(loader._get_subsystem_names(system)) + for item in args[1:]: + if not item in all_deployments: + break + deployments.add(item) + env_vars = args[len(deployments) + 1:] + self.validate_deployment_options( + env_vars, all_deployments, all_subsystems) + + bb = morphlib.buildbranch.BuildBranch(sb, build_ref_prefix) + pbb = morphlib.buildbranch.pushed_build_branch( + bb, loader=loader, changes_need_pushing=False, + name=name, email=email, build_uuid=build_uuid, + status=self.app.status) + with pbb as (repo, ref): + # Create a tempdir for this deployment to work in + deploy_tempdir = tempfile.mkdtemp( + dir=os.path.join(self.app.settings['tempdir'], 'deployments')) + try: + for system in cluster_morphology['systems']: + self.deploy_system(build_command, deploy_tempdir, + root_repo_dir, repo, ref, system, + env_vars, deployments, + parent_location='') + finally: + shutil.rmtree(deploy_tempdir) + + self.app.status(msg='Finished deployment') + + def validate_deployment_options( + self, env_vars, all_deployments, all_subsystems): + for var in env_vars: + for subsystem in all_subsystems: + if subsystem == var: + raise cliapp.AppException( + 'Cannot directly deploy subsystems. Create a top ' + 'level deployment for the subsystem %s instead.' % + subsystem) + if (not any(deployment in var + for deployment in all_deployments) + and not subsystem in var): + raise cliapp.AppException( + 'Variable referenced a non-existent deployment ' + 'name: %s' % var) + + def deploy_system(self, build_command, deploy_tempdir, + root_repo_dir, build_repo, ref, system, env_vars, + deployment_filter, parent_location): + sys_ids = set(system['deploy'].iterkeys()) + if deployment_filter and not \ + any(sys_id in deployment_filter for sys_id in sys_ids): + return + old_status_prefix = self.app.status_prefix + system_status_prefix = '%s[%s]' % (old_status_prefix, system['morph']) + self.app.status_prefix = system_status_prefix + try: + # Find the artifact to build + morph = morphlib.util.sanitise_morphology_path(system['morph']) + srcpool = build_command.create_source_pool(build_repo, ref, morph) + + artifact = build_command.resolve_artifacts(srcpool) + + deploy_defaults = system.get('deploy-defaults', {}) + for system_id, deploy_params in system['deploy'].iteritems(): + if not system_id in deployment_filter and deployment_filter: + continue + deployment_status_prefix = '%s[%s]' % ( + system_status_prefix, system_id) + self.app.status_prefix = deployment_status_prefix + try: + user_env = morphlib.util.parse_environment_pairs( + os.environ, + [pair[len(system_id)+1:] + for pair in env_vars + if pair.startswith(system_id)]) + + final_env = dict(deploy_defaults.items() + + deploy_params.items() + + user_env.items()) + + is_upgrade = ('yes' if self.app.settings['upgrade'] + else 'no') + final_env['UPGRADE'] = is_upgrade + + deployment_type = final_env.pop('type', None) + if not deployment_type: + raise morphlib.Error('"type" is undefined ' + 'for system "%s"' % system_id) + + location = final_env.pop('location', None) + if not location: + raise morphlib.Error('"location" is undefined ' + 'for system "%s"' % system_id) + + morphlib.util.sanitize_environment(final_env) + self.check_deploy(root_repo_dir, ref, deployment_type, + location, final_env) + system_tree = self.setup_deploy(build_command, + deploy_tempdir, + root_repo_dir, + ref, artifact, + deployment_type, + location, final_env) + for subsystem in system.get('subsystems', []): + self.deploy_system(build_command, deploy_tempdir, + root_repo_dir, build_repo, + ref, subsystem, env_vars, [], + parent_location=system_tree) + if parent_location: + deploy_location = os.path.join(parent_location, + location.lstrip('/')) + else: + deploy_location = location + self.run_deploy_commands(deploy_tempdir, final_env, + artifact, root_repo_dir, + ref, deployment_type, + system_tree, deploy_location) + finally: + self.app.status_prefix = system_status_prefix + finally: + self.app.status_prefix = old_status_prefix + + def upgrade(self, args): + '''Upgrade an existing set of instances using built images. + + See `morph help deploy` for documentation. + + ''' + + if not args: + raise cliapp.AppException( + 'Too few arguments to upgrade command (see `morph help ' + 'deploy`)') + + if self.app.settings['upgrade']: + raise cliapp.AppException( + 'Running `morph upgrade --upgrade` does not make sense.') + + self.app.settings['upgrade'] = True + self.deploy(args) + + def check_deploy(self, root_repo_dir, ref, deployment_type, location, env): + # Run optional write check extension. These are separate from the write + # extension because it may be several minutes before the write + # extension itself has the chance to raise an error. + try: + self._run_extension( + root_repo_dir, deployment_type, '.check', + [location], env) + except morphlib.extensions.ExtensionNotFoundError: + pass + + def setup_deploy(self, build_command, deploy_tempdir, root_repo_dir, ref, + artifact, deployment_type, location, env): + # deployment_type, location and env are only used for saving metadata + + # Create a tempdir to extract the rootfs in + system_tree = tempfile.mkdtemp(dir=deploy_tempdir) + + try: + # Unpack the artifact (tarball) to a temporary directory. + self.app.status(msg='Unpacking system for configuration') + + if build_command.lac.has(artifact): + f = build_command.lac.get(artifact) + elif build_command.rac.has(artifact): + build_command.cache_artifacts_locally([artifact]) + f = build_command.lac.get(artifact) + else: + raise cliapp.AppException('Deployment failed as system is' + ' not yet built.\nPlease ensure' + ' the system is built before' + ' deployment.') + tf = tarfile.open(fileobj=f) + tf.extractall(path=system_tree) + + self.app.status( + msg='System unpacked at %(system_tree)s', + system_tree=system_tree) + + self.app.status( + msg='Writing deployment metadata file') + metadata = self.create_metadata( + artifact, root_repo_dir, deployment_type, location, env) + metadata_path = os.path.join( + system_tree, 'baserock', 'deployment.meta') + with morphlib.savefile.SaveFile(metadata_path, 'w') as f: + json.dump(metadata, f, indent=4, + sort_keys=True, encoding='unicode-escape') + return system_tree + except Exception: + shutil.rmtree(system_tree) + raise + + def run_deploy_commands(self, deploy_tempdir, env, artifact, root_repo_dir, + ref, deployment_type, system_tree, location): + # Extensions get a private tempdir so we can more easily clean + # up any files an extension left behind + deploy_private_tempdir = tempfile.mkdtemp(dir=deploy_tempdir) + env['TMPDIR'] = deploy_private_tempdir + + try: + # Run configuration extensions. + self.app.status(msg='Configure system') + names = artifact.source.morphology['configuration-extensions'] + for name in names: + self._run_extension( + root_repo_dir, + name, + '.configure', + [system_tree], + env) + + # Run write extension. + self.app.status(msg='Writing to device') + self._run_extension( + root_repo_dir, + deployment_type, + '.write', + [system_tree, location], + env) + + finally: + # Cleanup. + self.app.status(msg='Cleaning up') + shutil.rmtree(deploy_private_tempdir) + + def _report_extension_stdout(self, line): + self.app.status(msg=line.replace('%s', '%%')) + def _report_extension_stderr(self, error_list): + def cb(line): + error_list.append(line) + sys.stderr.write('%s\n' % line) + return cb + def _report_extension_logger(self, name, kind): + return lambda line: logging.debug('%s%s: %s', name, kind, line) + def _run_extension(self, gd, name, kind, args, env): + '''Run an extension. + + The ``kind`` should be either ``.configure`` or ``.write``, + depending on the kind of extension that is sought. + + The extension is found either in the git repository of the + system morphology (repo, ref), or with the Morph code. + + ''' + error_list = [] + with morphlib.extensions.get_extension_filename(name, kind) as fn: + ext = morphlib.extensions.ExtensionSubprocess( + report_stdout=self._report_extension_stdout, + report_stderr=self._report_extension_stderr(error_list), + report_logger=self._report_extension_logger(name, kind), + ) + returncode = ext.run(fn, args, env=env, cwd=gd.dirname) + if returncode == 0: + logging.info('%s%s succeeded', name, kind) + else: + message = '%s%s failed with code %s: %s' % ( + name, kind, returncode, '\n'.join(error_list)) + raise cliapp.AppException(message) + + def create_metadata(self, system_artifact, root_repo_dir, deployment_type, + location, env): + '''Deployment-specific metadata. + + The `build` and `deploy` operations must be from the same ref, so full + info on the root repo that the system came from is in + /baserock/${system_artifact}.meta and is not duplicated here. We do + store a `git describe` of the definitions.git repo as a convenience for + post-upgrade hooks that we may need to implement at a future date: + the `git describe` output lists the last tag, which will hopefully help + us to identify which release of a system was deployed without having to + keep a list of SHA1s somewhere or query a Trove. + + ''' + + def remove_passwords(env): + is_password = morphlib.util.env_variable_is_password + return { k:v for k, v in env.iteritems() if not is_password(k) } + + meta = { + 'system-artifact-name': system_artifact.name, + 'configuration': remove_passwords(env), + 'deployment-type': deployment_type, + 'location': location, + 'definitions-version': { + 'describe': root_repo_dir.describe(), + }, + 'morph-version': { + 'ref': morphlib.gitversion.ref, + 'tree': morphlib.gitversion.tree, + 'commit': morphlib.gitversion.commit, + 'version': morphlib.gitversion.version, + }, + } + + return meta diff --git a/morphlib/plugins/distbuild_plugin.py b/morphlib/plugins/distbuild_plugin.py new file mode 100644 index 00000000..7e8188dd --- /dev/null +++ b/morphlib/plugins/distbuild_plugin.py @@ -0,0 +1,324 @@ +# distbuild_plugin.py -- Morph distributed build plugin +# +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.. + + +import cliapp +import logging +import re +import sys + +import morphlib +import distbuild + + +group_distbuild = 'Distributed Build Options' + +class DistbuildOptionsPlugin(cliapp.Plugin): + + def enable(self): + self.app.settings.string_list( + ['crash-condition'], + 'add FILENAME:FUNCNAME:MAXCALLS to list of crash conditions ' + '(this is for testing only)', + metavar='FILENAME:FUNCNAME:MAXCALLS', + group=group_distbuild) + + def disable(self): + pass + + +class SerialiseArtifactPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand('serialise-artifact', self.serialise_artifact, + arg_synopsis='REPO REF MORPHOLOGY') + + def disable(self): + pass + + def serialise_artifact(self, args): + '''Internal use only: Serialise Artifact build graph as JSON.''' + + distbuild.add_crash_conditions(self.app.settings['crash-condition']) + + if len(args) != 3: + raise cliapp.AppException('Must get triplet') + + repo_name, ref, morph_name = args + filename = morphlib.util.sanitise_morphology_path(morph_name) + build_command = morphlib.buildcommand.BuildCommand(self.app) + srcpool = build_command.create_source_pool(repo_name, ref, filename) + artifact = build_command.resolve_artifacts(srcpool) + self.app.output.write(distbuild.serialise_artifact(artifact)) + self.app.output.write('\n') + + +class WorkerBuild(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand( + 'worker-build', self.worker_build, arg_synopsis='') + + def disable(self): + pass + + def worker_build(self, args): + '''Internal use only: Build an artifact in a worker. + + All build dependencies are assumed to have been built already + and available in the local or remote artifact cache. + + ''' + + distbuild.add_crash_conditions(self.app.settings['crash-condition']) + + serialized = sys.stdin.readline() + artifact = distbuild.deserialise_artifact(serialized) + + bc = morphlib.buildcommand.BuildCommand(self.app) + + # Now, before we start the build, we garbage collect the caches + # to ensure we have room. First we remove all system artifacts + # since we never need to recover those from workers post-hoc + for cachekey, artifacts, last_used in bc.lac.list_contents(): + if any(self.is_system_artifact(f) for f in artifacts): + logging.debug("Removing all artifacts for system %s" % + cachekey) + bc.lac.remove(cachekey) + + self.app.subcommands['gc']([]) + + arch = artifact.arch + bc.build_artifact(artifact, bc.new_build_env(arch)) + + def is_system_artifact(self, filename): + return re.match(r'^[0-9a-fA-F]{64}\.system\.', filename) + +class WorkerDaemon(cliapp.Plugin): + + def enable(self): + self.app.settings.string( + ['worker-daemon-address'], + 'listen for connections on ADDRESS (domain / IP address)', + default='', + group=group_distbuild) + self.app.settings.integer( + ['worker-daemon-port'], + 'listen for connections on PORT', + default=3434, + group=group_distbuild) + self.app.settings.string( + ['worker-daemon-port-file'], + 'write port used by worker-daemon to FILE', + default='', + group=group_distbuild) + self.app.add_subcommand( + 'worker-daemon', + self.worker_daemon, + arg_synopsis='') + + def disable(self): + pass + + def worker_daemon(self, args): + '''Daemon that controls builds on a single worker node.''' + + distbuild.add_crash_conditions(self.app.settings['crash-condition']) + + address = self.app.settings['worker-daemon-address'] + port = self.app.settings['worker-daemon-port'] + port_file = self.app.settings['worker-daemon-port-file'] + router = distbuild.ListenServer(address, port, distbuild.JsonRouter, + port_file=port_file) + loop = distbuild.MainLoop() + loop.add_state_machine(router) + loop.run() + + +class ControllerDaemon(cliapp.Plugin): + + def enable(self): + self.app.settings.string( + ['controller-initiator-address'], + 'listen for initiator connections on ADDRESS ' + '(domain / IP address)', + default='', + group=group_distbuild) + self.app.settings.integer( + ['controller-initiator-port'], + 'listen for initiator connections on PORT', + default=7878, + group=group_distbuild) + self.app.settings.string( + ['controller-initiator-port-file'], + 'write the port to listen for initiator connections to FILE', + default='', + group=group_distbuild) + + self.app.settings.string( + ['controller-helper-address'], + 'listen for helper connections on ADDRESS (domain / IP address)', + default='localhost', + group=group_distbuild) + self.app.settings.integer( + ['controller-helper-port'], + 'listen for helper connections on PORT', + default=5656, + group=group_distbuild) + self.app.settings.string( + ['controller-helper-port-file'], + 'write the port to listen for helper connections to FILE', + default='', + group=group_distbuild) + + self.app.settings.string_list( + ['worker'], + 'specify a build worker (WORKER is ADDRESS or ADDRESS:PORT, ' + 'with PORT defaulting to 3434)', + metavar='WORKER', + default=[], + group=group_distbuild) + self.app.settings.integer( + ['worker-cache-server-port'], + 'port number for the artifact cache server on each worker', + metavar='PORT', + default=8080, + group=group_distbuild) + self.app.settings.string( + ['writeable-cache-server'], + 'specify the shared cache server writeable instance ' + '(SERVER is ADDRESS or ADDRESS:PORT, with PORT defaulting ' + 'to 80', + metavar='SERVER', + group=group_distbuild) + + self.app.settings.string( + ['morph-instance'], + 'use FILENAME to invoke morph (default: %default)', + metavar='FILENAME', + default='morph', + group=group_distbuild) + + self.app.add_subcommand( + 'controller-daemon', self.controller_daemon, arg_synopsis='') + + def disable(self): + pass + + def controller_daemon(self, args): + '''Daemon that gives jobs to worker daemons.''' + + distbuild.add_crash_conditions(self.app.settings['crash-condition']) + + artifact_cache_server = ( + self.app.settings['artifact-cache-server'] or + self.app.settings['cache-server']) + writeable_cache_server = self.app.settings['writeable-cache-server'] + worker_cache_server_port = \ + self.app.settings['worker-cache-server-port'] + morph_instance = self.app.settings['morph-instance'] + + listener_specs = [ + # address, port, class to initiate on connection, class init args + ('controller-helper-address', 'controller-helper-port', + 'controller-helper-port-file', + distbuild.HelperRouter, []), + ('controller-initiator-address', 'controller-initiator-port', + 'controller-initiator-port-file', + distbuild.InitiatorConnection, + [artifact_cache_server, morph_instance]), + ] + + loop = distbuild.MainLoop() + + queuer = distbuild.WorkerBuildQueuer() + loop.add_state_machine(queuer) + + for addr, port, port_file, sm, extra_args in listener_specs: + addr = self.app.settings[addr] + port = self.app.settings[port] + port_file = self.app.settings[port_file] + listener = distbuild.ListenServer( + addr, port, sm, extra_args=extra_args) + loop.add_state_machine(listener) + + for worker in self.app.settings['worker']: + if ':' in worker: + addr, port = worker.split(':', 1) + port = int(port) + else: + addr = worker + port = 3434 + cm = distbuild.ConnectionMachine( + addr, port, distbuild.WorkerConnection, + [writeable_cache_server, worker_cache_server_port, + morph_instance]) + loop.add_state_machine(cm) + + loop.run() + +class GraphStateMachines(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand( + 'graph-state-machines', + self.graph_state_machines, + arg_synopsis='') + + def disable(self): + pass + + def graph_state_machines(self, args): + cm = distbuild.ConnectionMachine(None, None, None, None) + cm._start_connect = lambda *args: None + self.graph_one(cm) + + self.graph_one(distbuild.BuildController(None, None, None)) + self.graph_one(distbuild.HelperRouter(None)) + self.graph_one(distbuild.InitiatorConnection(None, None, None)) + self.graph_one(distbuild.JsonMachine(None)) + self.graph_one(distbuild.WorkerBuildQueuer()) + + # FIXME: These need more mocking to work. + # self.graph_one(distbuild.Initiator(None, None, + # self, None, None, None)) + # self.graph_one(distbuild.JsonRouter(None)) + # self.graph_one(distbuild.SocketBuffer(None, None)) + # self.graph_one(distbuild.ListenServer(None, None, None)) + + def graph_one(self, sm): + class_name = sm.__class__.__name__.split('.')[-1] + filename = '%s.gv' % class_name + sm.mainloop = self + sm.setup() + sm.dump_dot(filename) + + # Some methods to mock this class as other classes, which the + # state machine class need to access, just enough to allow the + # transitions to be set up for graphing. + + def queue_event(self, *args, **kwargs): + pass + + def add_event_source(self, *args, **kwargs): + pass + + def add_state_machine(self, sm): + pass + + def status(self, *args, **kwargs): + pass diff --git a/morphlib/plugins/expand_repo_plugin.py b/morphlib/plugins/expand_repo_plugin.py new file mode 100644 index 00000000..721287ca --- /dev/null +++ b/morphlib/plugins/expand_repo_plugin.py @@ -0,0 +1,60 @@ +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp + +import morphlib + + +class ExpandRepoPlugin(cliapp.Plugin): + + '''Expand an aliased repo URL to be unaliases.''' + + def enable(self): + self.app.add_subcommand( + 'expand-repo', self.expand_repo, arg_synopsis='[REPOURL...]') + + def disable(self): + pass + + def expand_repo(self, args): + '''Expand repo aliases in URLs. + + Command line arguments: + + * `REPOURL` is a URL that may or may not be using repository + aliases. + + See the `--repo-alias` option for more about repository aliases. + + Example: + + $ morph expand-repo baserock:baserock/morphs + Original: baserock:baserock/morphs + pull: git://trove.baserock.org/baserock/baserock/morphs + push: ssh://git@git.baserock.org/baserock/baserock/morphs + + ''' + + aliases = self.app.settings['repo-alias'] + resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases) + for repourl in args: + self.app.output.write( + 'Original: %s\npull: %s\npush: %s\n\n' % + (repourl, + resolver.pull_url(repourl), + resolver.push_url(repourl))) + diff --git a/morphlib/plugins/gc_plugin.py b/morphlib/plugins/gc_plugin.py new file mode 100644 index 00000000..68f386eb --- /dev/null +++ b/morphlib/plugins/gc_plugin.py @@ -0,0 +1,172 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import logging +import os +import shutil +import time + +import fs.osfs +import cliapp + +import morphlib + + +class GCPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand('gc', self.gc, + arg_synopsis='') + self.app.settings.integer(['cachedir-artifact-delete-older-than'], + 'always delete artifacts older than this ' + 'period in seconds, (default: 1 week)', + metavar='PERIOD', + group="Storage Options", + default=(60*60*24*7)) + self.app.settings.integer(['cachedir-artifact-keep-younger-than'], + 'allow deletion of artifacts older than ' + 'this period in seconds, (default: 1 day)', + metavar='PERIOD', + group="Storage Options", + default=(60*60*24)) + + def disable(self): + pass + + def gc(self, args): + '''Make space by removing unused files. + + This command removes all artifacts older than + --cachedir-artifact-delete-older-than if the file system + that holds the cache directory has less than --cachedir-min-space + bytes free. + + It may delete artifacts older than + --cachedir-artifact-keep-younger-than if it still needs to make + space. + + It also removes any left over temporary chunks and staging areas + from failed builds. + + In addition we remove failed deployments, generally these are + cleared up by morph during deployment but in some cases they + won't be e.g. if morph gets a SIGKILL or the machine running + morph loses power. + + ''' + + tempdir = self.app.settings['tempdir'] + cachedir = self.app.settings['cachedir'] + tempdir_min_space, cachedir_min_space = \ + morphlib.util.unify_space_requirements( + tempdir, self.app.settings['tempdir-min-space'], + cachedir, self.app.settings['cachedir-min-space']) + + self.cleanup_tempdir(tempdir, tempdir_min_space) + self.cleanup_cachedir(cachedir, cachedir_min_space) + + def cleanup_tempdir(self, temp_path, min_space): + # The subdirectories in tempdir are created at Morph startup time. Code + # assumes that they exist in various places. + self.app.status(msg='Cleaning up temp dir %(temp_path)s', + temp_path=temp_path, chatty=True) + for subdir in ('deployments', 'failed', 'chunks'): + if morphlib.util.get_bytes_free_in_path(temp_path) >= min_space: + self.app.status(msg='Not Removing subdirectory ' + '%(subdir)s, enough space already cleared', + subdir=os.path.join(temp_path, subdir), + chatty=True) + break + self.app.status(msg='Removing temp subdirectory: %(subdir)s', + subdir=subdir) + path = os.path.join(temp_path, subdir) + if os.path.exists(path): + shutil.rmtree(path) + os.mkdir(path) + + def calculate_delete_range(self): + now = time.time() + always_delete_age = \ + now - self.app.settings['cachedir-artifact-delete-older-than'] + may_delete_age = \ + now - self.app.settings['cachedir-artifact-keep-younger-than'] + return always_delete_age, may_delete_age + + def find_deletable_artifacts(self, lac, max_age, min_age): + '''Get a list of cache keys in order of how old they are.''' + contents = list(lac.list_contents()) + always = set(cachekey + for cachekey, artifacts, mtime in contents + if mtime < max_age) + maybe = ((cachekey, mtime) + for cachekey, artifacts, mtime in contents + if max_age <= mtime < min_age) + return always, [cachekey for cachekey, mtime + in sorted(maybe, key=lambda x: x[1])] + + def cleanup_cachedir(self, cache_path, min_space): + def sufficient_free(): + free = morphlib.util.get_bytes_free_in_path(cache_path) + return (free >= min_space) + if sufficient_free(): + self.app.status(msg='Not cleaning up cachedir, ' + 'sufficient space already cleared', + chatty=True) + return + lac = morphlib.localartifactcache.LocalArtifactCache( + fs.osfs.OSFS(os.path.join(cache_path, 'artifacts'))) + max_age, min_age = self.calculate_delete_range() + logging.debug('Must remove artifacts older than timestamp %d' + % max_age) + always_delete, may_delete = \ + self.find_deletable_artifacts(lac, max_age, min_age) + removed = 0 + source_count = len(always_delete) + len(may_delete) + logging.debug('Must remove artifacts %s' % repr(always_delete)) + logging.debug('Can remove artifacts %s' % repr(may_delete)) + + # Remove all old artifacts + for cachekey in always_delete: + self.app.status(msg='Removing source %(cachekey)s', + cachekey=cachekey, chatty=True) + lac.remove(cachekey) + removed += 1 + + # Maybe remove remaining middle-aged artifacts + for cachekey in may_delete: + if sufficient_free(): + self.app.status(msg='Finished cleaning up cachedir with ' + '%(remaining)d old sources remaining', + remaining=(source_count - removed), + chatty=True) + break + self.app.status(msg='Removing source %(cachekey)s', + cachekey=cachekey, chatty=True) + lac.remove(cachekey) + removed += 1 + + if sufficient_free(): + self.app.status(msg='Made sufficient space in %(cache_path)s ' + 'after removing %(removed)d sources', + removed=removed, cache_path=cache_path) + return + self.app.status(msg='Unable to clear enough space in %(cache_path)s ' + 'after removing %(removed)d sources. Please ' + 'reduce cachedir-artifact-keep-younger-than, ' + 'clear space from elsewhere, enlarge the disk ' + 'or reduce cachedir-min-space.', + cache_path=cache_path, removed=removed, + error=True) diff --git a/morphlib/plugins/graphing_plugin.py b/morphlib/plugins/graphing_plugin.py new file mode 100644 index 00000000..57166e51 --- /dev/null +++ b/morphlib/plugins/graphing_plugin.py @@ -0,0 +1,98 @@ +# Copyright (C) 2012, 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import os + +import morphlib + + +class GraphingPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand('graph-build-depends', + self.graph_build_depends, + arg_synopsis='REPO REF MORPHOLOGY') + + def disable(self): + pass + + def graph_build_depends(self, args): + '''Create a visualisation of build dependencies in a system. + + Command line arguments: + + * `REPO` is a repository URL. + * `REF` is a git reference (usually branch name). + * `MORPHOLOGY` is a system morphology. + + This produces a GraphViz DOT file representing all the build + dependencies within a system, based on information in the + morphologies. The GraphViz `dot` program can then be used to + create a graphical representation of the dependencies. This + can be helpful for inspecting whether there are any problems in + the dependencies. + + Example: + + morph graph-build-depends baserock:baserock/morphs master \ + devel-system-x86_64-generic > foo.dot + dot -Tpng foo.dot > foo.png + + The above would create a picture with the build dependencies of + everything in the development system for 64-bit Intel systems. + + GraphViz is not, currently, part of Baserock, so you need to run + `dot` on another system. + + ''' + + for repo_name, ref, filename in self.app.itertriplets(args): + self.app.status(msg='Creating build order for ' + '%(repo_name)s %(ref)s %(filename)s', + repo_name=repo_name, ref=ref, filename=filename) + builder = morphlib.buildcommand.BuildCommand(self.app) + srcpool = builder.create_source_pool(repo_name, ref, filename) + root_artifact = builder.resolve_artifacts(srcpool) + + basename, ext = os.path.splitext(filename) + dot_filename = basename + '.gv' + dep_fmt = ' "%s" -> "%s";\n' + shape_name = { + 'system': 'octagon', + 'stratum': 'box', + 'chunk': 'ellipse', + } + + self.app.status(msg='Writing DOT file to %(filename)s', + filename=dot_filename) + + with open(dot_filename, 'w') as f: + f.write('digraph "%s" {\n' % basename) + for a in root_artifact.walk(): + f.write( + ' "%s" [shape=%s];\n' % + (a.name, + shape_name[a.source.morphology['kind']])) + for dep in a.dependencies: + if a.source.morphology['kind'] == 'stratum': + if dep.dependents == [a]: + f.write(dep_fmt % + (a.name, dep.name)) + else: + f.write(dep_fmt % (a.name, dep.name)) + f.write('}\n') + diff --git a/morphlib/plugins/list_artifacts_plugin.py b/morphlib/plugins/list_artifacts_plugin.py new file mode 100644 index 00000000..8074206b --- /dev/null +++ b/morphlib/plugins/list_artifacts_plugin.py @@ -0,0 +1,125 @@ +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# This plugin is used as part of the Baserock automated release process. +# +# See: <http://wiki.baserock.org/guides/release-process> for more information. + + +import cliapp +import morphlib + + +class ListArtifactsPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand( + 'list-artifacts', self.list_artifacts, + arg_synopsis='REPO REF MORPH [MORPH]...') + + def disable(self): + pass + + def list_artifacts(self, args): + '''List every artifact in the build graph of a system. + + Command line arguments: + + * `REPO` is a git repository URL. + * `REF` is a branch or other commit reference in that repository. + * `MORPH` is a system morphology name at that ref. + + You can pass multiple values for `MORPH`, in which case the command + outputs the union of the build graphs of all the systems passed in. + + The output includes any meta-artifacts such as .meta and .build-log + files. + + ''' + + if len(args) < 3: + raise cliapp.AppException( + 'Wrong number of arguments to list-artifacts command ' + '(see help)') + + repo, ref = args[0], args[1] + system_filenames = map(morphlib.util.sanitise_morphology_path, + args[2:]) + + self.lrc, self.rrc = morphlib.util.new_repo_caches(self.app) + self.resolver = morphlib.artifactresolver.ArtifactResolver() + + artifact_files = set() + for system_filename in system_filenames: + system_artifact_files = self.list_artifacts_for_system( + repo, ref, system_filename) + artifact_files.update(system_artifact_files) + + for artifact_file in sorted(artifact_files): + print artifact_file + + def list_artifacts_for_system(self, repo, ref, system_filename): + '''List all artifact files in the build graph of a single system.''' + + # Sadly, we must use a fresh source pool and a fresh list of artifacts + # for each system. Creating a source pool is slow (queries every Git + # repo involved in the build) and resolving artifacts isn't so quick + # either. Unfortunately, each Source object can only have one set of + # Artifact objects associated, which means the source pool cannot mix + # sources that are being built for multiple architectures: the build + # graph representation does not distinguish chunks or strata of + # different architectures right now. + + self.app.status( + msg='Creating source pool for %s' % system_filename, chatty=True) + source_pool = self.app.create_source_pool( + self.lrc, self.rrc, repo, ref, system_filename) + + self.app.status( + msg='Resolving artifacts for %s' % system_filename, chatty=True) + artifacts = self.resolver.resolve_artifacts(source_pool) + + def find_artifact_by_name(artifacts_list, filename): + for a in artifacts_list: + if a.source.filename == filename: + return a + raise ValueError + + system_artifact = find_artifact_by_name(artifacts, system_filename) + + self.app.status( + msg='Computing cache keys for %s' % system_filename, chatty=True) + build_env = morphlib.buildenvironment.BuildEnvironment( + self.app.settings, system_artifact.source.morphology['arch']) + ckc = morphlib.cachekeycomputer.CacheKeyComputer(build_env) + + artifact_files = set() + for artifact in system_artifact.walk(): + artifact.cache_key = ckc.compute_key(artifact) + artifact.cache_id = ckc.get_cache_id(artifact) + + artifact_files.add(artifact.basename()) + + if artifact.source.morphology.needs_artifact_metadata_cached: + artifact_files.add('%s.meta' % artifact.basename()) + + # This is unfortunate hardwiring of behaviour; in future we + # should list all artifacts in the meta-artifact file, so we + # don't have to guess what files there will be. + artifact_files.add('%s.meta' % artifact.cache_key) + if artifact.source.morphology['kind'] == 'chunk': + artifact_files.add('%s.build-log' % artifact.cache_key) + + return artifact_files diff --git a/morphlib/plugins/print_architecture_plugin.py b/morphlib/plugins/print_architecture_plugin.py new file mode 100644 index 00000000..08f500d0 --- /dev/null +++ b/morphlib/plugins/print_architecture_plugin.py @@ -0,0 +1,35 @@ +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import os + +import morphlib + + +class PrintArchitecturePlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand( + 'print-architecture', self.print_architecture, arg_synopsis='') + + def disable(self): + pass + + def print_architecture(self, args): + '''Print the name of the architecture of the host.''' + + self.app.output.write('%s\n' % morphlib.util.get_host_architecture()) diff --git a/morphlib/plugins/push_pull_plugin.py b/morphlib/plugins/push_pull_plugin.py new file mode 100644 index 00000000..843de1a6 --- /dev/null +++ b/morphlib/plugins/push_pull_plugin.py @@ -0,0 +1,93 @@ +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import cliapp +import logging +import os + +import morphlib + + +class PushPullPlugin(cliapp.Plugin): + + '''Add subcommands to wrap the git push and pull commands.''' + + def enable(self): + self.app.add_subcommand( + 'push', self.push, arg_synopsis='REPO TARGET') + self.app.add_subcommand('pull', self.pull, arg_synopsis='[REMOTE]') + + def disable(self): + pass + + def push(self, args): + '''Push a branch to a remote repository. + + Command line arguments: + + * `REPO` is the repository to push your changes to. + + * `TARGET` is the branch to push to the repository. + + This is a wrapper for the `git push` command. It also deals with + pushing any binary files that have been added using git-fat. + + Example: + + morph push origin jrandom/new-feature + + ''' + if len(args) != 2: + raise morphlib.Error('push must get exactly two arguments') + + gd = morphlib.gitdir.GitDirectory(os.getcwd()) + remote, branch = args + rs = morphlib.gitdir.RefSpec(branch) + gd.get_remote(remote).push(rs) + if gd.has_fat(): + gd.fat_init() + gd.fat_push() + + def pull(self, args): + '''Pull changes to the current branch from a repository. + + Command line arguments: + + * `REMOTE` is the remote branch to pull from. By default this is the + branch being tracked by your current git branch (ie origin/master + for branch master) + + This is a wrapper for the `git pull` command. It also deals with + pulling any binary files that have been added to the repository using + git-fat. + + Example: + + morph pull + + ''' + if len(args) > 1: + raise morphlib.Error('pull takes at most one argument') + + gd = morphlib.gitdir.GitDirectory(os.getcwd()) + remote = gd.get_remote('origin') + if args: + branch = args[0] + remote.pull(branch) + else: + remote.pull() + if gd.has_fat(): + gd.fat_init() + gd.fat_pull() diff --git a/morphlib/plugins/show_dependencies_plugin.py b/morphlib/plugins/show_dependencies_plugin.py new file mode 100644 index 00000000..e70f6bfb --- /dev/null +++ b/morphlib/plugins/show_dependencies_plugin.py @@ -0,0 +1,79 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import os + +import morphlib + + +class ShowDependenciesPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand('show-dependencies', + self.show_dependencies, + arg_synopsis='(REPO REF MORPHOLOGY)...') + + def disable(self): + pass + + def show_dependencies(self, args): + '''Dumps the dependency tree of all input morphologies. + + Command line arguments: + + * `REPO` is a git repository URL. + * `REF` is a branch or other commit reference in that repository. + * `FILENAME` is a morphology filename at that ref. + + This command analyses a system morphology and produces a listing + of build dependencies of the constituent components. + + ''' + + if not os.path.exists(self.app.settings['cachedir']): + os.mkdir(self.app.settings['cachedir']) + cachedir = os.path.join(self.app.settings['cachedir'], 'gits') + tarball_base_url = self.app.settings['tarball-server'] + repo_resolver = morphlib.repoaliasresolver.RepoAliasResolver( + self.app.settings['repo-alias']) + lrc = morphlib.localrepocache.LocalRepoCache( + self.app, cachedir, repo_resolver, tarball_base_url) + + remote_url = morphlib.util.get_git_resolve_cache_server( + self.app.settings) + if remote_url: + rrc = morphlib.remoterepocache.RemoteRepoCache( + remote_url, repo_resolver) + else: + rrc = None + + build_command = morphlib.buildcommand.BuildCommand(self.app) + + # traverse the morphs to list all the sources + for repo, ref, filename in self.app.itertriplets(args): + morph = morphlib.util.sanitise_morphology_path(filename) + self.app.output.write('dependency graph for %s|%s|%s:\n' % + (repo, ref, morph)) + + srcpool = build_command.create_source_pool(repo, ref, filename) + root_artifact = build_command.resolve_artifacts(srcpool) + + for artifact in reversed(root_artifact.walk()): + self.app.output.write(' %s\n' % artifact) + for dep in sorted(artifact.source.dependencies, key=str): + self.app.output.write(' -> %s\n' % dep) + diff --git a/morphlib/plugins/trovectl_plugin.py b/morphlib/plugins/trovectl_plugin.py new file mode 100644 index 00000000..80f3b4cf --- /dev/null +++ b/morphlib/plugins/trovectl_plugin.py @@ -0,0 +1,53 @@ +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import cliapp + +import morphlib + +class TrovectlPlugin(cliapp.Plugin): + + def enable(self): + self.app.add_subcommand( + 'trovectl', self.trovectl, arg_synopsis='GITANO-COMMAND [ARG...]') + + def disable(self): + pass + + def trovectl(self, args, **kwargs): + '''Invoke Gitano commands on the Trove host. + + Command line arguments: + + * `GITANO-COMMAND` is the Gitano command to invoke on the Trove. + * `ARG` is a Gitano command argument. + + This invokes Gitano commands on the Trove host configured + in the Morph configuration (see `--trove-host`). + + Trove is the Codethink code hosting appliance. Gitano is the + git server management component of that. + + Example: + + morph trovectl whoami + morph trovectl help + + ''' + + trove = 'git@' + self.app.settings['trove-host'] + self.app.runcmd(['ssh', trove] + args, + stdin=None, stdout=None, stderr=None) + diff --git a/morphlib/recv-hole b/morphlib/recv-hole new file mode 100755 index 00000000..d6504bf6 --- /dev/null +++ b/morphlib/recv-hole @@ -0,0 +1,159 @@ +#!/bin/sh +# +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +# Receive a data stream describing a sparse file, and reproduce it, +# either to a named file or stdout. +# +# The data stream is simple: it's a sequence of DATA or HOLE records: +# +# DATA +# 123 +# <123 bytes of binary data, NOT including newline at the end> +# +# HOLE +# 123 +# +# This shell script can be executed over ssh (given to ssh as an arguemnt, +# with suitable escaping) on a different computer. This allows a large +# sparse file (e.g., disk image) be transferred quickly. +# +# This script should be called in one of the following ways: +# +# recv-hole file FILENAME +# recv-hole vbox FILENAME DISKSIZE +# +# In both cases, FILENAME is the pathname of the disk image on the +# receiving end. DISKSIZE is the size of the disk image in bytes. The +# first form is used when transferring a disk image to become an +# identical file on the receiving end. +# +# The second form is used when the disk image should be converted for +# use by VirtualBox. In this case, we want to avoid writing a +# temporary file on disk, and then calling the VirtualBox VBoxManage +# tool to do the conversion, since that would involve large amounts of +# unnecessary I/O and disk usage. Instead we pipe the file directly to +# VBoxManage, avoiding those issues. The piping is done here in this +# script, instead of in the caller, to make it easier to run things +# over ssh. +# +# However, since it's not possible seek in a Unix pipe, we have to +# explicitly write the zeroes into the pipe. This is not +# super-efficient, but the way to avoid that would be to avoid sending +# a sparse file, and do the conversion to a VDI on the sending end. +# That is out of scope for xfer-hole and recv-hole. + + +set -eu + + +die() +{ + echo "$@" 1>&2 + exit 1 +} + + +recv_hole_to_file() +{ + local n + + read n + truncate --size "+$n" "$1" +} + + +recv_data_to_file() +{ + local n + read n + + local blocksize=1048576 + local blocks=$(($n / $blocksize)) + local extra=$(($n % $blocksize)) + + xfer_data_to_stdout "$blocksize" "$blocks" >> "$1" + xfer_data_to_stdout 1 "$extra" >> "$1" +} + + +recv_hole_to_stdout() +{ + local n + read n + (echo "$n"; cat /dev/zero) | recv_data_to_stdout +} + + +recv_data_to_stdout() +{ + local n + read n + + local blocksize=1048576 + local blocks=$(($n / $blocksize)) + local extra=$(($n % $blocksize)) + + xfer_data_to_stdout "$blocksize" "$blocks" + xfer_data_to_stdout 1 "$extra" +} + + +xfer_data_to_stdout() +{ + local log="$(mktemp)" + if ! dd "bs=$1" count="$2" iflag=fullblock status=noxfer 2> "$log" + then + cat "$log" 1>&2 + rm -f "$log" + exit 1 + else + rm -f "$log" + fi +} + + +type="$1" +case "$type" in + file) + output="$2" + truncate --size=0 "$output" + while read what + do + case "$what" in + DATA) recv_data_to_file "$output" ;; + HOLE) recv_hole_to_file "$output" ;; + *) die "Unknown instruction: $what" ;; + esac + done + ;; + vbox) + output="$2" + disk_size="$3" + while read what + do + case "$what" in + DATA) recv_data_to_stdout ;; + HOLE) recv_hole_to_stdout ;; + *) die "Unknown instruction: $what" ;; + esac + done | + VBoxManage convertfromraw stdin "$output" "$disk_size" + ;; +esac diff --git a/morphlib/remoteartifactcache.py b/morphlib/remoteartifactcache.py new file mode 100644 index 00000000..4e09ce34 --- /dev/null +++ b/morphlib/remoteartifactcache.py @@ -0,0 +1,117 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import logging +import urllib +import urllib2 +import urlparse + + +class HeadRequest(urllib2.Request): # pragma: no cover + + def get_method(self): + return 'HEAD' + + +class GetError(cliapp.AppException): + + def __init__(self, cache, artifact): + cliapp.AppException.__init__( + self, 'Failed to get the artifact %s ' + 'from the artifact cache %s' % + (artifact.basename(), cache)) + + +class GetArtifactMetadataError(GetError): + + def __init__(self, cache, artifact, name): + cliapp.AppException.__init__( + self, 'Failed to get metadata %s for the artifact %s ' + 'from the artifact cache %s' % + (name, artifact.basename(), cache)) + + +class GetSourceMetadataError(GetError): + + def __init__(self, cache, source, cache_key, name): + cliapp.AppException.__init__( + self, 'Failed to get metadata %s for source %s ' + 'and cache key %s from the artifact cache %s' % + (name, source, cache_key, cache)) + + +class RemoteArtifactCache(object): + + def __init__(self, server_url): + self.server_url = server_url + + def has(self, artifact): + return self._has_file(artifact.basename()) + + def has_artifact_metadata(self, artifact, name): + return self._has_file(artifact.metadata_basename(name)) + + def has_source_metadata(self, source, cachekey, name): + filename = '%s.%s' % (cachekey, name) + return self._has_file(filename) + + def get(self, artifact, log=logging.error): + try: + return self._get_file(artifact.basename()) + except urllib2.URLError, e: + log(str(e)) + raise GetError(self, artifact) + + def get_artifact_metadata(self, artifact, name, log=logging.error): + try: + return self._get_file(artifact.metadata_basename(name)) + except urllib2.URLError, e: + log(str(e)) + raise GetArtifactMetadataError(self, artifact, name) + + def get_source_metadata(self, source, cachekey, name): + filename = '%s.%s' % (cachekey, name) + try: + return self._get_file(filename) + except urllib2.URLError: + raise GetSourceMetadataError(self, source, cachekey, name) + + def _has_file(self, filename): # pragma: no cover + url = self._request_url(filename) + logging.debug('RemoteArtifactCache._has_file: url=%s' % url) + request = HeadRequest(url) + try: + urllib2.urlopen(request) + return True + except (urllib2.HTTPError, urllib2.URLError): + return False + + def _get_file(self, filename): # pragma: no cover + url = self._request_url(filename) + logging.debug('RemoteArtifactCache._get_file: url=%s' % url) + return urllib2.urlopen(url) + + def _request_url(self, filename): # pragma: no cover + server_url = self.server_url + if not server_url.endswith('/'): + server_url += '/' + return urlparse.urljoin( + server_url, '/1.0/artifacts?filename=%s' % + urllib.quote(filename)) + + def __str__(self): # pragma: no cover + return self.server_url diff --git a/morphlib/remoteartifactcache_tests.py b/morphlib/remoteartifactcache_tests.py new file mode 100644 index 00000000..788882c2 --- /dev/null +++ b/morphlib/remoteartifactcache_tests.py @@ -0,0 +1,164 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import StringIO +import unittest +import urllib2 + +import morphlib + + +class RemoteArtifactCacheTests(unittest.TestCase): + + def setUp(self): + loader = morphlib.morphloader.MorphologyLoader() + morph = loader.load_from_string( + ''' + name: chunk + kind: chunk + products: + - artifact: chunk-runtime + include: + - usr/bin + - usr/sbin + - usr/lib + - usr/libexec + - artifact: chunk-devel + include: + - usr/include + - artifact: chunk-doc + include: + - usr/share/doc + ''') + sources = morphlib.source.make_sources('repo', 'original/ref', + 'chunk.morph', 'sha1', + 'tree', morph) + self.source, = sources + self.runtime_artifact = morphlib.artifact.Artifact( + self.source, 'chunk-runtime') + self.runtime_artifact.cache_key = 'CHUNK' + self.devel_artifact = morphlib.artifact.Artifact( + self.source, 'chunk-devel') + self.devel_artifact.cache_key = 'CHUNK' + self.doc_artifact = morphlib.artifact.Artifact( + self.source, 'chunk-doc') + self.doc_artifact.cache_key = 'CHUNK' + + self.existing_files = set([ + self.runtime_artifact.basename(), + self.devel_artifact.basename(), + self.runtime_artifact.metadata_basename('meta'), + '%s.%s' % (self.runtime_artifact.cache_key, 'meta'), + ]) + + self.server_url = 'http://foo.bar:8080' + self.cache = morphlib.remoteartifactcache.RemoteArtifactCache( + self.server_url) + self.cache._has_file = self._has_file + self.cache._get_file = self._get_file + + def _has_file(self, filename): + return filename in self.existing_files + + def _get_file(self, filename): + if filename in self.existing_files: + return StringIO.StringIO('%s' % filename) + else: + raise urllib2.URLError('foo') + + def test_sets_server_url(self): + self.assertEqual(self.cache.server_url, self.server_url) + + def test_has_existing_artifact(self): + self.assertTrue(self.cache.has(self.runtime_artifact)) + + def test_has_a_different_existing_artifact(self): + self.assertTrue(self.cache.has(self.devel_artifact)) + + def test_does_not_have_a_non_existent_artifact(self): + self.assertFalse(self.cache.has(self.doc_artifact)) + + def test_has_existing_artifact_metadata(self): + self.assertTrue(self.cache.has_artifact_metadata( + self.runtime_artifact, 'meta')) + + def test_does_not_have_non_existent_artifact_metadata(self): + self.assertFalse(self.cache.has_artifact_metadata( + self.runtime_artifact, 'non-existent-meta')) + + def test_has_existing_source_metadata(self): + self.assertTrue(self.cache.has_source_metadata( + self.runtime_artifact.source, + self.runtime_artifact.cache_key, + 'meta')) + + def test_does_not_have_non_existent_source_metadata(self): + self.assertFalse(self.cache.has_source_metadata( + self.runtime_artifact.source, + self.runtime_artifact.cache_key, + 'non-existent-meta')) + + def test_get_existing_artifact(self): + handle = self.cache.get(self.runtime_artifact) + data = handle.read() + self.assertEqual(data, self.runtime_artifact.basename()) + + def test_get_a_different_existing_artifact(self): + handle = self.cache.get(self.devel_artifact) + data = handle.read() + self.assertEqual(data, self.devel_artifact.basename()) + + def test_fails_to_get_a_non_existent_artifact(self): + self.assertRaises(morphlib.remoteartifactcache.GetError, + self.cache.get, self.doc_artifact, + log=lambda *args: None) + + def test_get_existing_artifact_metadata(self): + handle = self.cache.get_artifact_metadata( + self.runtime_artifact, 'meta') + data = handle.read() + self.assertEqual( + data, '%s.%s' % (self.runtime_artifact.basename(), 'meta')) + + def test_fails_to_get_non_existent_artifact_metadata(self): + self.assertRaises( + morphlib.remoteartifactcache.GetArtifactMetadataError, + self.cache.get_artifact_metadata, + self.runtime_artifact, + 'non-existent-meta', + log=lambda *args: None) + + def test_get_existing_source_metadata(self): + handle = self.cache.get_source_metadata( + self.runtime_artifact.source, + self.runtime_artifact.cache_key, + 'meta') + data = handle.read() + self.assertEqual( + data, '%s.%s' % (self.runtime_artifact.cache_key, 'meta')) + + def test_fails_to_get_non_existent_source_metadata(self): + self.assertRaises( + morphlib.remoteartifactcache.GetSourceMetadataError, + self.cache.get_source_metadata, + self.runtime_artifact.source, + self.runtime_artifact.cache_key, + 'non-existent-meta') + + def test_escapes_pluses_in_request_urls(self): + returned_url = self.cache._request_url('gtk+') + correct_url = '%s/1.0/artifacts?filename=gtk%%2B' % self.server_url + self.assertEqual(returned_url, correct_url) diff --git a/morphlib/remoterepocache.py b/morphlib/remoterepocache.py new file mode 100644 index 00000000..004ba86e --- /dev/null +++ b/morphlib/remoterepocache.py @@ -0,0 +1,106 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import json +import logging +import urllib2 +import urlparse +import urllib + + +class ResolveRefError(cliapp.AppException): + + def __init__(self, repo_name, ref): + cliapp.AppException.__init__( + self, 'Failed to resolve ref %s for repo %s' % + (ref, repo_name)) + + +class CatFileError(cliapp.AppException): + + def __init__(self, repo_name, ref, filename): + cliapp.AppException.__init__( + self, 'Failed to cat file %s in ref %s of repo %s' % + (filename, ref, repo_name)) + +class LsTreeError(cliapp.AppException): + + def __init__(self, repo_name, ref): + cliapp.AppException.__init__( + self, 'Failed to list tree in ref %s of repo %s' % + (ref, repo_name)) + + +class RemoteRepoCache(object): + + def __init__(self, server_url, resolver): + self.server_url = server_url + self._resolver = resolver + + def resolve_ref(self, repo_name, ref): + repo_url = self._resolver.pull_url(repo_name) + try: + return self._resolve_ref_for_repo_url(repo_url, ref) + except BaseException, e: + logging.error('Caught exception: %s' % str(e)) + raise ResolveRefError(repo_name, ref) + + def cat_file(self, repo_name, ref, filename): + repo_url = self._resolver.pull_url(repo_name) + try: + return self._cat_file_for_repo_url(repo_url, ref, filename) + except urllib2.HTTPError as e: + logging.error('Caught exception: %s' % str(e)) + if e.code == 404: + raise CatFileError(repo_name, ref, filename) + raise # pragma: no cover + + def ls_tree(self, repo_name, ref): + repo_url = self._resolver.pull_url(repo_name) + try: + info = json.loads(self._ls_tree_for_repo_url(repo_url, ref)) + return info['tree'].keys() + except BaseException, e: + logging.error('Caught exception: %s' % str(e)) + raise LsTreeError(repo_name, ref) + + def _resolve_ref_for_repo_url(self, repo_url, ref): # pragma: no cover + data = self._make_request( + 'sha1s?repo=%s&ref=%s' % self._quote_strings(repo_url, ref)) + info = json.loads(data) + return info['sha1'], info['tree'] + + def _cat_file_for_repo_url(self, repo_url, ref, + filename): # pragma: no cover + return self._make_request( + 'files?repo=%s&ref=%s&filename=%s' + % self._quote_strings(repo_url, ref, filename)) + + def _ls_tree_for_repo_url(self, repo_url, ref): # pragma: no cover + return self._make_request( + 'trees?repo=%s&ref=%s' % self._quote_strings(repo_url, ref)) + + def _quote_strings(self, *args): # pragma: no cover + return tuple(urllib.quote(string) for string in args) + + def _make_request(self, path): # pragma: no cover + server_url = self.server_url + if not server_url.endswith('/'): + server_url += '/' + url = urlparse.urljoin(server_url, '/1.0/%s' % path) + handle = urllib2.urlopen(url) + return handle.read() diff --git a/morphlib/remoterepocache_tests.py b/morphlib/remoterepocache_tests.py new file mode 100644 index 00000000..ef81506f --- /dev/null +++ b/morphlib/remoterepocache_tests.py @@ -0,0 +1,138 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import json +import unittest +import urllib2 + +import morphlib + + +class RemoteRepoCacheTests(unittest.TestCase): + + def _resolve_ref_for_repo_url(self, repo_url, ref): + return self.sha1s[repo_url][ref] + + def _cat_file_for_repo_url(self, repo_url, sha1, filename): + try: + return self.files[repo_url][sha1][filename] + except KeyError: + raise urllib2.HTTPError(url='', code=404, msg='Not found', + hdrs={}, fp=None) + + def _ls_tree_for_repo_url(self, repo_url, sha1): + return json.dumps({ + 'repo': repo_url, + 'ref': sha1, + 'tree': self.files[repo_url][sha1] + }) + + def setUp(self): + self.sha1s = { + 'git://gitorious.org/baserock/morph': { + 'master': 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9' + } + } + self.files = { + 'git://gitorious.org/baserock-morphs/linux': { + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9': { + 'linux.morph': 'linux morphology' + } + } + } + self.server_url = 'http://foo.bar' + aliases = [ + 'upstream=git://gitorious.org/baserock-morphs/#foo', + 'baserock=git://gitorious.org/baserock/#foo' + ] + resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases) + self.cache = morphlib.remoterepocache.RemoteRepoCache( + self.server_url, resolver) + self.cache._resolve_ref_for_repo_url = self._resolve_ref_for_repo_url + self.cache._cat_file_for_repo_url = self._cat_file_for_repo_url + self.cache._ls_tree_for_repo_url = self._ls_tree_for_repo_url + + def test_sets_server_url(self): + self.assertEqual(self.cache.server_url, self.server_url) + + def test_resolve_existing_ref_for_existing_repo(self): + sha1 = self.cache.resolve_ref('baserock:morph', 'master') + self.assertEqual( + sha1, + self.sha1s['git://gitorious.org/baserock/morph']['master']) + + def test_fail_resolving_existing_ref_for_non_existent_repo(self): + self.assertRaises(morphlib.remoterepocache.ResolveRefError, + self.cache.resolve_ref, 'non-existent-repo', + 'master') + + def test_fail_resolving_non_existent_ref_for_existing_repo(self): + self.assertRaises(morphlib.remoterepocache.ResolveRefError, + self.cache.resolve_ref, 'baserock:morph', + 'non-existent-ref') + + def test_fail_resolving_non_existent_ref_for_non_existent_repo(self): + self.assertRaises(morphlib.remoterepocache.ResolveRefError, + self.cache.resolve_ref, 'non-existent-repo', + 'non-existent-ref') + + def test_cat_existing_file_in_existing_repo_and_ref(self): + content = self.cache.cat_file( + 'upstream:linux', 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + 'linux.morph') + self.assertEqual(content, 'linux morphology') + + def test_fail_cat_file_using_invalid_sha1(self): + self.assertRaises(morphlib.remoterepocache.CatFileError, + self.cache.cat_file, 'upstream:linux', 'blablabla', + 'linux.morph') + + def test_fail_cat_non_existent_file_in_existing_repo_and_ref(self): + self.assertRaises(morphlib.remoterepocache.CatFileError, + self.cache.cat_file, 'upstream:linux', + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + 'non-existent-file') + + def test_fail_cat_file_in_non_existent_ref_in_existing_repo(self): + self.assertRaises(morphlib.remoterepocache.CatFileError, + self.cache.cat_file, 'upstream:linux', + 'ecd7a325095a0d19b8c3d76f578d85b979461d41', + 'linux.morph') + + def test_fail_cat_file_in_non_existent_repo(self): + self.assertRaises(morphlib.remoterepocache.CatFileError, + self.cache.cat_file, 'non-existent-repo', + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9', + 'some-file') + + def test_ls_tree_in_existing_repo_and_ref(self): + content = self.cache.ls_tree( + 'upstream:linux', 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9') + self.assertEqual(content, ['linux.morph']) + + def test_fail_ls_tree_using_invalid_sha1(self): + self.assertRaises(morphlib.remoterepocache.LsTreeError, + self.cache.ls_tree, 'upstream:linux', 'blablabla') + + def test_fail_ls_file_in_non_existent_ref_in_existing_repo(self): + self.assertRaises(morphlib.remoterepocache.LsTreeError, + self.cache.ls_tree, 'upstream:linux', + 'ecd7a325095a0d19b8c3d76f578d85b979461d41') + + def test_fail_ls_tree_in_non_existent_repo(self): + self.assertRaises(morphlib.remoterepocache.LsTreeError, + self.cache.ls_tree, 'non-existent-repo', + 'e28a23812eadf2fce6583b8819b9c5dbd36b9fb9') diff --git a/morphlib/repoaliasresolver.py b/morphlib/repoaliasresolver.py new file mode 100644 index 00000000..bc759dd4 --- /dev/null +++ b/morphlib/repoaliasresolver.py @@ -0,0 +1,116 @@ +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import logging +import re + + +class RepoAlias(object): + + def __init__(self, alias, prefix, pullpat, pushpat): + self.alias = alias + self.prefix = prefix + self.pullpat = pullpat + self.pushpat = pushpat + + def _pattern_to_regex(self, pattern): + if '%s' in pattern: + return r'(?P<path>.+)'.join(map(re.escape, pattern.split('%s'))) + else: + return re.escape(pattern) + r'(?P<path>.+)' + + def match_url(self, url): + '''Given a URL, return what its alias would be if it matches''' + for pat in (self.pullpat, self.pushpat): + m = re.match(self._pattern_to_regex(pat), url) + if m: + return '%s:%s' % (self.prefix, m.group('path')) + return None + +class RepoAliasResolver(object): + + def __init__(self, aliases): + self.aliases = {} + + alias_pattern = (r'^(?P<prefix>[a-z][a-z0-9-]+)' + r'=(?P<pullpat>[^#]+)#(?P<pushpat>[^#]+)$') + for alias in aliases: + m = re.match(alias_pattern, alias) + if not m: + logging.warning('Alias %s is malformed' % alias) + continue + prefix = m.group('prefix') + self.aliases[prefix] = RepoAlias(alias, prefix, m.group('pullpat'), + m.group('pushpat')) + + + def pull_url(self, reponame): + '''Expand a possibly shortened repo name to a pull url.''' + return self._expand_reponame(reponame, 'pullpat') + + def push_url(self, reponame): + '''Expand a possibly shortened repo name to a push url.''' + return self._expand_reponame(reponame, 'pushpat') + + def aliases_from_url(self, url): + '''Find aliases the url could have expanded from. + + Returns an ascii-betically sorted list. + ''' + potential_matches = (repo_alias.match_url(url) + for repo_alias in self.aliases.itervalues()) + known_aliases = (url_alias for url_alias in potential_matches + if url_alias is not None) + return sorted(known_aliases) + + def _expand_reponame(self, reponame, patname): + prefix, suffix = self._split_reponame(reponame) + + # There was no prefix. + if prefix is None: + result = reponame + elif prefix not in self.aliases: + # Unknown prefix. Which means it may be a real URL instead. + # Let the caller deal with it. + result = reponame + else: + pat = getattr(self.aliases[prefix], patname) + result = self._apply_url_pattern(pat, suffix) + + logging.debug("Expansion of %s for %s yielded: %s" % + (reponame, patname, result)) + + return result + + def _split_reponame(self, reponame): + '''Split reponame into prefix and suffix. + + The prefix is returned as None if there was no prefix. + + ''' + + pat = r'^(?P<prefix>[a-z][a-z0-9-]+):(?P<rest>.*)$' + m = re.match(pat, reponame) + if m: + return m.group('prefix'), m.group('rest') + else: + return None, reponame + + def _apply_url_pattern(self, pattern, shortname): + if '%s' in pattern: + return shortname.join(pattern.split('%s')) + else: + return pattern + shortname diff --git a/morphlib/repoaliasresolver_tests.py b/morphlib/repoaliasresolver_tests.py new file mode 100644 index 00000000..c4ea16b0 --- /dev/null +++ b/morphlib/repoaliasresolver_tests.py @@ -0,0 +1,140 @@ +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import morphlib +import logging +import unittest + + +class RepoAliasResolverTests(unittest.TestCase): + + def setUp(self): + logging.disable(logging.critical) + self.aliases = [ + ('upstream=' + 'git://gitorious.org/baserock-morphs/%s#' + 'git@gitorious.org:baserock-morphs/%s.git'), + ('baserock=' + 'git://gitorious.org/baserock/%s#' + 'git@gitorious.org:baserock/%s.git'), + ('append=' + 'git://append/#' + 'git@append/'), + ('footrove-01=' + 'git://footrove.machine/%s#' + 'ssh://git@footrove.machine/%s.git'), + ] + self.resolver = morphlib.repoaliasresolver.RepoAliasResolver( + self.aliases) + + def test_resolve_urls_without_alias_prefix(self): + self.assertEqual(self.resolver.pull_url('bar'), 'bar') + self.assertEqual(self.resolver.push_url('bar'), 'bar') + + self.assertEqual(self.resolver.pull_url('foo'), 'foo') + self.assertEqual(self.resolver.push_url('foo'), 'foo') + + def test_resolve_urls_for_repos_of_one_alias(self): + url = self.resolver.pull_url('upstream:foo') + self.assertEqual(url, 'git://gitorious.org/baserock-morphs/foo') + url = self.resolver.push_url('upstream:foo') + self.assertEqual(url, 'git@gitorious.org:baserock-morphs/foo.git') + + url = self.resolver.pull_url('upstream:bar') + self.assertEqual(url, 'git://gitorious.org/baserock-morphs/bar') + url = self.resolver.push_url('upstream:bar') + self.assertEqual(url, 'git@gitorious.org:baserock-morphs/bar.git') + + def test_resolve_urls_for_repos_of_another_alias(self): + url = self.resolver.pull_url('baserock:foo') + self.assertEqual(url, 'git://gitorious.org/baserock/foo') + url = self.resolver.push_url('baserock:foo') + self.assertEqual(url, 'git@gitorious.org:baserock/foo.git') + + url = self.resolver.pull_url('baserock:bar') + self.assertEqual(url, 'git://gitorious.org/baserock/bar') + url = self.resolver.push_url('baserock:bar') + self.assertEqual(url, 'git@gitorious.org:baserock/bar.git') + + def test_resolve_urls_for_alias_with_dash(self): + url = self.resolver.pull_url('footrove-01:baz') + self.assertEqual(url, 'git://footrove.machine/baz') + url = self.resolver.push_url('footrove-01:baz') + self.assertEqual(url, 'ssh://git@footrove.machine/baz.git') + + def test_resolve_urls_for_unknown_alias(self): + self.assertEqual(self.resolver.pull_url('unknown:foo'), 'unknown:foo') + self.assertEqual(self.resolver.push_url('unknown:foo'), 'unknown:foo') + + self.assertEqual(self.resolver.pull_url('unknown:bar'), 'unknown:bar') + self.assertEqual(self.resolver.push_url('unknown:bar'), 'unknown:bar') + + def test_resolve_urls_for_pattern_without_placeholder(self): + self.assertEqual( + self.resolver.pull_url('append:foo'), 'git://append/foo') + self.assertEqual( + self.resolver.push_url('append:foo'), 'git@append/foo') + + self.assertEqual( + self.resolver.pull_url('append:bar'), 'git://append/bar') + self.assertEqual( + self.resolver.push_url('append:bar'), 'git@append/bar') + + def test_ignores_malformed_aliases(self): + resolver = morphlib.repoaliasresolver.RepoAliasResolver([ + 'malformed=git://git.malformed.url.org' + ]) + self.assertEqual(resolver.pull_url('malformed:foo'), 'malformed:foo') + self.assertEqual(resolver.push_url('malformed:foo'), 'malformed:foo') + + def test_gets_aliases_from_interpolated_patterns(self): + self.assertEqual( + self.resolver.aliases_from_url('git://gitorious.org/baserock/foo'), + ['baserock:foo']) + self.assertEqual( + self.resolver.aliases_from_url( + 'git@gitorious.org:baserock/foo.git'), + ['baserock:foo']) + self.assertEqual( + self.resolver.aliases_from_url( + 'git://gitorious.org/baserock-morphs/bar'), + ['upstream:bar']) + self.assertEqual( + self.resolver.aliases_from_url( + 'git@gitorious.org:baserock-morphs/bar.git'), + ['upstream:bar']) + + def test_gets_aliases_from_append_pattern(self): + self.assertEqual( + ['append:foo'], self.resolver.aliases_from_url('git://append/foo')) + self.assertEqual( + ['append:foo'], self.resolver.aliases_from_url('git@append/foo')) + + self.assertEqual( + ['append:bar'], self.resolver.aliases_from_url('git://append/bar')) + self.assertEqual( + ['append:bar'], self.resolver.aliases_from_url('git@append/bar')) + + def test_handles_multiple_possible_aliases(self): + resolver = morphlib.repoaliasresolver.RepoAliasResolver([ + 'trove=git://git.baserock.org/#ssh://git@git.baserock.org/', + 'baserock=git://git.baserock.org/baserock/' + '#ssh://git@git.baserock.org/baserock/', + ]) + self.assertEqual( + ['baserock:baserock/morphs', 'trove:baserock/baserock/morphs'], + resolver.aliases_from_url( + 'git://git.baserock.org/baserock/baserock/morphs')) diff --git a/morphlib/savefile.py b/morphlib/savefile.py new file mode 100644 index 00000000..2d87a54f --- /dev/null +++ b/morphlib/savefile.py @@ -0,0 +1,69 @@ +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import logging +import os +import tempfile + + +class SaveFile(file): + + '''Save files with a temporary name and rename when they're ready. + + This class acts exactly like the normal ``file`` class, except that + it is meant only for saving data to files. The data is written to + a temporary file, which gets renamed to the target name when the + open file is closed. This avoids readers of the file from getting + an incomplete file. + + Example: + + f = SaveFile('foo', 'w') + f.write(stuff) + f.close() + + The file will be called something like ``tmpCAFEBEEF`` until ``close`` + is called, at which point it gets renamed to ``foo``. + + If the writer decides the file is not worth saving, they can call the + ``abort`` method, which deletes the temporary file. + + ''' + + def __init__(self, filename, *args, **kwargs): + self.real_filename = filename + dirname = os.path.dirname(filename) + fd, self._savefile_tempname = tempfile.mkstemp(dir=dirname) + os.close(fd) + file.__init__(self, self._savefile_tempname, *args, **kwargs) + + def abort(self): + '''Abort file saving. + + The temporary file will be removed, and the universe is almost + exactly as if the file save had never started. + + ''' + + os.remove(self._savefile_tempname) + return file.close(self) + + def close(self): + ret = file.close(self) + logging.debug('Rename temporary file %s to %s' % + (self._savefile_tempname, self.real_filename)) + os.rename(self._savefile_tempname, self.real_filename) + return ret diff --git a/morphlib/savefile_tests.py b/morphlib/savefile_tests.py new file mode 100644 index 00000000..7ae2360d --- /dev/null +++ b/morphlib/savefile_tests.py @@ -0,0 +1,97 @@ +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import os +import shutil +import tempfile +import unittest + +import savefile + + +class SaveFileTests(unittest.TestCase): + + def cat(self, filename): + with open(filename) as f: + return f.read() + + def mkfile(self, filename, contents): + with open(filename, 'w') as f: + f.write(contents) + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.basename = 'filename' + self.filename = os.path.join(self.tempdir, self.basename) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_there_are_no_files_initially(self): + self.assertEqual(os.listdir(self.tempdir), []) + + def test_sets_real_filename(self): + f = savefile.SaveFile(self.filename, 'w') + self.assertEqual(f.real_filename, self.filename) + + def test_sets_name_to_temporary_name(self): + f = savefile.SaveFile(self.filename, 'w') + self.assertNotEqual(f.name, self.filename) + + def test_saves_new_file(self): + f = savefile.SaveFile(self.filename, 'w') + f.write('foo') + f.close() + self.assertEqual(os.listdir(self.tempdir), [self.basename]) + self.assertEqual(self.cat(self.filename), 'foo') + + def test_overwrites_existing_file(self): + self.mkfile(self.filename, 'yo!') + f = savefile.SaveFile(self.filename, 'w') + f.write('foo') + f.close() + self.assertEqual(os.listdir(self.tempdir), [self.basename]) + self.assertEqual(self.cat(self.filename), 'foo') + + def test_leaves_no_file_after_aborted_new_file(self): + f = savefile.SaveFile(self.filename, 'w') + f.write('foo') + f.abort() + self.assertEqual(os.listdir(self.tempdir), []) + + def test_leaves_original_file_after_aborted_overwrite(self): + self.mkfile(self.filename, 'yo!') + f = savefile.SaveFile(self.filename, 'w') + f.write('foo') + f.abort() + self.assertEqual(os.listdir(self.tempdir), [self.basename]) + self.assertEqual(self.cat(self.filename), 'yo!') + + def test_saves_normally_with_with(self): + with savefile.SaveFile(self.filename, 'w') as f: + f.write('foo') + self.assertEqual(os.listdir(self.tempdir), [self.basename]) + self.assertEqual(self.cat(self.filename), 'foo') + + def test_saves_normally_with_exception_within_with(self): + try: + with savefile.SaveFile(self.filename, 'w') as f: + f.write('foo') + raise Exception() + except Exception: + pass + self.assertEqual(os.listdir(self.tempdir), [self.basename]) + self.assertEqual(self.cat(self.filename), 'foo') diff --git a/morphlib/source.py b/morphlib/source.py new file mode 100644 index 00000000..4ad54ed9 --- /dev/null +++ b/morphlib/source.py @@ -0,0 +1,110 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import morphlib + + +class Source(object): + + '''Represent the source to be built. + + Has the following properties: + + * ``repo`` -- the git repository which contains the source + * ``repo_name`` -- name of the git repository which contains the source + * ``original_ref`` -- the git ref provided by the user or a morphology + * ``sha1`` -- the absolute git commit id for the revision we use + * ``tree`` -- the SHA1 of the tree corresponding to the commit + * ``morphology`` -- the in-memory representation of the morphology we use + * ``filename`` -- basename of the morphology filename + * ``cache_id`` -- a dict describing the components of the cache key + * ``cache_key`` -- a cache key to uniquely identify the artifact + * ``dependencies`` -- list of Artifacts that need to be built beforehand + * ``split_rules`` -- rules for splitting the source's produced artifacts + * ``artifacts`` -- the set of artifacts this source produces. + + ''' + + def __init__(self, name, repo_name, original_ref, sha1, tree, morphology, + filename, split_rules): + self.name = name + self.repo = None + self.repo_name = repo_name + self.original_ref = original_ref + self.sha1 = sha1 + self.tree = tree + self.morphology = morphology + self.filename = filename + self.cache_id = None + self.cache_key = None + self.dependencies = [] + + self.split_rules = split_rules + self.artifacts = None + + def __str__(self): # pragma: no cover + return '%s|%s|%s|%s' % (self.repo_name, + self.original_ref, + self.filename, + self.name) + + def __repr__(self): # pragma: no cover + return 'Source(%s)' % str(self) + + def basename(self): # pragma: no cover + return '%s.%s' % (self.cache_key, str(self.morphology['kind'])) + + def add_dependency(self, artifact): # pragma: no cover + if artifact not in self.dependencies: + self.dependencies.append(artifact) + if self not in artifact.dependents: + artifact.dependents.append(self) + + def depends_on(self, artifact): # pragma: no cover + '''Do we depend on ``artifact``?''' + return artifact in self.dependencies + + +def make_sources(reponame, ref, filename, absref, tree, morphology): + kind = morphology['kind'] + if kind in ('system', 'chunk'): + unifier = getattr(morphlib.artifactsplitrule, + 'unify_%s_matches' % kind) + split_rules = unifier(morphology) + # chunk and system sources are named after the morphology + source_name = morphology['name'] + source = morphlib.source.Source(source_name, reponame, ref, + absref, tree, morphology, + filename, split_rules) + source.artifacts = {name: morphlib.artifact.Artifact(source, name) + for name in split_rules.artifacts} + yield source + elif kind == 'stratum': # pragma: no cover + unifier = morphlib.artifactsplitrule.unify_stratum_matches + split_rules = unifier(morphology) + for name in split_rules.artifacts: + source = morphlib.source.Source( + name, # stratum source name is artifact name + reponame, ref, absref, tree, morphology, filename, + # stratum sources need to match the unified + # split rules, so they know to yield the match + # to a different source + split_rules) + source.artifacts = {name: morphlib.artifact.Artifact(source, name)} + yield source + else: + # cluster morphologies don't have sources + pass diff --git a/morphlib/source_tests.py b/morphlib/source_tests.py new file mode 100644 index 00000000..695041d3 --- /dev/null +++ b/morphlib/source_tests.py @@ -0,0 +1,59 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import unittest + +import morphlib + + +class SourceTests(unittest.TestCase): + + morphology_text = ''' + name: foo + kind: chunk + ''' + + def setUp(self): + self.repo_name = 'foo.repo' + self.original_ref = 'original/ref' + self.sha1 = 'CAFEF00D' + self.tree = 'F000000D' + loader = morphlib.morphloader.MorphologyLoader() + self.morphology = loader.load_from_string(self.morphology_text) + self.filename = 'foo.morph' + self.source, = morphlib.source.make_sources(self.repo_name, + self.original_ref, + self.filename, + self.sha1, self.tree, + self.morphology) + + def test_sets_repo_name(self): + self.assertEqual(self.source.repo_name, self.repo_name) + + def test_sets_repo_to_none_initially(self): + self.assertEqual(self.source.repo, None) + + def test_sets_original_ref(self): + self.assertEqual(self.source.original_ref, self.original_ref) + + def test_sets_sha1(self): + self.assertEqual(self.source.sha1, self.sha1) + + def test_sets_morphology(self): + self.assertEqual(self.source.morphology, self.morphology) + + def test_sets_filename(self): + self.assertEqual(self.source.filename, self.filename) diff --git a/morphlib/sourcepool.py b/morphlib/sourcepool.py new file mode 100644 index 00000000..6dfcb2c3 --- /dev/null +++ b/morphlib/sourcepool.py @@ -0,0 +1,56 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import collections + + +class SourcePool(object): + + '''Manage a collection of Source objects.''' + + def __init__(self): + self._sources = collections.defaultdict(dict) + self._order = [] + + def _key(self, repo_name, original_ref, filename): + return (repo_name, original_ref, filename) + + def add(self, source): + '''Add a source to the pool.''' + key = self._key(source.repo_name, + source.original_ref, + source.filename) + if key not in self._sources or source.name not in self._sources[key]: + self._sources[key][source.name] = source + self._order.append(source) + + def lookup(self, repo_name, original_ref, filename): + '''Find a source in the pool. + + Raise KeyError if it is not found. + + ''' + + key = self._key(repo_name, original_ref, filename) + return self._sources[key].values() + + def __iter__(self): + '''Iterate over sources in the pool, in the order they were added.''' + for source in self._order: + yield source + + def __len__(self): + return len(self._sources) diff --git a/morphlib/sourcepool_tests.py b/morphlib/sourcepool_tests.py new file mode 100644 index 00000000..f3740049 --- /dev/null +++ b/morphlib/sourcepool_tests.py @@ -0,0 +1,63 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import unittest + +import morphlib + + +class DummySource(object): + + def __init__(self): + self.name = 'dummy' + self.repo_name = 'repo' + self.original_ref = 'original/ref' + self.sha1 = 'dummy.sha1' + self.filename = 'dummy.morph' + self.morphology = {} + self.dependencies = [] + self.dependents = [] + + +class SourcePoolTests(unittest.TestCase): + + def setUp(self): + self.pool = morphlib.sourcepool.SourcePool() + self.source = DummySource() + + def test_is_empty_initially(self): + self.assertEqual(list(self.pool), []) + self.assertEqual(len(self.pool), 0) + + def test_adds_source(self): + self.pool.add(self.source) + self.assertEqual(list(self.pool), [self.source]) + + def test_looks_up_source(self): + self.pool.add(self.source) + result = self.pool.lookup(self.source.repo_name, + self.source.original_ref, + self.source.filename) + self.assertEqual(result, [self.source]) + + def test_iterates_in_add_order(self): + sources = [] + for i in range(10): + source = DummySource() + source.filename = str(i) + self.pool.add(source) + sources.append(source) + self.assertEqual(list(self.pool), sources) diff --git a/morphlib/stagingarea.py b/morphlib/stagingarea.py new file mode 100644 index 00000000..bfe0a716 --- /dev/null +++ b/morphlib/stagingarea.py @@ -0,0 +1,337 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import logging +import os +import shutil +import stat +import cliapp +from urlparse import urlparse +import tempfile + +import morphlib + + +class StagingArea(object): + + '''Represent the staging area for building software. + + The staging area is a temporary directory. In normal operation the build + dependencies of the artifact being built are installed into the staging + area and then 'chroot' is used to isolate the build processes from the host + system. Chunks built in 'test' or 'build-essential' mode have an empty + staging area and are allowed to use the tools of the host. + + ''' + + _base_path = ['/sbin', '/usr/sbin', '/bin', '/usr/bin'] + + def __init__(self, app, dirname, build_env, use_chroot=True, extra_env={}, + extra_path=[]): + self._app = app + self.dirname = dirname + self.builddirname = None + self.destdirname = None + self.mounted = [] + self._bind_readonly_mount = None + + self.use_chroot = use_chroot + self.env = build_env.env + self.env.update(extra_env) + + if use_chroot: + path = extra_path + build_env.extra_path + self._base_path + else: + rel_path = extra_path + build_env.extra_path + full_path = [os.path.normpath(dirname + p) for p in rel_path] + path = full_path + os.environ['PATH'].split(':') + self.env['PATH'] = ':'.join(path) + + # Wrapper to be overridden by unit tests. + def _mkdir(self, dirname): # pragma: no cover + os.makedirs(dirname) + + def _dir_for_source(self, source, suffix): + dirname = os.path.join(self.dirname, + '%s.%s' % (str(source.name), suffix)) + self._mkdir(dirname) + return dirname + + def builddir(self, source): + '''Create a build directory for a given source project. + + Return path to directory. + + ''' + + return self._dir_for_source(source, 'build') + + def destdir(self, source): + '''Create an installation target directory for a given source project. + + This is meant to be used as $DESTDIR when installing chunks. + Return path to directory. + + ''' + + return self._dir_for_source(source, 'inst') + + def relative(self, filename): + '''Return a filename relative to the staging area.''' + + if not self.use_chroot: + return filename + + dirname = self.dirname + if not dirname.endswith('/'): + dirname += '/' + + assert filename.startswith(dirname) + return filename[len(dirname) - 1:] # include leading slash + + def hardlink_all_files(self, srcpath, destpath): # pragma: no cover + '''Hardlink every file in the path to the staging-area + + If an exception is raised, the staging-area is indeterminate. + + ''' + + file_stat = os.lstat(srcpath) + mode = file_stat.st_mode + + if stat.S_ISDIR(mode): + # Ensure directory exists in destination, then recurse. + if not os.path.lexists(destpath): + os.makedirs(destpath) + dest_stat = os.stat(os.path.realpath(destpath)) + if not stat.S_ISDIR(dest_stat.st_mode): + raise IOError('Destination not a directory. source has %s' + ' destination has %s' % (srcpath, destpath)) + + for entry in os.listdir(srcpath): + self.hardlink_all_files(os.path.join(srcpath, entry), + os.path.join(destpath, entry)) + elif stat.S_ISLNK(mode): + # Copy the symlink. + if os.path.lexists(destpath): + os.remove(destpath) + os.symlink(os.readlink(srcpath), destpath) + + elif stat.S_ISREG(mode): + # Hardlink the file. + if os.path.lexists(destpath): + os.remove(destpath) + os.link(srcpath, destpath) + + elif stat.S_ISCHR(mode) or stat.S_ISBLK(mode): + # Block or character device. Put contents of st_dev in a mknod. + if os.path.lexists(destpath): + os.remove(destpath) + os.mknod(destpath, file_stat.st_mode, file_stat.st_rdev) + os.chmod(destpath, file_stat.st_mode) + + else: + # Unsupported type. + raise IOError('Cannot extract %s into staging-area. Unsupported' + ' type.' % srcpath) + + def install_artifact(self, handle): + '''Install a build artifact into the staging area. + + We access the artifact via an open file handle. For now, we assume + the artifact is a tarball. + + ''' + + chunk_cache_dir = os.path.join(self._app.settings['tempdir'], 'chunks') + unpacked_artifact = os.path.join( + chunk_cache_dir, os.path.basename(handle.name) + '.d') + if not os.path.exists(unpacked_artifact): + self._app.status( + msg='Unpacking chunk from cache %(filename)s', + filename=os.path.basename(handle.name)) + savedir = tempfile.mkdtemp(dir=chunk_cache_dir) + try: + morphlib.bins.unpack_binary_from_file( + handle, savedir + '/') + except BaseException, e: # pragma: no cover + shutil.rmtree(savedir) + raise + # TODO: This rename is not concurrency safe if two builds are + # extracting the same chunk, one build will fail because + # the other renamed its tempdir here first. + os.rename(savedir, unpacked_artifact) + + if not os.path.exists(self.dirname): + self._mkdir(self.dirname) + + self.hardlink_all_files(unpacked_artifact, self.dirname) + + def remove(self): + '''Remove the entire staging area. + + Do not expect anything with the staging area to work after this + method is called. Be careful about calling this method if + the filesystem root directory was given as the dirname. + + ''' + + shutil.rmtree(self.dirname) + + to_mount = ( + ('proc', 'proc', 'none'), + ('dev/shm', 'tmpfs', 'none'), + ) + + def mount_ccachedir(self, source): #pragma: no cover + ccache_dir = self._app.settings['compiler-cache-dir'] + if not os.path.isdir(ccache_dir): + os.makedirs(ccache_dir) + # Get a path for the repo's ccache + ccache_url = source.repo.url + ccache_path = urlparse(ccache_url).path + ccache_repobase = os.path.basename(ccache_path) + if ':' in ccache_repobase: # the basename is a repo-alias + resolver = morphlib.repoaliasresolver.RepoAliasResolver( + self._app.settings['repo-alias']) + ccache_url = resolver.pull_url(ccache_repobase) + ccache_path = urlparse(ccache_url).path + ccache_repobase = os.path.basename(ccache_path) + if ccache_repobase.endswith('.git'): + ccache_repobase = ccache_repobase[:-len('.git')] + + ccache_repodir = os.path.join(ccache_dir, ccache_repobase) + # Make sure that directory exists + if not os.path.isdir(ccache_repodir): + os.mkdir(ccache_repodir) + # Get the destination path + ccache_destdir= os.path.join(self.dirname, 'tmp', 'ccache') + # Make sure that the destination exists. We'll create /tmp if necessary + # to avoid breaking when faced with an empty staging area. + if not os.path.isdir(ccache_destdir): + os.makedirs(ccache_destdir) + # Mount it into the staging-area + self._app.runcmd(['mount', '--bind', ccache_repodir, + ccache_destdir]) + return ccache_destdir + + def do_mounts(self, setup_mounts): # pragma: no cover + if not setup_mounts: + return + for mount_point, mount_type, source in self.to_mount: + logging.debug('Mounting %s in staging area' % mount_point) + path = os.path.join(self.dirname, mount_point) + if not os.path.exists(path): + os.makedirs(path) + morphlib.fsutils.mount(self._app.runcmd, source, path, mount_type) + self.mounted.append(path) + return + + def do_unmounts(self): # pragma: no cover + for path in reversed(self.mounted): + logging.debug('Unmounting %s in staging area' % path) + morphlib.fsutils.unmount(self._app.runcmd, path) + + def chroot_open(self, source, setup_mounts): # pragma: no cover + '''Setup staging area for use as a chroot.''' + + assert self.builddirname == None and self.destdirname == None + + builddir = self.builddir(source) + destdir = self.destdir(source) + self.builddirname = builddir + self.destdirname = destdir + + self.do_mounts(setup_mounts) + + if not self._app.settings['no-ccache']: + self.mounted.append(self.mount_ccachedir(source)) + + return builddir, destdir + + def chroot_close(self): # pragma: no cover + '''Undo changes by chroot_open. + + This should be called after the staging area is no longer needed. + + ''' + + self.do_unmounts() + + def runcmd(self, argv, **kwargs): # pragma: no cover + '''Run a command in a chroot in the staging area.''' + assert 'env' not in kwargs + kwargs['env'] = self.env + if 'extra_env' in kwargs: + kwargs['env'].update(kwargs['extra_env']) + del kwargs['extra_env'] + + if 'cwd' in kwargs: + cwd = kwargs['cwd'] + del kwargs['cwd'] + else: + cwd = '/' + + chroot_dir = self.dirname if self.use_chroot else '/' + temp_dir = kwargs["env"].get("TMPDIR", "/tmp") + + staging_dirs = [self.builddirname, self.destdirname] + if self.use_chroot: + staging_dirs += ["dev", "proc", temp_dir.lstrip('/')] + do_not_mount_dirs = [os.path.join(self.dirname, d) + for d in staging_dirs] + if not self.use_chroot: + do_not_mount_dirs += [temp_dir] + + logging.debug("Not mounting dirs %r" % do_not_mount_dirs) + + real_argv = ['linux-user-chroot', '--chdir', cwd, '--unshare-net'] + for d in morphlib.fsutils.invert_paths(os.walk(chroot_dir), + do_not_mount_dirs): + if not os.path.islink(d): + real_argv += ['--mount-readonly', self.relative(d)] + + real_argv += [chroot_dir] + + real_argv += argv + + try: + if 'logfile' in kwargs and kwargs['logfile'] != None: + logfile = kwargs['logfile'] + del kwargs['logfile'] + + teecmd = ['tee', '-a', logfile] + return self._app.runcmd(real_argv, teecmd, **kwargs) + else: + return self._app.runcmd(real_argv, **kwargs) + except cliapp.AppException as e: + raise cliapp.AppException('In staging area %s: running ' + 'command \'%s\' failed.' % + (self.dirname, ' '.join(argv))) + + def abort(self): # pragma: no cover + '''Handle what to do with a staging area in the case of failure. + This may either remove it or save it for later inspection. + ''' + # TODO: when we add the option to throw away failed builds, + # hook it up here + + + dest_dir = os.path.join(self._app.settings['tempdir'], + 'failed', os.path.basename(self.dirname)) + os.rename(self.dirname, dest_dir) + self.dirname = dest_dir + diff --git a/morphlib/stagingarea_tests.py b/morphlib/stagingarea_tests.py new file mode 100644 index 00000000..dc43e4f6 --- /dev/null +++ b/morphlib/stagingarea_tests.py @@ -0,0 +1,143 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import os +import shutil +import tarfile +import tempfile +import unittest + +import morphlib + + +class FakeBuildEnvironment(object): + + def __init__(self): + self.env = { + } + self.extra_path = ['/extra-path'] + +class FakeSource(object): + + def __init__(self): + self.morphology = { + 'name': 'le-name', + } + self.name = 'le-name' + + +class FakeApplication(object): + + def __init__(self, cachedir, tempdir): + self.settings = { + 'cachedir': cachedir, + 'tempdir': tempdir, + } + for leaf in ('chunks',): + d = os.path.join(tempdir, leaf) + if not os.path.exists(d): + os.makedirs(d) + + def runcmd(self, *args, **kwargs): + return cliapp.runcmd(*args, **kwargs) + + def runcmd_unchecked(self, *args, **kwargs): + return cliapp.runcmd_unchecked(*args, **kwargs) + + def status(self, **kwargs): + pass + + +class StagingAreaTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.cachedir = os.path.join(self.tempdir, 'cachedir') + os.mkdir(self.cachedir) + os.mkdir(os.path.join(self.cachedir, 'artifacts')) + self.staging = os.path.join(self.tempdir, 'staging') + self.created_dirs = [] + self.build_env = FakeBuildEnvironment() + self.sa = morphlib.stagingarea.StagingArea( + FakeApplication(self.cachedir, self.tempdir), self.staging, + self.build_env) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def create_chunk(self): + chunkdir = os.path.join(self.tempdir, 'chunk') + os.mkdir(chunkdir) + with open(os.path.join(chunkdir, 'file.txt'), 'w'): + pass + chunk_tar = os.path.join(self.tempdir, 'chunk.tar') + tf = tarfile.TarFile(name=chunk_tar, mode='w') + tf.add(chunkdir, arcname='.') + tf.close() + + return chunk_tar + + def list_tree(self, root): + files = [] + for dirname, subdirs, basenames in os.walk(root): + paths = [os.path.join(dirname, x) for x in basenames] + for x in [dirname] + sorted(paths): + files.append(x[len(root):] or '/') + return files + + def fake_mkdir(self, dirname): + self.created_dirs.append(dirname) + + def test_remembers_specified_directory(self): + self.assertEqual(self.sa.dirname, self.staging) + + def test_creates_build_directory(self): + source = FakeSource() + self.sa._mkdir = self.fake_mkdir + dirname = self.sa.builddir(source) + self.assertEqual(self.created_dirs, [dirname]) + self.assertTrue(dirname.startswith(self.staging)) + + def test_creates_install_directory(self): + source = FakeSource() + self.sa._mkdir = self.fake_mkdir + dirname = self.sa.destdir(source) + self.assertEqual(self.created_dirs, [dirname]) + self.assertTrue(dirname.startswith(self.staging)) + + def test_makes_relative_name(self): + filename = os.path.join(self.staging, 'foobar') + self.assertEqual(self.sa.relative(filename), '/foobar') + + def test_installs_artifact(self): + chunk_tar = self.create_chunk() + with open(chunk_tar, 'rb') as f: + self.sa.install_artifact(f) + self.assertEqual(self.list_tree(self.staging), ['/', '/file.txt']) + + def test_removes_everything(self): + chunk_tar = self.create_chunk() + with open(chunk_tar, 'rb') as f: + self.sa.install_artifact(f) + self.sa.remove() + self.assertFalse(os.path.exists(self.staging)) + + def test_supports_non_isolated_mode(self): + sa = morphlib.stagingarea.StagingArea( + object(), self.staging, self.build_env, use_chroot=False) + filename = os.path.join(self.staging, 'foobar') + self.assertEqual(sa.relative(filename), filename) diff --git a/morphlib/stopwatch.py b/morphlib/stopwatch.py new file mode 100644 index 00000000..29e584bd --- /dev/null +++ b/morphlib/stopwatch.py @@ -0,0 +1,71 @@ +# Copyright (C) 2011-2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import operator +import datetime + + +class Stopwatch(object): + + def __init__(self): + self.ticks = {} + self.context_stack = [] + + def tick(self, reference_object, name): + if not reference_object in self.ticks: + self.ticks[reference_object] = {} + self.ticks[reference_object][name] = datetime.datetime.now() + + def start(self, reference_object): + self.tick(reference_object, 'start') + + def stop(self, reference_object): + self.tick(reference_object, 'stop') + + def times(self, reference_object): + return self.ticks[reference_object] + + def time(self, reference_object, name): + return self.ticks[reference_object][name] + + def start_time(self, reference_object): + return self.ticks[reference_object]['start'] + + def stop_time(self, reference_object): + return self.ticks[reference_object]['stop'] + + def start_stop_delta(self, reference_object): + return (self.stop_time(reference_object) - + self.start_time(reference_object)) + + def start_stop_seconds(self, reference_object): + delta = self.start_stop_delta(reference_object) + return (delta.days * 24 * 3600 + + delta.seconds + + operator.truediv(delta.microseconds, 10 ** 6)) + + def __call__(self, reference_object): + self.context_stack.append(reference_object) + return self + + def __enter__(self): + self.start(self.context_stack[-1]) + return self + + def __exit__(self, *args): + self.stop(self.context_stack[-1]) + self.context_stack.pop() + return False # cause any exception to be re-raised diff --git a/morphlib/stopwatch_tests.py b/morphlib/stopwatch_tests.py new file mode 100644 index 00000000..deb528d5 --- /dev/null +++ b/morphlib/stopwatch_tests.py @@ -0,0 +1,71 @@ +# Copyright (C) 2011-2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import datetime +import unittest + +import morphlib + + +class StopwatchTests(unittest.TestCase): + + def setUp(self): + self.stopwatch = morphlib.stopwatch.Stopwatch() + + def test_tick(self): + self.stopwatch.tick('tick', 'a') + self.assertTrue(self.stopwatch.times('tick')) + self.assertTrue(self.stopwatch.time('tick', 'a')) + self.assertTrue('a' in self.stopwatch.times('tick')) + self.assertEqual(self.stopwatch.time('tick', 'a'), + self.stopwatch.times('tick')['a']) + + now = datetime.datetime.now() + self.assertTrue(self.stopwatch.time('tick', 'a') < now) + + def test_start_stop(self): + self.stopwatch.start('start-stop') + self.assertTrue(self.stopwatch.times('start-stop')) + self.assertTrue(self.stopwatch.start_time('start-stop')) + + self.stopwatch.stop('start-stop') + self.assertTrue(self.stopwatch.times('start-stop')) + self.assertTrue(self.stopwatch.stop_time('start-stop')) + + start = self.stopwatch.start_time('start-stop') + stop = self.stopwatch.stop_time('start-stop') + + our_delta = stop - start + watch_delta = self.stopwatch.start_stop_delta('start-stop') + self.assertEqual(our_delta, watch_delta) + + self.assertTrue(self.stopwatch.start_stop_seconds('start-stop') > 0) + + def test_with(self): + with self.stopwatch('foo'): + pass + self.assertTrue(self.stopwatch.start_stop_seconds('foo') < 1.0) + + def test_with_within_with(self): + with self.stopwatch('foo'): + with self.stopwatch('bar'): + pass + self.assertTrue(self.stopwatch.start_time('foo') is not None) + self.assertTrue(self.stopwatch.stop_time('foo') is not None) + self.assertTrue(self.stopwatch.start_time('bar') is not None) + self.assertTrue(self.stopwatch.stop_time('bar') is not None) + self.assertTrue(self.stopwatch.start_stop_seconds('foo') < 1.0) + self.assertTrue(self.stopwatch.start_stop_seconds('bar') < 1.0) diff --git a/morphlib/sysbranchdir.py b/morphlib/sysbranchdir.py new file mode 100644 index 00000000..4351c6b3 --- /dev/null +++ b/morphlib/sysbranchdir.py @@ -0,0 +1,263 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import cliapp +import os +import urlparse +import uuid + +import morphlib + + +class SystemBranchDirectoryAlreadyExists(morphlib.Error): + + def __init__(self, root_directory): + self.msg = ( + "%s: File exists" % + root_directory) + + +class NotInSystemBranch(morphlib.Error): + + def __init__(self, dirname): + self.msg = ( + "Can't find the system branch directory.\n" + "Morph must be built and deployed within " + "the system branch checkout.") + + +class SystemBranchDirectory(object): + + '''A directory containing a checked out system branch.''' + + def __init__(self, + root_directory, root_repository_url, system_branch_name): + self.root_directory = os.path.abspath(root_directory) + self.root_repository_url = root_repository_url + self.system_branch_name = system_branch_name + + @property + def _magic_path(self): + return os.path.join(self.root_directory, '.morph-system-branch') + + @property + def _config_path(self): + return os.path.join(self._magic_path, 'config') + + def set_config(self, key, value): + '''Set a configuration key/value pair.''' + morphlib.git.gitcmd(cliapp.runcmd, 'config', '-f', + self._config_path, key, value) + + def get_config(self, key): + '''Get a configuration value for a given key.''' + value = morphlib.git.gitcmd(cliapp.runcmd, 'config', '-f', + self._config_path, key) + return value.strip() + + def _find_git_directory(self, repo_url): + for gd in self.list_git_directories(): + if gd.get_config('morph.repository') == repo_url: + return gd.dirname + return None + + def _fabricate_git_directory_name(self, repo_url): + # Parse the URL. If the path component is absolute, we assume + # it's a real URL; otherwise, an aliased URL. + parts = urlparse.urlparse(repo_url) + + if os.path.isabs(parts.path): + # Remove .git suffix, if any. + path = parts.path + if path.endswith('.git'): + path = path[:-len('.git')] + + # Add the domain name etc (netloc). Ignore any other parts. + # Note that we _know_ the path starts with a slash, so we avoid + # adding one here. + relative = '%s%s' % (parts.netloc, path) + else: + relative = repo_url + + # Replace colons with slashes. + relative = '/'.join(relative.split(':')) + + # Remove anyleading slashes, or os.path.join below will only + # use the relative part (since it's absolute, not relative). + relative = relative.lstrip('/') + + return os.path.join(self.root_directory, relative) + + def get_git_directory_name(self, repo_url): + '''Return directory pathname for a given git repository. + + If the repository has already been cloned, then it returns the + path to that, if not it will fabricate a path based on the url. + + If the URL is a real one (not aliased), the schema and leading // + are removed from it, as is a .git suffix. + + Any colons in the URL path or network location are replaced + with slashes, so that directory paths do not contain colons. + This avoids problems with PYTHONPATH, PATH, and other things + that use colon as a separator. + + ''' + found_repo = self._find_git_directory(repo_url) + if not found_repo: + return self._fabricate_git_directory_name(repo_url) + return found_repo + + def get_filename(self, repo_url, relative): + '''Return full pathname to a file in a checked out repository. + + This is a convenience function. + + ''' + + return os.path.join(self.get_git_directory_name(repo_url), relative) + + def clone_cached_repo(self, cached_repo, checkout_ref): + '''Clone a cached git repository into the system branch directory. + + The cloned repository will NOT have the system branch's git branch + checked out: instead, checkout_ref is checked out (this is for + backwards compatibility with older implementation of "morph + branch"; it may change later). The system branch's git branch + is NOT created: the caller will need to do that. Submodules are + NOT checked out. + + The "origin" remote will be set to follow the cached repository's + upstream. Remotes are not updated. + + ''' + + # Do the clone. + dirname = self.get_git_directory_name(cached_repo.original_name) + gd = morphlib.gitdir.clone_from_cached_repo( + cached_repo, dirname, checkout_ref) + + # Remember the repo name we cloned from in order to be able + # to identify the repo again later using the same name, even + # if the user happens to rename the directory. + gd.set_config('morph.repository', cached_repo.original_name) + + # Create a UUID for the clone. We will use this for naming + # temporary refs, e.g. for building. + gd.set_config('morph.uuid', uuid.uuid4().hex) + + # Configure the "origin" remote to use the upstream git repository, + # and not the locally cached copy. + resolver = morphlib.repoaliasresolver.RepoAliasResolver( + cached_repo.app.settings['repo-alias']) + remote = gd.get_remote('origin') + remote.set_fetch_url(resolver.pull_url(cached_repo.url)) + gd.set_config( + 'url.%s.pushInsteadOf' % + resolver.push_url(cached_repo.original_name), + resolver.pull_url(cached_repo.url)) + + return gd + + def list_git_directories(self): + '''List all git directories in a system branch directory. + + The list will contain zero or more GitDirectory objects. + + ''' + + return (morphlib.gitdir.GitDirectory(dirname) + for dirname in + morphlib.util.find_leaves(self.root_directory, '.git')) + + # Not covered by unit tests, since testing the functionality spans + # multiple modules and only tests useful output with a full system + # branch, so it is instead covered by integration tests. + def load_all_morphologies(self, loader): # pragma: no cover + gd_name = self.get_git_directory_name(self.root_repository_url) + gd = morphlib.gitdir.GitDirectory(gd_name) + mf = morphlib.morphologyfinder.MorphologyFinder(gd) + for filename in (f for f in mf.list_morphologies() + if not gd.is_symlink(f)): + text = mf.read_morphology(filename) + m = loader.load_from_string(text, filename=filename) + m.repo_url = self.root_repository_url + m.ref = self.system_branch_name + yield m + + +def create(root_directory, root_repository_url, system_branch_name): + '''Create a new system branch directory on disk. + + Return a SystemBranchDirectory object that represents the directory. + + The directory MUST NOT exist already. If it does, + SystemBranchDirectoryAlreadyExists is raised. + + Note that this does NOT check out the root repository, or do any + other git cloning. + + ''' + + if os.path.exists(root_directory): + raise SystemBranchDirectoryAlreadyExists(root_directory) + + magic_dir = os.path.join(root_directory, '.morph-system-branch') + os.makedirs(root_directory) + os.mkdir(magic_dir) + + sb = SystemBranchDirectory( + root_directory, root_repository_url, system_branch_name) + sb.set_config('branch.name', system_branch_name) + sb.set_config('branch.root', root_repository_url) + sb.set_config('branch.uuid', uuid.uuid4().hex) + + return sb + + +def open(root_directory): + '''Open an existing system branch directory.''' + + # Ugly hack follows. + sb = SystemBranchDirectory(root_directory, None, None) + root_repository_url = sb.get_config('branch.root') + system_branch_name = sb.get_config('branch.name') + + return SystemBranchDirectory( + root_directory, root_repository_url, system_branch_name) + + +def open_from_within(dirname): + '''Open a system branch directory, given any directory. + + The directory can be within the system branch root directory, + or it can be a parent, in some cases. If each parent on the + path from dirname to the system branch root directory has no + siblings, this function will find it. + + ''' + + root_directory = morphlib.util.find_root( + dirname, '.morph-system-branch') + if root_directory is None: + root_directory = morphlib.util.find_leaf( + dirname, '.morph-system-branch') + if root_directory is None: + raise NotInSystemBranch(dirname) + return morphlib.sysbranchdir.open(root_directory) + diff --git a/morphlib/sysbranchdir_tests.py b/morphlib/sysbranchdir_tests.py new file mode 100644 index 00000000..1aca54e6 --- /dev/null +++ b/morphlib/sysbranchdir_tests.py @@ -0,0 +1,222 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import cliapp +import os +import shutil +import tempfile +import unittest + +import morphlib + + +class SystemBranchDirectoryTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.root_directory = os.path.join(self.tempdir, 'rootdir') + self.root_repository_url = 'test:morphs' + self.system_branch_name = 'foo/bar' + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def create_fake_cached_repo(self): + + class FakeCachedRepo(object): + + def __init__(self, url, path): + self.app = self + self.settings = { + 'repo-alias': [], + } + self.original_name = url + self.url = 'git://blahlbah/blah/blahblahblah.git' + self.path = path + + os.mkdir(self.path) + morphlib.git.gitcmd(cliapp.runcmd, 'init', self.path) + with open(os.path.join(self.path, 'filename'), 'w') as f: + f.write('this is a file\n') + morphlib.git.gitcmd(cliapp.runcmd, 'add', 'filename', + cwd=self.path) + morphlib.git.gitcmd(cliapp.runcmd, 'commit', '-m', 'initial', + cwd=self.path) + + def clone_checkout(self, ref, target_dir): + morphlib.git.gitcmd(cliapp.runcmd, 'clone', '-b', ref, + self.path, target_dir) + + subdir = tempfile.mkdtemp(dir=self.tempdir) + path = os.path.join(subdir, 'foo') + return FakeCachedRepo(self.root_repository_url, path) + + def test_creates_system_branch_directory(self): + sb = morphlib.sysbranchdir.create( + self.root_directory, + self.root_repository_url, + self.system_branch_name) + self.assertEqual(sb.root_directory, self.root_directory) + self.assertEqual(sb.root_repository_url, self.root_repository_url) + self.assertEqual(sb.system_branch_name, self.system_branch_name) + + magic_dir = os.path.join(self.root_directory, '.morph-system-branch') + self.assertTrue(os.path.isdir(self.root_directory)) + self.assertTrue(os.path.isdir(magic_dir)) + self.assertTrue(os.path.isfile(os.path.join(magic_dir, 'config'))) + self.assertEqual( + sb.get_config('branch.root'), self.root_repository_url) + self.assertEqual( + sb.get_config('branch.name'), self.system_branch_name) + self.assertTrue(sb.get_config('branch.uuid')) + + def test_opens_system_branch_directory(self): + morphlib.sysbranchdir.create( + self.root_directory, + self.root_repository_url, + self.system_branch_name) + sb = morphlib.sysbranchdir.open(self.root_directory) + self.assertEqual(sb.root_directory, self.root_directory) + self.assertEqual(sb.root_repository_url, self.root_repository_url) + self.assertEqual(sb.system_branch_name, self.system_branch_name) + + def test_opens_system_branch_directory_from_a_subdirectory(self): + morphlib.sysbranchdir.create( + self.root_directory, + self.root_repository_url, + self.system_branch_name) + subdir = os.path.join(self.root_directory, 'a', 'b', 'c') + os.makedirs(subdir) + sb = morphlib.sysbranchdir.open_from_within(subdir) + self.assertEqual(sb.root_directory, self.root_directory) + self.assertEqual(sb.root_repository_url, self.root_repository_url) + self.assertEqual(sb.system_branch_name, self.system_branch_name) + + def test_fails_opening_system_branch_directory_when_none_exists(self): + self.assertRaises( + morphlib.sysbranchdir.NotInSystemBranch, + morphlib.sysbranchdir.open_from_within, + self.tempdir) + + def test_opens_system_branch_directory_when_it_is_the_only_child(self): + deep_root = os.path.join(self.tempdir, 'a', 'b', 'c') + morphlib.sysbranchdir.create( + deep_root, + self.root_repository_url, + self.system_branch_name) + sb = morphlib.sysbranchdir.open(deep_root) + self.assertEqual(sb.root_directory, deep_root) + self.assertEqual(sb.root_repository_url, self.root_repository_url) + self.assertEqual(sb.system_branch_name, self.system_branch_name) + + def test_fails_to_create_if_directory_already_exists(self): + os.mkdir(self.root_directory) + self.assertRaises( + morphlib.sysbranchdir.SystemBranchDirectoryAlreadyExists, + morphlib.sysbranchdir.create, + self.root_directory, + self.root_repository_url, + self.system_branch_name) + + def test_sets_and_gets_configuration_values(self): + sb = morphlib.sysbranchdir.create( + self.root_directory, + self.root_repository_url, + self.system_branch_name) + sb.set_config('foo.key', 'foovalue') + + sb2 = morphlib.sysbranchdir.open(self.root_directory) + self.assertEqual(sb2.get_config('foo.key'), 'foovalue') + + def test_reports_correct_name_for_git_directory_from_aliases_url(self): + sb = morphlib.sysbranchdir.create( + self.root_directory, + self.root_repository_url, + self.system_branch_name) + self.assertEqual( + sb.get_git_directory_name('baserock:baserock/morph'), + os.path.join(self.root_directory, 'baserock/baserock/morph')) + + def test_reports_correct_name_for_git_directory_from_real_url(self): + stripped = 'git.baserock.org/baserock/baserock/morph' + url = 'git://%s.git' % stripped + sb = morphlib.sysbranchdir.create( + self.root_directory, + url, + self.system_branch_name) + self.assertEqual( + sb.get_git_directory_name(url), + os.path.join(self.root_directory, stripped)) + + def test_reports_correct_path_for_file_in_repository(self): + sb = morphlib.sysbranchdir.create( + self.root_directory, + self.root_repository_url, + self.system_branch_name) + self.assertEqual( + sb.get_filename('test:chunk', 'foo'), + os.path.join(self.root_directory, 'test/chunk/foo')) + + def test_reports_correct_name_for_git_directory_from_file_url(self): + stripped = 'foobar/morphs' + url = 'file:///%s.git' % stripped + sb = morphlib.sysbranchdir.create( + self.root_directory, + url, + self.system_branch_name) + self.assertEqual( + sb.get_git_directory_name(url), + os.path.join(self.root_directory, stripped)) + + def test_clones_git_repository(self): + + sb = morphlib.sysbranchdir.create( + self.root_directory, + self.root_repository_url, + self.system_branch_name) + + cached_repo = self.create_fake_cached_repo() + gd = sb.clone_cached_repo(cached_repo, 'master') + + self.assertEqual( + gd.dirname, + sb.get_git_directory_name(cached_repo.original_name)) + + def test_lists_git_directories(self): + + def fake_git_clone(dirname, url, branch): + os.mkdir(dirname) + subdir = os.path.join(dirname, '.git') + os.mkdir(subdir) + + sb = morphlib.sysbranchdir.create( + self.root_directory, + self.root_repository_url, + self.system_branch_name) + + sb._git_clone = fake_git_clone + + cached_repo = self.create_fake_cached_repo() + sb.clone_cached_repo(cached_repo, 'master') + + gd_list = list(sb.list_git_directories()) + self.assertEqual(len(gd_list), 1) + self.assertEqual( + gd_list[0].dirname, + sb.get_git_directory_name(cached_repo.original_name)) + diff --git a/morphlib/systemmetadatadir.py b/morphlib/systemmetadatadir.py new file mode 100644 index 00000000..7e89142c --- /dev/null +++ b/morphlib/systemmetadatadir.py @@ -0,0 +1,88 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import collections +import glob +import json +import os + + +class SystemMetadataDir(collections.MutableMapping): + + '''An abstraction over the /baserock metadata directory. + + This allows methods of iterating over it, and accessing it like + a dict. + + The /baserock metadata directory contains information about all of + the chunks in a built system. It exists to provide traceability from + the input sources to the output. + + If you create the object with smd = SystemMetadataDir('/baserock') + data = smd['key'] will read /baserock/key.meta and return its JSON + encoded contents as native python objects. + + smd['key'] = data will write data to /baserock/key.meta as JSON + + The key may not have '\0' characters in it since the underlying + system calls don't support embedded NUL bytes. + + The key may not have '/' characters in it since we do not support + morphologies with slashes in their names. + + ''' + + def __init__(self, metadata_path): + collections.MutableMapping.__init__(self) + self._metadata_path = metadata_path + + def _join_path(self, *args): + return os.path.join(self._metadata_path, *args) + + def _raw_path_iter(self): + return glob.iglob(self._join_path('*.meta')) + + @staticmethod + def _check_key(key): + if any(c in key for c in "\0/"): + raise KeyError(key) + + def __getitem__(self, key): + self._check_key(key) + try: + with open(self._join_path('%s.meta' % key), 'r') as f: + return json.load(f, encoding='unicode-escape') + except IOError: + raise KeyError(key) + + def __setitem__(self, key, value): + self._check_key(key) + with open(self._join_path('%s.meta' % key), 'w') as f: + json.dump(value, f, indent=4, sort_keys=True, + encoding='unicode-escape') + + def __delitem__(self, key): + self._check_key(key) + os.unlink(self._join_path('%s.meta' % key)) + + def __iter__(self): + return (os.path.basename(fn)[:-len('.meta')] + for fn in self._raw_path_iter()) + + def __len__(self): + return len(list(self._raw_path_iter())) diff --git a/morphlib/systemmetadatadir_tests.py b/morphlib/systemmetadatadir_tests.py new file mode 100644 index 00000000..0126f862 --- /dev/null +++ b/morphlib/systemmetadatadir_tests.py @@ -0,0 +1,75 @@ +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import operator +import os +import shutil +import tempfile +import unittest + +import morphlib + + +class SystemMetadataDirTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.metadatadir = os.path.join(self.tempdir, 'baserock') + os.mkdir(self.metadatadir) + self.smd = morphlib.systemmetadatadir.SystemMetadataDir( + self.metadatadir) + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_add_new(self): + self.smd['key'] = {'foo': 'bar'} + self.assertEqual(self.smd['key']['foo'], 'bar') + + def test_replace(self): + self.smd['key'] = {'foo': 'bar'} + self.smd['key'] = {'foo': 'baz'} + self.assertEqual(self.smd['key']['foo'], 'baz') + + def test_remove(self): + self.smd['key'] = {'foo': 'bar'} + del self.smd['key'] + self.assertTrue('key' not in self.smd) + + def test_iterate(self): + self.smd['build-essential'] = "Some data" + self.smd['core'] = "More data" + self.smd['foundation'] = "Yet more data" + self.assertEqual(sorted(self.smd.keys()), + ['build-essential', 'core', 'foundation']) + self.assertEqual(dict(self.smd.iteritems()), + { + 'build-essential': "Some data", + 'core': "More data", + 'foundation': "Yet more data", + }) + + def test_raises_KeyError(self): + self.assertRaises(KeyError, operator.getitem, self.smd, 'key') + + def test_validates_keys(self): + for key in ('foo/bar', 'baz\0quux'): + self.assertRaises(KeyError, operator.getitem, self.smd, key) + self.assertRaises(KeyError, operator.setitem, + self.smd, key, 'value') + self.assertRaises(KeyError, operator.delitem, self.smd, key) diff --git a/morphlib/util.py b/morphlib/util.py new file mode 100644 index 00000000..dc3dd474 --- /dev/null +++ b/morphlib/util.py @@ -0,0 +1,503 @@ +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import contextlib +import itertools +import os +import re +import subprocess + +import fs.osfs + +import morphlib + +'''Utility functions for morph.''' + + +# It is intentional that if collections does not have OrderedDict that +# simplejson is also used in preference to json, as OrderedDict became +# a member of collections in the same release json got its object_pairs_hook +try: # pragma: no cover + from collections import OrderedDict + import json +except ImportError: # pragma: no cover + from ordereddict import OrderedDict + import simplejson as json + +try: + from multiprocessing import cpu_count +except NotImplementedError: # pragma: no cover + cpu_count = lambda: 1 +import os + + +def indent(string, spaces=4): + '''Return ``string`` indented by ``spaces`` spaces. + + The final line is not terminated by a newline. This makes it easy + to use this function for indenting long text for logging: the + logging library adds a newline, so not including it in the indented + text avoids a spurious empty line in the log file. + + This also makes the result be a plain ASCII encoded string. + + ''' + + if type(string) == unicode: # pragma: no cover + string = string.decode('utf-8') + lines = string.splitlines() + lines = ['%*s%s' % (spaces, '', line) for line in lines] + return '\n'.join(lines) + + +def sanitise_morphology_path(morph_name): + '''Turn morph_name into a file path to a morphology. + + We support both a file path being provided, and just the morphology + name for backwards compatibility. + + ''' + # If it has a / it must be a path, so return it unmolested + if '/' in morph_name: + return morph_name + # Must be an old format, which is always name + .morph + elif not morph_name.endswith('.morph'): + return morph_name + '.morph' + # morphology already ends with .morph + else: + return morph_name + + +def make_concurrency(cores=None): + '''Return the number of concurrent jobs for make. + + This will be given to make as the -j argument. + + ''' + + n = cpu_count() if cores is None else cores + # Experimental results (ref. Kinnison) says a factor of 1.5 + # gives about the optimal result for build times, since much of + # builds are I/O bound, not CPU bound. + return max(int(n * 1.5 + 0.5), 1) + + +def create_cachedir(settings): # pragma: no cover + '''Return cache directory, creating it if necessary.''' + + cachedir = settings['cachedir'] + if not os.path.exists(cachedir): + os.mkdir(cachedir) + return cachedir + + +def get_artifact_cache_server(settings): # pragma: no cover + if settings['artifact-cache-server']: + return settings['artifact-cache-server'] + if settings['cache-server']: + return settings['cache-server'] + return None + + +def get_git_resolve_cache_server(settings): # pragma: no cover + if settings['git-resolve-cache-server']: + return settings['git-resolve-cache-server'] + if settings['cache-server']: + return settings['cache-server'] + return None + + +def new_artifact_caches(settings): # pragma: no cover + '''Create new objects for local and remote artifact caches. + + This includes creating the directories on disk, if missing. + + ''' + + cachedir = create_cachedir(settings) + artifact_cachedir = os.path.join(cachedir, 'artifacts') + if not os.path.exists(artifact_cachedir): + os.mkdir(artifact_cachedir) + + lac = morphlib.localartifactcache.LocalArtifactCache( + fs.osfs.OSFS(artifact_cachedir)) + + rac_url = get_artifact_cache_server(settings) + rac = None + if rac_url: + rac = morphlib.remoteartifactcache.RemoteArtifactCache(rac_url) + return lac, rac + + +def combine_aliases(app): # pragma: no cover + '''Create a full repo-alias set from the app's settings.''' + trove_host = app.settings['trove-host'] + trove_ids = app.settings['trove-id'] + repo_aliases = app.settings['repo-alias'] + repo_pat = r'^(?P<prefix>[a-z][a-z0-9-]+)=(?P<pull>[^#]+)#(?P<push>[^#]+)$' + trove_pat = (r'^(?P<prefix>[a-z][a-z0-9-]+)=(?P<path>[^#]+)#' + '(?P<pull>[^#]+)#(?P<push>[^#]+)$') + alias_map = {} + def _expand(protocol, path): + if protocol == "git": + return "git://%s/%s/%%s" % (trove_host, path) + elif protocol == "ssh": + return "ssh://git@%s/%s/%%s" % (trove_host, path) + else: + raise morphlib.Error( + 'Unknown protocol in trove_id: %s' % protocol) + + if trove_host: + alias_map['baserock'] = "baserock=%s#%s" % ( + _expand('git', 'baserock'), + _expand('ssh', 'baserock')) + alias_map['upstream'] = "upstream=%s#%s" % ( + _expand('git', 'delta'), + _expand('ssh', 'delta')) + for trove_id in trove_ids: + m = re.match(trove_pat, trove_id) + if m: + alias_map[m.group('prefix')] = "%s=%s#%s" % ( + m.group('prefix'), + _expand(m.group('pull'), m.group('path')), + _expand(m.group('push'), m.group('path'))) + elif '=' not in trove_id: + alias_map[trove_id] = "%s=%s#%s" % ( + trove_id, + _expand('ssh', trove_id), + _expand('ssh', trove_id)) + for repo_alias in repo_aliases: + m = re.match(repo_pat, repo_alias) + if m: + alias_map[m.group('prefix')] = repo_alias + else: + raise morphlib.Error( + 'Invalid repo-alias: %s' % repo_alias) + + + return alias_map.values() + +def new_repo_caches(app): # pragma: no cover + '''Create new objects for local, remote git repository caches.''' + + aliases = app.settings['repo-alias'] + cachedir = create_cachedir(app.settings) + gits_dir = os.path.join(cachedir, 'gits') + tarball_base_url = app.settings['tarball-server'] + repo_resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases) + lrc = morphlib.localrepocache.LocalRepoCache( + app, gits_dir, repo_resolver, tarball_base_url=tarball_base_url) + + url = get_git_resolve_cache_server(app.settings) + if url: + rrc = morphlib.remoterepocache.RemoteRepoCache(url, repo_resolver) + else: + rrc = None + + return lrc, rrc + +def env_variable_is_password(key): # pragma: no cover + return 'PASSWORD' in key + +@contextlib.contextmanager +def hide_password_environment_variables(env): # pragma: no cover + is_password = env_variable_is_password + password_env = { k:v for k,v in env.iteritems() if is_password(k) } + for k in password_env: + env[k] = '(value hidden)' + yield + for k, v in password_env.iteritems(): + env[k] = v + +def log_environment_changes(app, current_env, previous_env): # pragma: no cover + '''Log the differences between two environments to debug log.''' + def log_event(key, value, event): + if env_variable_is_password(key): + value_msg = '(value hidden)' + else: + value_msg = '= "%s"' % value + app.status(msg='%(event)s environment variable %(key)s %(value)s', + event=event, key=key, value=value_msg, chatty=True) + + for key in current_env.keys(): + if key not in previous_env: + log_event(key, current_env[key], 'new') + elif current_env[key] != previous_env[key]: + log_event(key, current_env[key], 'changed') + for key in previous_env.keys(): + if key not in current_env: + log_event(key, previous_env[key], 'unset') + +# This acquired from rdiff-backup which is GPLv2+ and a patch from 2011 +# which has not yet been merged, combined with a tad of tidying from us. +def copyfileobj(inputfp, outputfp, blocksize=1024*1024): # pragma: no cover + """Copies file inputfp to outputfp in blocksize intervals""" + + sparse = False + buf = None + while 1: + inbuf = inputfp.read(blocksize) + if not inbuf: break + if not buf: + buf = inbuf + else: + buf += inbuf + + # Combine "short" reads + if (len(buf) < blocksize): + continue + + buflen = len(buf) + if buf == "\x00" * buflen: + outputfp.seek(buflen, os.SEEK_CUR) + buf = None + # flag sparse=True, that we seek()ed, but have not written yet + # The filesize is wrong until we write + sparse = True + else: + outputfp.write(buf) + buf = None + # We wrote, so clear sparse. + sparse = False + + if buf: + outputfp.write(buf) + elif sparse: + outputfp.seek(-1, os.SEEK_CUR) + outputfp.write("\x00") + +def get_bytes_free_in_path(path): # pragma: no cover + """Returns the bytes free in the filesystem that path is part of""" + + fsinfo = os.statvfs(path) + return fsinfo.f_bavail * fsinfo.f_bsize + +def on_same_filesystem(path_a, path_b): # pragma: no cover + """Tests whether both paths are on the same fileystem + + Note behaviour may be unexpected on btrfs, since subvolumes + appear to be on a different device, but share a storage pool. + + """ + # TODO: return true if one path is a subvolume of the other on btrfs? + return os.stat(path_a).st_dev == os.stat(path_b).st_dev + +def unify_space_requirements(tmp_path, tmp_min_size, + cache_path, cache_min_size): # pragma: no cover + """Adjust minimum sizes when paths share a disk. + + Given pairs of path and minimum size, return the minimum sizes such + that when the paths are on the same disk, the sizes are added together. + + """ + # TODO: make this work for variable number of (path, size) pairs as needed + # hint: try list.sort and itertools.groupby + if not on_same_filesystem(tmp_path, cache_path): + return tmp_min_size, cache_min_size + unified_size = tmp_min_size + cache_min_size + return unified_size, unified_size + +def check_disk_available(tmp_path, tmp_min_size, + cache_path, cache_min_size): # pragma: no cover + # if both are on the same filesystem, assume they share a storage pool, + # so the sum of the two sizes needs to be available + # TODO: if we need to do this on any more than 2 paths + # extend it to take a [(path, min)] + tmp_min_size, cache_min_size = unify_space_requirements( + tmp_path, tmp_min_size, cache_path, cache_min_size) + tmp_size, cache_size = map(get_bytes_free_in_path, (tmp_path, cache_path)) + errors = [] + for path, min in [(tmp_path, tmp_min_size), (cache_path, cache_min_size)]: + free = get_bytes_free_in_path(path) + if free < min: + errors.append('\t%(path)s requires %(min)d bytes free, ' + 'has %(free)d' % locals()) + if not errors: + return + raise morphlib.Error('Insufficient space on disk:\n' + + '\n'.join(errors) + '\n' + 'Please run `morph gc`. If the problem persists ' + 'increase the disk size, manually clean up some ' + 'space or reduce the disk space required by the ' + 'tempdir-min-space and cachedir-min-space ' + 'configuration options.') + + + + +def find_root(dirname, subdir_name): + '''Find parent of a directory, at or above a given directory. + + The sought-after directory is indicated by the existence of a + subdirectory of the indicated name. For example, dirname might + be the current working directory of the process, and subdir_name + might be ".morph"; then the returned value would be the Morph + workspace root directory, which has a subdirectory called + ".morph". + + Return path to desired directory, or None if not found. + + ''' + + dirname = os.path.normpath(os.path.abspath(dirname)) + while not os.path.isdir(os.path.join(dirname, subdir_name)): + if dirname == '/': + return None + dirname = os.path.dirname(dirname) + return dirname + + +def find_leaves(search_dir, subdir_name): + '''This is like find_root, except it looks towards leaves. + + The directory tree, starting at search_dir is traversed. + + If a directory has a subdirectory called subdir_name, then + the directory is returned. + + It does not recurse into a leaf's subdirectories. + + ''' + + for dirname, subdirs, filenames in os.walk(search_dir): + if subdir_name in subdirs: + del subdirs[:] + yield dirname + + +def find_leaf(dirname, subdir_name): + '''This is like find_root, except it looks towards leaves. + + If there are no subdirectories, or more than one, fail. + + ''' + + leaves = list(find_leaves(dirname, subdir_name)) + if len(leaves) == 1: + return leaves[0] + return None + + +class EnvironmentAlreadySetError(morphlib.Error): + + def __init__(self, conflicts): + self.conflicts = conflicts + morphlib.Error.__init__( + self, 'Keys %r are already set in the environment' % conflicts) + + +def parse_environment_pairs(env, pairs): + '''Add key=value pairs to the environment dict. + + Given a dict and a list of strings of the form key=value, + set dict[key] = value, unless key is already set in the + environment, at which point raise an exception. + + This does not modify the passed in dict. + + Returns the extended dict. + + ''' + + extra_env = dict(p.split('=', 1) for p in pairs) + conflicting = [k for k in extra_env if k in env] + if conflicting: + raise EnvironmentAlreadySetError(conflicting) + + # Return a dict that is the union of the two + # This is not the most performant, since it creates + # 3 unnecessary lists, but I felt this was the most + # easy to read. Using itertools.chain may be more efficicent + return dict(env.items() + extra_env.items()) + + +def has_hardware_fp(): # pragma: no cover + ''' + This function returns whether the binary /proc/self/exe is compiled + with hardfp _not_ whether the platform is hardfp. + + We expect the binaries on our build platform to be compiled with + hardfp. + + This is not ideal but at the time of writing this is the only + reliable way to decide whether our architecture is a hardfp + architecture. + ''' + + output = subprocess.check_output(['readelf', '-A', '/proc/self/exe']) + return 'Tag_ABI_VFP_args: VFP registers' in output + + +def get_host_architecture(): # pragma: no cover + '''Get the canonical Morph name for the host's architecture.''' + + machine = os.uname()[-1] + + table = { + 'x86_64': 'x86_64', + 'i386': 'x86_32', + 'i486': 'x86_32', + 'i586': 'x86_32', + 'i686': 'x86_32', + 'armv7l': 'armv7l', + 'armv7b': 'armv7b', + 'ppc64': 'ppc64' + } + + if machine not in table: + raise morphlib.Error('Unknown host architecture %s' % machine) + + if machine == 'armv7l' and has_hardware_fp(): + return 'armv7lhf' + + return table[machine] + + +def sanitize_environment(env): + for k in env: + env[k] = str(env[k]) + +def iter_trickle(iterable, limit): + '''Split an iterable up into `limit` length chunks.''' + it = iter(iterable) + while True: + buf = list(itertools.islice(it, limit)) + if len(buf) == 0: + break + yield buf + + +def get_data_path(relative_path): # pragma: no cover + '''Return path to a data file in the morphlib Python package. + + ``relative_path`` is the name of the data file, relative to the + location in morphlib where the data files are. + + ''' + + morphlib_dir = os.path.dirname(morphlib.__file__) + return os.path.join(morphlib_dir, relative_path) + + +def get_data(relative_path): # pragma: no cover + '''Return contents of a data file from the morphlib Python package. + + ``relative_path`` is the name of the data file, relative to the + location in morphlib where the data files are. + + ''' + + with open(get_data_path(relative_path)) as f: + return f.read() diff --git a/morphlib/util_tests.py b/morphlib/util_tests.py new file mode 100644 index 00000000..715892b6 --- /dev/null +++ b/morphlib/util_tests.py @@ -0,0 +1,138 @@ +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import os +import shutil +import tempfile +import unittest + +import morphlib + + +class IndentTests(unittest.TestCase): + + def test_returns_empty_string_for_empty_string(self): + self.assertEqual(morphlib.util.indent(''), '') + + def test_indents_single_line(self): + self.assertEqual(morphlib.util.indent('foo'), ' foo') + + def test_obeys_spaces_setting(self): + self.assertEqual(morphlib.util.indent('foo', spaces=2), ' foo') + + def test_indents_multiple_lines(self): + self.assertEqual(morphlib.util.indent('foo\nbar\n'), + ' foo\n bar') + + +class SanitiseMorphologyPathTests(unittest.TestCase): + + def test_appends_morph_to_string(self): + self.assertEqual(morphlib.util.sanitise_morphology_path('a'), + 'a.morph') + + def test_returns_morph_when_given_a_filename(self): + self.assertEqual(morphlib.util.sanitise_morphology_path('a.morph'), + 'a.morph') + + def test_returns_morph_when_given_a_path(self): + self.assertEqual('stratum/a.morph', + morphlib.util.sanitise_morphology_path('stratum/a.morph')) + + +class MakeConcurrencyTests(unittest.TestCase): + + def test_returns_2_for_1_core(self): + self.assertEqual(morphlib.util.make_concurrency(cores=1), 2) + + def test_returns_3_for_2_cores(self): + self.assertEqual(morphlib.util.make_concurrency(cores=2), 3) + + def test_returns_5_for_3_cores(self): + self.assertEqual(morphlib.util.make_concurrency(cores=3), 5) + + def test_returns_6_for_4_cores(self): + self.assertEqual(morphlib.util.make_concurrency(cores=4), 6) + + +class FindParentOfTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + os.makedirs(os.path.join(self.tempdir, 'a', 'b', 'c')) + self.a = os.path.join(self.tempdir, 'a') + self.b = os.path.join(self.tempdir, 'a', 'b') + self.c = os.path.join(self.tempdir, 'a', 'b', 'c') + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def test_find_root_finds_starting_directory(self): + os.mkdir(os.path.join(self.a, '.magic')) + self.assertEqual(morphlib.util.find_root(self.a, '.magic'), self.a) + + def test_find_root_finds_ancestor(self): + os.mkdir(os.path.join(self.a, '.magic')) + self.assertEqual(morphlib.util.find_root(self.c, '.magic'), self.a) + + def test_find_root_returns_none_if_not_found(self): + self.assertEqual(morphlib.util.find_root(self.c, '.magic'), None) + + def test_find_leaf_finds_starting_directory(self): + os.mkdir(os.path.join(self.a, '.magic')) + self.assertEqual(morphlib.util.find_leaf(self.a, '.magic'), self.a) + + def test_find_leaf_finds_child(self): + os.mkdir(os.path.join(self.c, '.magic')) + self.assertEqual(morphlib.util.find_leaf(self.a, '.magic'), self.c) + + def test_find_leaf_returns_none_if_not_found(self): + self.assertEqual(morphlib.util.find_leaf(self.a, '.magic'), None) + + +class ParseEnvironmentPairsTests(unittest.TestCase): + + def test_parse_environment_pairs_adds_key(self): + ret = morphlib.util.parse_environment_pairs({}, ["foo=bar"]) + self.assertEqual(ret.get("foo"), "bar") + + def test_parse_environment_does_not_alter_passed_dict(self): + d = {} + morphlib.util.parse_environment_pairs(d, ["foo=bar"]) + self.assertTrue("foo" not in d) + + def test_parse_environment_raises_on_duplicates(self): + self.assertRaises( + morphlib.util.EnvironmentAlreadySetError, + morphlib.util.parse_environment_pairs, + {"foo": "bar"}, + ["foo=bar"]) + + def test_sanitize_environment(self): + d = { 'a': 1 } + morphlib.util.sanitize_environment(d) + self.assertTrue(isinstance(d['a'], str)) + +class IterTrickleTests(unittest.TestCase): + + def test_splits(self): + self.assertEqual(list(morphlib.util.iter_trickle("foobarbazqux", 3)), + [["f", "o", "o"], ["b", "a", "r"], + ["b", "a", "z"], ["q", "u", "x"]]) + + def test_truncated_final_sequence(self): + self.assertEqual(list(morphlib.util.iter_trickle("barquux", 3)), + [["b", "a", "r"], ["q", "u", "u"], ["x"]]) diff --git a/morphlib/workspace.py b/morphlib/workspace.py new file mode 100644 index 00000000..27ccbe65 --- /dev/null +++ b/morphlib/workspace.py @@ -0,0 +1,146 @@ +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +'''A module to create, query, and manipulate Morph workspaces.''' + + +import os + +import morphlib + + +class WorkspaceDirExists(morphlib.Error): + + def __init__(self, dirname): + self.msg = ( + 'can only initialize empty directory as a workspace: %s' % + dirname) + + +class NotInWorkspace(morphlib.Error): + + def __init__(self, dirname): + self.msg = ( + "Can't find the workspace directory.\n" + "Morph must be built and deployed within the " + "system branch checkout within the workspace directory.") + + +class Workspace(object): + + '''A Morph workspace. + + This class should be instantiated with the open() or create() + functions in this module. + + ''' + + def __init__(self, root_directory): + self.root = root_directory + + def get_default_system_branch_directory_name(self, system_branch_name): + '''Determine directory where a system branch would be checked out. + + Return the fully qualified pathname to the directory where + a system branch would be checked out. The directory may or may + not exist already. + + If the system branch is checked out, but into a directory of + a different name (which is allowed), that is ignored: this method + only computed the default name. + + ''' + + return os.path.join(self.root, system_branch_name) + + def create_system_branch_directory(self, + root_repository_url, system_branch_name): + '''Create a directory for a system branch. + + Return a SystemBranchDirectory object that represents the + directory. The directory must not already exist. The directory + gets created and initialised (the .morph-system-branch/config + file gets created and populated). The root repository of the + system branch does NOT get checked out, the caller needs to + do that. + + ''' + + dirname = self.get_default_system_branch_directory_name( + system_branch_name) + sb = morphlib.sysbranchdir.create( + dirname, root_repository_url, system_branch_name) + return sb + + def list_system_branches(self): + return (morphlib.sysbranchdir.open(dirname) + for dirname in + morphlib.util.find_leaves(self.root, '.morph-system-branch')) + + +def open(dirname): + '''Open an existing workspace. + + The given directory name may be to a subdirectory of the + workspace. This makes it easy to instantiate the Workspace + class even when the user invokes Morph in a subdirectory. + The workspace MUST exist already, or NotInWorkspace is + raised. + + Return a Workspace instance. + + ''' + + root = _find_root(dirname) + if root is None: + raise NotInWorkspace(dirname) + return Workspace(root) + + +def create(dirname): + '''Create a new workspace. + + The given directory must not be inside an existing workspace. + The workspace directory is created, unless it already exists. If it + does exist, it must be empty. Otherwise WorkspaceDirExists is raised. + + ''' + + root = _find_root(dirname) + if root is not None: + raise WorkspaceDirExists(root) + + if os.path.exists(dirname): + if os.listdir(dirname): + raise WorkspaceDirExists(dirname) + else: + os.mkdir(dirname) + os.mkdir(os.path.join(dirname, '.morph')) + return Workspace(dirname) + + +def _find_root(dirname): + '''Find the workspace root directory at or above a given directory.''' + + dirname = os.path.normpath(os.path.abspath(dirname)) + while not os.path.isdir(os.path.join(dirname, '.morph')): + if dirname == '/': + return None + dirname = os.path.dirname(dirname) + return dirname + diff --git a/morphlib/workspace_tests.py b/morphlib/workspace_tests.py new file mode 100644 index 00000000..9eef1053 --- /dev/null +++ b/morphlib/workspace_tests.py @@ -0,0 +1,111 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import os +import shutil +import tempfile +import unittest + +import morphlib + + +class WorkspaceTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.workspace_dir = os.path.join(self.tempdir, 'workspace') + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def assertIsWorkspace(self, dirname): + self.assertTrue(os.path.isdir(dirname)) + self.assertTrue(os.path.isdir(os.path.join(dirname, '.morph'))) + + def create_it(self): + morphlib.workspace.create(self.workspace_dir) + + def test_creates_workspace(self): + ws = morphlib.workspace.create(self.workspace_dir) + self.assertIsWorkspace(self.workspace_dir) + self.assertEqual(ws.root, self.workspace_dir) + + def test_create_initialises_existing_but_empty_directory(self): + os.mkdir(self.workspace_dir) + ws = morphlib.workspace.create(self.workspace_dir) + self.assertIsWorkspace(self.workspace_dir) + self.assertEqual(ws.root, self.workspace_dir) + + def test_fails_to_create_workspace_when_dir_exists_and_is_not_empty(self): + os.mkdir(self.workspace_dir) + os.mkdir(os.path.join(self.workspace_dir, 'somedir')) + self.assertRaises( + morphlib.workspace.WorkspaceDirExists, + morphlib.workspace.create, self.workspace_dir) + + def test_fails_to_recreate_workspace(self): + # Create it once. + morphlib.workspace.create(self.workspace_dir) + # Creating it again must fail. + self.assertRaises( + morphlib.workspace.WorkspaceDirExists, + morphlib.workspace.create, self.workspace_dir) + + def test_opens_workspace_when_given_its_root(self): + self.create_it() + ws = morphlib.workspace.open(self.workspace_dir) + self.assertEqual(ws.root, self.workspace_dir) + + def test_opens_workspace_when_given_subdirectory(self): + self.create_it() + subdir = os.path.join(self.workspace_dir, 'subdir') + os.mkdir(subdir) + ws = morphlib.workspace.open(subdir) + self.assertEqual(ws.root, self.workspace_dir) + + def test_fails_to_open_workspace_when_no_workspace_anywhere(self): + self.assertRaises( + morphlib.workspace.NotInWorkspace, + morphlib.workspace.open, self.tempdir) + + def test_invents_appropriate_name_for_system_branch_directory(self): + self.create_it() + ws = morphlib.workspace.open(self.workspace_dir) + branch = 'foo/bar' + self.assertEqual( + ws.get_default_system_branch_directory_name(branch), + os.path.join(self.workspace_dir, branch)) + + def test_creates_system_branch_directory(self): + self.create_it() + ws = morphlib.workspace.open(self.workspace_dir) + url = 'test:morphs' + branch = 'my/new/thing' + sb = ws.create_system_branch_directory(url, branch) + self.assertEqual(type(sb), morphlib.sysbranchdir.SystemBranchDirectory) + + def test_lists_created_system_branches(self): + self.create_it() + ws = morphlib.workspace.open(self.workspace_dir) + + branches = ["branch/1", "branch/2"] + for branch in branches: + ws.create_system_branch_directory('test:morphs', branch) + self.assertEqual(sorted(sb.system_branch_name + for sb in ws.list_system_branches()), + branches) diff --git a/morphlib/writeexts.py b/morphlib/writeexts.py new file mode 100644 index 00000000..0fd0ad7b --- /dev/null +++ b/morphlib/writeexts.py @@ -0,0 +1,574 @@ +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import logging +import os +import re +import shutil +import sys +import time +import tempfile + +import morphlib + + +class Fstab(object): + '''Small helper class for parsing and adding lines to /etc/fstab.''' + + # There is an existing Python helper library for editing of /etc/fstab. + # However it is unmaintained and has an incompatible license (GPL3). + # + # https://code.launchpad.net/~computer-janitor-hackers/python-fstab/trunk + + def __init__(self, filepath='/etc/fstab'): + if os.path.exists(filepath): + with open(filepath, 'r') as f: + self.text= f.read() + else: + self.text = '' + self.filepath = filepath + self.lines_added = 0 + + def get_mounts(self): + '''Return list of mount devices and targets in /etc/fstab. + + Return value is a dict of target -> device. + ''' + mounts = dict() + for line in self.text.splitlines(): + words = line.split() + if len(words) >= 2 and not words[0].startswith('#'): + device, target = words[0:2] + mounts[target] = device + return mounts + + def add_line(self, line): + '''Add a new entry to /etc/fstab. + + Lines are appended, and separated from any entries made by configure + extensions with a comment. + + ''' + if self.lines_added == 0: + if len(self.text) == 0 or self.text[-1] is not '\n': + self.text += '\n' + self.text += '# Morph default system layout\n' + self.lines_added += 1 + + self.text += line + '\n' + + def write(self): + '''Rewrite the fstab file to include all new entries.''' + with morphlib.savefile.SaveFile(self.filepath, 'w') as f: + f.write(self.text) + + +class WriteExtension(cliapp.Application): + + '''A base class for deployment write extensions. + + A subclass should subclass this class, and add a + ``process_args`` method. + + Note that it is not necessary to subclass this class for write + extensions. This class is here just to collect common code for + write extensions. + + ''' + + def setup_logging(self): + '''Direct all logging output to MORPH_LOG_FD, if set. + + This file descriptor is read by Morph and written into its own log + file. + + This overrides cliapp's usual configurable logging setup. + + ''' + log_write_fd = int(os.environ.get('MORPH_LOG_FD', 0)) + + if log_write_fd == 0: + return + + formatter = logging.Formatter('%(message)s') + + handler = logging.StreamHandler(os.fdopen(log_write_fd, 'w')) + handler.setFormatter(formatter) + + logger = logging.getLogger() + logger.addHandler(handler) + logger.setLevel(logging.DEBUG) + + def log_config(self): + with morphlib.util.hide_password_environment_variables(os.environ): + cliapp.Application.log_config(self) + + def process_args(self, args): + raise NotImplementedError() + + def status(self, **kwargs): + '''Provide status output. + + The ``msg`` keyword argument is the actual message, + the rest are values for fields in the message as interpolated + by %. + + ''' + + self.output.write('%s\n' % (kwargs['msg'] % kwargs)) + self.output.flush() + + def check_for_btrfs_in_deployment_host_kernel(self): + with open('/proc/filesystems') as f: + text = f.read() + return '\tbtrfs\n' in text + + def require_btrfs_in_deployment_host_kernel(self): + if not self.check_for_btrfs_in_deployment_host_kernel(): + raise cliapp.AppException( + 'Error: Btrfs is required for this deployment, but was not ' + 'detected in the kernel of the machine that is running Morph.') + + def create_local_system(self, temp_root, raw_disk): + '''Create a raw system image locally.''' + size = self.get_disk_size() + if not size: + raise cliapp.AppException('DISK_SIZE is not defined') + self.create_raw_disk_image(raw_disk, size) + try: + self.mkfs_btrfs(raw_disk) + mp = self.mount(raw_disk) + except BaseException: + sys.stderr.write('Error creating disk image') + os.remove(raw_disk) + raise + try: + self.create_btrfs_system_layout( + temp_root, mp, version_label='factory', + disk_uuid=self.get_uuid(raw_disk)) + except BaseException, e: + sys.stderr.write('Error creating Btrfs system layout') + self.unmount(mp) + os.remove(raw_disk) + raise + else: + self.unmount(mp) + + def _parse_size(self, size): + '''Parse a size from a string. + + Return size in bytes. + + ''' + + m = re.match('^(\d+)([kmgKMG]?)$', size) + if not m: + return None + + factors = { + '': 1, + 'k': 1024, + 'm': 1024**2, + 'g': 1024**3, + } + factor = factors[m.group(2).lower()] + + return int(m.group(1)) * factor + + def _parse_size_from_environment(self, env_var, default): + '''Parse a size from an environment variable.''' + + size = os.environ.get(env_var, default) + if size is None: + return None + bytes = self._parse_size(size) + if bytes is None: + raise morphlib.Error('Cannot parse %s value %s' % (env_var, size)) + return bytes + + def get_disk_size(self): + '''Parse disk size from environment.''' + return self._parse_size_from_environment('DISK_SIZE', None) + + def get_ram_size(self): + '''Parse RAM size from environment.''' + return self._parse_size_from_environment('RAM_SIZE', '1G') + + def get_vcpu_count(self): + '''Parse the virtual cpu count from environment.''' + return self._parse_size_from_environment('VCPUS', '1') + + def create_raw_disk_image(self, filename, size): + '''Create a raw disk image.''' + + self.status(msg='Creating empty disk image') + with open(filename, 'wb') as f: + if size > 0: + f.seek(size-1) + f.write('\0') + + def mkfs_btrfs(self, location): + '''Create a btrfs filesystem on the disk.''' + self.status(msg='Creating btrfs filesystem') + cliapp.runcmd(['mkfs.btrfs', '-L', 'baserock', location]) + + def get_uuid(self, location): + '''Get the UUID of a block device's file system.''' + # Requires util-linux blkid; busybox one ignores options and + # lies by exiting successfully. + return cliapp.runcmd(['blkid', '-s', 'UUID', '-o', 'value', + location]).strip() + + def mount(self, location): + '''Mount the filesystem so it can be tweaked. + + Return path to the mount point. + The mount point is a newly created temporary directory. + The caller must call self.unmount to unmount on the return value. + + ''' + + self.status(msg='Mounting filesystem') + tempdir = tempfile.mkdtemp() + cliapp.runcmd(['mount', '-o', 'loop', location, tempdir]) + return tempdir + + def unmount(self, mount_point): + '''Unmount the filesystem mounted by self.mount. + + Also, remove the temporary directory. + + ''' + + self.status(msg='Unmounting filesystem') + cliapp.runcmd(['umount', mount_point]) + os.rmdir(mount_point) + + def create_btrfs_system_layout(self, temp_root, mountpoint, version_label, + disk_uuid): + '''Separate base OS versions from state using subvolumes. + + ''' + initramfs = self.find_initramfs(temp_root) + version_root = os.path.join(mountpoint, 'systems', version_label) + state_root = os.path.join(mountpoint, 'state') + + os.makedirs(version_root) + os.makedirs(state_root) + + self.create_orig(version_root, temp_root) + system_dir = os.path.join(version_root, 'orig') + + state_dirs = self.complete_fstab_for_btrfs_layout(system_dir, + disk_uuid) + + for state_dir in state_dirs: + self.create_state_subvolume(system_dir, mountpoint, state_dir) + + self.create_run(version_root) + + os.symlink( + version_label, os.path.join(mountpoint, 'systems', 'default')) + + if self.bootloader_config_is_wanted(): + self.install_kernel(version_root, temp_root) + if self.get_dtb_path() != '': + self.install_dtb(version_root, temp_root) + self.install_syslinux_menu(mountpoint, version_root) + if initramfs is not None: + self.install_initramfs(initramfs, version_root) + self.generate_bootloader_config(mountpoint, disk_uuid) + else: + self.generate_bootloader_config(mountpoint) + self.install_bootloader(mountpoint) + + def create_orig(self, version_root, temp_root): + '''Create the default "factory" system.''' + + orig = os.path.join(version_root, 'orig') + + self.status(msg='Creating orig subvolume') + cliapp.runcmd(['btrfs', 'subvolume', 'create', orig]) + self.status(msg='Copying files to orig subvolume') + cliapp.runcmd(['cp', '-a', temp_root + '/.', orig + '/.']) + + def create_run(self, version_root): + '''Create the 'run' snapshot.''' + + self.status(msg='Creating run subvolume') + orig = os.path.join(version_root, 'orig') + run = os.path.join(version_root, 'run') + cliapp.runcmd( + ['btrfs', 'subvolume', 'snapshot', orig, run]) + + def create_state_subvolume(self, system_dir, mountpoint, state_subdir): + '''Create a shared state subvolume. + + We need to move any files added to the temporary rootfs by the + configure extensions to their correct home. For example, they might + have added keys in `/root/.ssh` which we now need to transfer to + `/state/root/.ssh`. + + ''' + self.status(msg='Creating %s subvolume' % state_subdir) + subvolume = os.path.join(mountpoint, 'state', state_subdir) + cliapp.runcmd(['btrfs', 'subvolume', 'create', subvolume]) + os.chmod(subvolume, 0755) + + existing_state_dir = os.path.join(system_dir, state_subdir) + files = [] + if os.path.exists(existing_state_dir): + files = os.listdir(existing_state_dir) + if len(files) > 0: + self.status(msg='Moving existing data to %s subvolume' % subvolume) + for filename in files: + filepath = os.path.join(existing_state_dir, filename) + cliapp.runcmd(['mv', filepath, subvolume]) + + def complete_fstab_for_btrfs_layout(self, system_dir, rootfs_uuid=None): + '''Fill in /etc/fstab entries for the default Btrfs disk layout. + + In the future we should move this code out of the write extension and + in to a configure extension. To do that, though, we need some way of + informing the configure extension what layout should be used. Right now + a configure extension doesn't know if the system is going to end up as + a Btrfs disk image, a tarfile or something else and so it can't come + up with a sensible default fstab. + + Configuration extensions can already create any /etc/fstab that they + like. This function only fills in entries that are missing, so if for + example the user configured /home to be on a separate partition, that + decision will be honoured and /state/home will not be created. + + ''' + shared_state_dirs = {'home', 'root', 'opt', 'srv', 'var'} + + fstab = Fstab(os.path.join(system_dir, 'etc', 'fstab')) + existing_mounts = fstab.get_mounts() + + if '/' in existing_mounts: + root_device = existing_mounts['/'] + else: + root_device = (self.get_root_device() if rootfs_uuid is None else + 'UUID=%s' % rootfs_uuid) + fstab.add_line('%s / btrfs defaults,rw,noatime 0 1' % root_device) + + state_dirs_to_create = set() + for state_dir in shared_state_dirs: + if '/' + state_dir not in existing_mounts: + state_dirs_to_create.add(state_dir) + state_subvol = os.path.join('/state', state_dir) + fstab.add_line( + '%s /%s btrfs subvol=%s,defaults,rw,noatime 0 2' % + (root_device, state_dir, state_subvol)) + + fstab.write() + return state_dirs_to_create + + def find_initramfs(self, temp_root): + '''Check whether the rootfs has an initramfs. + + Uses the INITRAMFS_PATH option to locate it. + ''' + if 'INITRAMFS_PATH' in os.environ: + initramfs = os.path.join(temp_root, os.environ['INITRAMFS_PATH']) + if not os.path.exists(initramfs): + raise morphlib.Error('INITRAMFS_PATH specified, ' + 'but file does not exist') + return initramfs + return None + + def install_initramfs(self, initramfs_path, version_root): + '''Install the initramfs outside of 'orig' or 'run' subvolumes. + + This is required because syslinux doesn't traverse subvolumes when + loading the kernel or initramfs. + ''' + self.status(msg='Installing initramfs') + initramfs_dest = os.path.join(version_root, 'initramfs') + cliapp.runcmd(['cp', '-a', initramfs_path, initramfs_dest]) + + def install_kernel(self, version_root, temp_root): + '''Install the kernel outside of 'orig' or 'run' subvolumes''' + + self.status(msg='Installing kernel') + image_names = ['vmlinuz', 'zImage', 'uImage'] + kernel_dest = os.path.join(version_root, 'kernel') + for name in image_names: + try_path = os.path.join(temp_root, 'boot', name) + if os.path.exists(try_path): + cliapp.runcmd(['cp', '-a', try_path, kernel_dest]) + break + + def install_dtb(self, version_root, temp_root): + '''Install the device tree outside of 'orig' or 'run' subvolumes''' + + self.status(msg='Installing devicetree') + device_tree_path = self.get_dtb_path() + dtb_dest = os.path.join(version_root, 'dtb') + try_path = os.path.join(temp_root, device_tree_path) + if os.path.exists(try_path): + cliapp.runcmd(['cp', '-a', try_path, dtb_dest]) + else: + logging.error("Failed to find device tree %s", device_tree_path) + raise cliapp.AppException( + 'Failed to find device tree %s' % device_tree_path) + + def get_dtb_path(self): + return os.environ.get('DTB_PATH', '') + + def get_bootloader_install(self): + # Do we actually want to install the bootloader? + # Set this to "none" to prevent the install + return os.environ.get('BOOTLOADER_INSTALL', 'extlinux') + + def get_bootloader_config_format(self): + # The config format for the bootloader, + # if not set we default to extlinux for x86 + return os.environ.get('BOOTLOADER_CONFIG_FORMAT', 'extlinux') + + def get_extra_kernel_args(self): + return os.environ.get('KERNEL_ARGS', '') + + def get_root_device(self): + return os.environ.get('ROOT_DEVICE', '/dev/sda') + + def generate_bootloader_config(self, real_root, disk_uuid=None): + '''Install extlinux on the newly created disk image.''' + config_function_dict = { + 'extlinux': self.generate_extlinux_config, + } + + config_type = self.get_bootloader_config_format() + if config_type in config_function_dict: + config_function_dict[config_type](real_root, disk_uuid) + else: + raise cliapp.AppException( + 'Invalid BOOTLOADER_CONFIG_FORMAT %s' % config_type) + + def generate_extlinux_config(self, real_root, disk_uuid=None): + '''Install extlinux on the newly created disk image.''' + + self.status(msg='Creating extlinux.conf') + config = os.path.join(real_root, 'extlinux.conf') + kernel_args = ( + 'rw ' # ro ought to work, but we don't test that regularly + 'init=/sbin/init ' # default, but it doesn't hurt to be explicit + 'rootfstype=btrfs ' # required when using initramfs, also boots + # faster when specified without initramfs + 'rootflags=subvol=systems/default/run ') # boot runtime subvol + kernel_args += 'root=%s ' % (self.get_root_device() + if disk_uuid is None + else 'UUID=%s' % disk_uuid) + kernel_args += self.get_extra_kernel_args() + with open(config, 'w') as f: + f.write('default linux\n') + f.write('timeout 1\n') + f.write('label linux\n') + f.write('kernel /systems/default/kernel\n') + if disk_uuid is not None: + f.write('initrd /systems/default/initramfs\n') + if self.get_dtb_path() != '': + f.write('devicetree /systems/default/dtb\n') + f.write('append %s\n' % kernel_args) + + def install_bootloader(self, real_root): + install_function_dict = { + 'extlinux': self.install_bootloader_extlinux, + } + + install_type = self.get_bootloader_install() + if install_type in install_function_dict: + install_function_dict[install_type](real_root) + elif install_type != 'none': + raise cliapp.AppException( + 'Invalid BOOTLOADER_INSTALL %s' % install_type) + + def install_bootloader_extlinux(self, real_root): + self.status(msg='Installing extlinux') + cliapp.runcmd(['extlinux', '--install', real_root]) + + # FIXME this hack seems to be necessary to let extlinux finish + cliapp.runcmd(['sync']) + time.sleep(2) + + def install_syslinux_menu(self, real_root, version_root): + '''Make syslinux/extlinux menu binary available. + + The syslinux boot menu is compiled to a file named menu.c32. Extlinux + searches a few places for this file but it does not know to look inside + our subvolume, so we copy it to the filesystem root. + + If the file is not available, the bootloader will still work but will + not be able to show a menu. + + ''' + menu_file = os.path.join(version_root, 'orig', + 'usr', 'share', 'syslinux', 'menu.c32') + if os.path.isfile(menu_file): + self.status(msg='Copying menu.c32') + shutil.copy(menu_file, real_root) + + def parse_attach_disks(self): + '''Parse $ATTACH_DISKS into list of disks to attach.''' + + if 'ATTACH_DISKS' in os.environ: + s = os.environ['ATTACH_DISKS'] + return s.split(':') + else: + return [] + + def bootloader_config_is_wanted(self): + '''Does the user want to generate a bootloader config? + + The user may set $BOOTLOADER_CONFIG_FORMAT to the desired + format (u-boot or extlinux). If not set, extlinux is the + default but will be generated on x86-32 and x86-64, but not + otherwise. + + ''' + + def is_x86(arch): + return (arch == 'x86_64' or + (arch.startswith('i') and arch.endswith('86'))) + + value = os.environ.get('BOOTLOADER_CONFIG_FORMAT', '') + if value == '': + if not is_x86(os.uname()[-1]): + return False + + return True + + def get_environment_boolean(self, variable): + '''Parse a yes/no boolean passed through the environment.''' + + value = os.environ.get(variable, 'no').lower() + if value in ['no', '0', 'false']: + return False + elif value in ['yes', '1', 'true']: + return True + else: + raise cliapp.AppException('Unexpected value for %s: %s' % + (variable, value)) + + def check_ssh_connectivity(self, ssh_host): + try: + cliapp.ssh_runcmd(ssh_host, ['true']) + except cliapp.AppException as e: + logging.error("Error checking SSH connectivity: %s", str(e)) + raise cliapp.AppException( + 'Unable to SSH to %s: %s' % (ssh_host, e)) diff --git a/morphlib/xfer-hole b/morphlib/xfer-hole new file mode 100755 index 00000000..0d4cee7a --- /dev/null +++ b/morphlib/xfer-hole @@ -0,0 +1,132 @@ +#!/usr/bin/env python +# +# Send a sparse file more space-efficiently. +# See recv-hole for a description of the protocol. +# +# Note that xfer-hole requires a version of Linux with support for +# SEEK_DATA and SEEK_HOLE. +# +# +# Copyright (C) 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + + +import errno +import os +import sys + + +SEEK_DATA = 3 +SEEK_HOLE = 4 + + +filename = sys.argv[1] +fd = os.open(filename, os.O_RDONLY) +pos = 0 + + +DATA = 'data' +HOLE = 'hole' +EOF = 'eof' + + +def safe_lseek(fd, pos, whence): + try: + return os.lseek(fd, pos, whence) + except OSError as e: + if e.errno == errno.ENXIO: + return -1 + raise + + +def current_data_or_pos(fd, pos): + length = safe_lseek(fd, 0, os.SEEK_END) + next_data = safe_lseek(fd, pos, SEEK_DATA) + next_hole = safe_lseek(fd, pos, SEEK_HOLE) + + if pos == length: + return EOF, pos + elif pos == next_data: + return DATA, pos + elif pos == next_hole: + return HOLE, pos + else: + assert False, \ + ("Do not understand: pos=%d next_data=%d next_hole=%d" % + (pos, next_data, next_hole)) + + +def next_data_or_hole(fd, pos): + length = safe_lseek(fd, 0, os.SEEK_END) + next_data = safe_lseek(fd, pos, SEEK_DATA) + next_hole = safe_lseek(fd, pos, SEEK_HOLE) + + if pos == length: + return EOF, pos + elif pos == next_data: + # We are at data. + if next_hole == -1 or next_hole == length: + return EOF, length + else: + return HOLE, next_hole + elif pos == next_hole: + # We are at a hole. + if next_data == -1 or next_data == length: + return EOF, length + else: + return DATA, next_data + else: + assert False, \ + ("Do not understand: pos=%d next_data=%d next_hole=%d" % + (pos, next_data, next_hole)) + + +def find_data_and_holes(fd): + pos = safe_lseek(fd, 0, os.SEEK_CUR) + + kind, pos = current_data_or_pos(fd, pos) + while kind != EOF: + yield kind, pos + kind, pos = next_data_or_hole(fd, pos) + yield kind, pos + + +def make_xfer_instructions(fd): + prev_kind = None + prev_pos = None + for kind, pos in find_data_and_holes(fd): + if prev_kind == DATA: + yield (DATA, prev_pos, pos) + elif prev_kind == HOLE: + yield (HOLE, prev_pos, pos) + prev_kind = kind + prev_pos = pos + + +def copy_slice_from_file(to, fd, start, end): + safe_lseek(fd, start, os.SEEK_SET) + data = os.read(fd, end - start) + to.write(data) + + +for kind, start, end in make_xfer_instructions(fd): + if kind == HOLE: + sys.stdout.write('HOLE\n%d\n' % (end - start)) + elif kind == DATA: + sys.stdout.write('DATA\n%d\n' % (end - start)) + copy_slice_from_file(sys.stdout, fd, start, end) diff --git a/morphlib/yamlparse.py b/morphlib/yamlparse.py new file mode 100644 index 00000000..6f139304 --- /dev/null +++ b/morphlib/yamlparse.py @@ -0,0 +1,39 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import morphlib +from morphlib.util import OrderedDict + +if morphlib.got_yaml: # pragma: no cover + yaml = morphlib.yaml + + +if morphlib.got_yaml: # pragma: no cover + + def load(*args, **kwargs): + return yaml.safe_load(*args, **kwargs) + + def dump(*args, **kwargs): + if 'default_flow_style' not in kwargs: + kwargs['default_flow_style'] = False + return yaml.dump(Dumper=morphlib.morphloader.MorphologyDumper, + *args, **kwargs) + +else: # pragma: no cover + def load(*args, **kwargs): + raise morphlib.Error('YAML not available') + def dump(*args, **kwargs): + raise morphlib.Error('YAML not available') diff --git a/morphlib/yamlparse_tests.py b/morphlib/yamlparse_tests.py new file mode 100644 index 00000000..38815168 --- /dev/null +++ b/morphlib/yamlparse_tests.py @@ -0,0 +1,64 @@ +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import unittest + +import morphlib +import morphlib.yamlparse as yamlparse +from morphlib.util import OrderedDict + +if morphlib.got_yaml: + yaml = morphlib.yaml + + +class YAMLParseTests(unittest.TestCase): + + def run(self, *args, **kwargs): + if morphlib.got_yaml: + return unittest.TestCase.run(self, *args, **kwargs) + + example_text = '''\ +name: foo +kind: chunk +build-system: manual +''' + + example_dict = OrderedDict([ + ('name', 'foo'), + ('kind', 'chunk'), + ('build-system', 'manual'), + ]) + + def test_non_map_raises(self): + incorrect_type = '''\ +!!map +- foo +- bar +''' + self.assertRaises(yaml.YAMLError, yamlparse.load, incorrect_type) + + def test_complex_key_fails_KNOWNFAILURE(self): + complex_key = '? { foo: bar, baz: qux }: True' + self.assertRaises(yaml.YAMLError, yamlparse.load, complex_key) + + def test_represents_non_scalar_nodes(self): + self.assertTrue( + yamlparse.dump( + { + ('a', 'b'): { + "foo": 1, + "bar": 2, + } + }, default_flow_style=None)) diff --git a/scripts/.gitconfig b/scripts/.gitconfig new file mode 100644 index 00000000..a1445eb3 --- /dev/null +++ b/scripts/.gitconfig @@ -0,0 +1,3 @@ +[user] + name = morph test suite + email = morph.tests@baserock.org diff --git a/scripts/check-copyright-year b/scripts/check-copyright-year new file mode 100755 index 00000000..e72eaeea --- /dev/null +++ b/scripts/check-copyright-year @@ -0,0 +1,109 @@ +#!/usr/bin/python +# +# Does the copyright statement include the year of the latest git commit? +# +# Copyright (C) 2012, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +from __future__ import print_function + +import datetime +import re +import sys + +import cliapp + +class CheckCopyrightYear(cliapp.Application): + + pat = re.compile(r'^[ #/*]*Copyright\s+(\(C\)\s*)' + r'(?P<years>[0-9, -]+)') + + def add_settings(self): + self.settings.boolean(['verbose', 'v'], 'be more verbose') + + def setup(self): + self.all_ok = True + self.uncommitted = self.get_uncommitted_files() + self.this_year = datetime.datetime.now().year + + def cleanup(self): + if not self.all_ok: + print('ERROR: Some copyright years need fixing', file=sys.stderr) + sys.exit(1) + + def get_uncommitted_files(self): + filenames = set() + status = self.runcmd(['git', 'status', '--porcelain', '-z']) + tokens = status.rstrip('\0').split('\0') + while tokens: + tok = tokens.pop(0) + filenames.add(tok[3:]) + if 'R' in tok[0:2]: + filenames.add(tokens.pop(0)) + return filenames + + def process_input_line(self, filename, line): + m = self.pat.match(line) + if not m: + return + + year = None + if filename not in self.uncommitted: + year = self.get_git_commit_year(filename) + + if year is None: + # git does not have a commit date for the file, which might + # happen if the file isn't committed yet. This happens during + # development, and it's OK. It's not quite a lumberjack, but + # let's not get into gender stereotypes here. + year = self.this_year + + ok = False + for start, end in self.get_copyright_years(m): + if start <= year <= end: + ok = True + + if ok: + if self.settings['verbose']: + self.output.write('OK %s\n' % filename) + else: + self.output.write('BAD %s:%s:%s\n' % + (filename, self.lineno, line.strip())) + + self.all_ok = self.all_ok and ok + + def get_git_commit_year(self, filename): + out = self.runcmd(['git', 'log', '-1', '--format=format:%cd', + filename]) + if not out: + return None + words = out.split() + return int(words[4]) + + def get_copyright_years(self, match): + years = match.group('years') + groups = [s.strip() for s in years.split(',')] + + for group in groups: + if '-' in group: + start, end = group.split('-') + else: + start = end = group + start = int(start) + end = int(end) + yield start, end + + +CheckCopyrightYear().run() diff --git a/scripts/check-silliness b/scripts/check-silliness new file mode 100755 index 00000000..597eb664 --- /dev/null +++ b/scripts/check-silliness @@ -0,0 +1,63 @@ +#!/bin/sh +# +# Does the file contain any of the code constructs deemed silly? +# +# Copyright (C) 2013, 2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +errors=0 + +for x; do + if tr -cd '\t' < "$x" | grep . > /dev/null + then + echo "ERROR: $x contains TAB characters" 1>&2 + grep -n -F "$(printf "\t")" "$x" 1>&2 + errors=1 + fi + + case "$x" in + # Excluding yarn files since it's not possible to split up the + # IMPLEMENTS lines of them + *.yarn) ;; + *) + if awk 'length($0) > 79' "$x" | grep . > /dev/null + then + echo "ERROR: $x has lines longer than 79 chars" 1>&2 + awk 'length($0) > 79 { print NR, $0 }' "$x" 1>&2 + errors=1 + fi + ;; + esac + + case "$x" in + *.py) + if head -1 "$x" | grep '^#!' > /dev/null + then + echo "ERROR: $x has a hashbang" 1>&2 + errors=1 + fi + if [ -x "$x" ]; then + echo "ERROR: $x is executable" 1>&2 + errors=1 + fi + if grep except: "$x" + then + echo "ERROR: $x has a bare except:" 1>&2 + errors=1 + fi + ;; + esac +done +exit "$errors" diff --git a/scripts/clean-artifact-cache b/scripts/clean-artifact-cache new file mode 100755 index 00000000..2fdd5605 --- /dev/null +++ b/scripts/clean-artifact-cache @@ -0,0 +1,95 @@ +#!/bin/sh + +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Remove all chunk artifacts in the cache except the newest. Morph does +# not currently clean its caches at any point, so this script is necessary +# to avoid running out of disk space. + +set -e + +usage() { + echo "Usage: clean-artifact-cache [--all] [CHUNK_NAME]" + echo + echo "WARNING: this script removes all but the chunks with the latest" + echo "mtimes. This is usually what you want, but you should try the" + echo "script on a small chunk first and trigger a rebuild to make sure" + echo "that you are not removing artifacts that you still want." +} + +if [ -z $1 ]; then + usage + exit 0 +fi + +CHUNK= +case $1 in + --all) + CHUNK=* + ;; + -*) + usage + exit 0 + ;; + *) + CHUNK=$1 +esac + + +clean_chunk() { + ARTIFACT_COUNT=$(ls *.chunk.$1 | wc -l) + + if [ $ARTIFACT_COUNT -lt 2 ]; then + return + fi + + echo "$1: $(expr $ARTIFACT_COUNT - 1) stale artifact(s)" + + SKIPPED_LATEST= + for f in $(ls -1t *.chunk.$1); do + if [ -z "$SKIPPED_LATEST" ]; then + SKIPPED_LATEST=yes + else + rm $(echo $f | cut -c -64).build-log + rm $(echo $f | cut -c -64).meta + rm $(echo $f | cut -c -64).chunk.$1 + fi + done +} + +test "x$MORPH" = "x" && MORPH=morph + +CACHE_DIR=$($MORPH --dump-config | grep cachedir | awk '{print $3}') +ARTIFACT_CACHE="${CACHE_DIR}/artifacts" + +cd $ARTIFACT_CACHE +SIZE_BEFORE=$(du -sh . | cut -f 1) + +if [ "$CHUNK" = "*" ]; then + echo "Removing ALL out-of-date chunk artifacts in $ARTIFACT_CACHE" + + for chunk in $(ls *.chunk.* | cut -d '.' -f 3-); do + clean_chunk $chunk + done +else + echo "Removing out of date artifacts for chunk $CHUNK in" \ + "$ARTIFACT_CACHE" + clean_chunk $CHUNK +fi + +SIZE_AFTER=$(du -sh . | cut -f 1) + +echo "Artifact cache size before: $SIZE_BEFORE after: $SIZE_AFTER" diff --git a/scripts/cmd-filter b/scripts/cmd-filter new file mode 100755 index 00000000..d2f4f784 --- /dev/null +++ b/scripts/cmd-filter @@ -0,0 +1,40 @@ +#!/bin/sh +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Wrapper around morph for use by cmdtest tests. This does several things: +# +# * simpler command lines for running morph, so that each test does not +# need to add --no-default-config and other options every time +# * replace temporary filenames ($DATADIR) in the output with a known +# string ("TMP"), so that test output is deterministic + +set -eu + +if "$@" > "$DATADIR/stdout" 2> "$DATADIR/stderr" +then + exit=0 +else + exit=1 +fi + +sed -i "s,$DATADIR,TMP,g" "$DATADIR/stdout" "$DATADIR/stderr" +cat "$DATADIR/stdout" +cat "$DATADIR/stderr" 1>&2 + +rm -f "$DATADIR/stdout" "$DATADIR/stderr" + +exit "$exit" + diff --git a/scripts/convert-git-cache b/scripts/convert-git-cache new file mode 100755 index 00000000..33a8edf1 --- /dev/null +++ b/scripts/convert-git-cache @@ -0,0 +1,48 @@ +#!/bin/sh +# +# Copyright (C) 2012,2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +set -eu + +CACHE_ROOT=$(morph --dump-config | grep cachedir | cut -d\ -f3) + +REPO_CACHE="${CACHE_ROOT}/gits" + +for REPO_DIR in $(cd "${REPO_CACHE}"; ls); do + cd "${REPO_CACHE}/${REPO_DIR}" + if test -d .git; then + echo "Converting ${REPO_DIR}" + mv .git/* . + rmdir .git + git config core.bare true + git config remote.origin.mirror true + git config remote.origin.fetch "+refs/*:refs/*" + echo "Migrating refs, please hold..." + rm -f refs/remotes/origin/HEAD + for REF in $(git branch -r); do + BRANCH=${REF#origin/} + git update-ref "refs/heads/${BRANCH}" \ + $(git rev-parse "refs/remotes/${REF}") + git update-ref -d "refs/remotes/${REF}" + done + echo "Re-running remote update with --prune" + if ! git remote update origin --prune; then + echo "${REPO_DIR} might be broken." + fi + else + echo "Do not need to convert ${REPO_DIR}" + fi +done diff --git a/scripts/edit-morph b/scripts/edit-morph new file mode 100755 index 00000000..90679b23 --- /dev/null +++ b/scripts/edit-morph @@ -0,0 +1,286 @@ +#!/usr/bin/env python +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import contextlib +import os +import re +import yaml + +import morphlib + +class EditMorph(cliapp.Application): + '''Tools for performing set operations on large morphologies''' + + def add_settings(self): + self.settings.boolean(['no-git-update'], + 'do not update the cached git repositories ' + 'automatically') + + def load_morphology(self, file_name, expected_kind = None): + loader = morphlib.morphloader.MorphologyLoader() + morphology = loader.load_from_file(file_name) + + if expected_kind is not None and morphology['kind'] != expected_kind: + raise morphlib.Error("Expected: a %s morphology" % expected_kind) + + return morphology, text + + def cmd_remove_chunk(self, args): + '''Removes from STRATUM all reference of CHUNK''' + + if len(args) != 2: + raise cliapp.AppException("remove-chunk expects a morphology file " + "name and a chunk name") + + file_name = args[0] + chunk_name = args[1] + morphology, text = self.load_morphology(file_name, + expected_kind='stratum') + + component_count = 0 + build_depends_count = 0 + new_chunks = morphology['chunks'] + for info in morphology['chunks']: + if info['name'] == chunk_name: + new_chunks.remove(info) + component_count += 1 + elif chunk_name in info['build-depends']: + info['build-depends'].remove(chunk_name) + build_depends_count += 1 + morphology['chunks'] = new_chunks + + loader = morphlib.morphloader.MorphologyLoader() + loader.save_to_file(file_name, morphology) + + self.output.write("Removed: %i chunk(s) and %i build depend(s).\n" % + (component_count, build_depends_count)) + + def cmd_sort(self, args): + """Sort STRATUM""" + + if len(args) != 1: + raise cliapp.AppException("sort expects a morphology file name") + + file_name = args[0] + morphology, text = self.load_morphology(file_name, + expected_kind='stratum') + + for chunk in morphology['chunks']: + chunk['build-depends'].sort() + + morphology['chunks'] = self.sort_chunks(morphology['chunks']) + + loader = morphlib.morphloader.MorphologyLoader() + loader.save_to_file(file_name, morphology) + + def sort_chunks(self, chunks_list): + """Sort stratum chunks + + The order is something like alphabetical reverse dependency order. + Chunks on which nothing depends are sorted at the bottom. + + The algorithm used is a simple-minded recursive sort. + """ + + chunk_dict = {} + for chunk in chunks_list: + chunk_dict[chunk['name']] = chunk + + reverse_deps_dict = {} + for chunk_name in chunk_dict.keys(): + chunk = chunk_dict[chunk_name] + for dep in chunk['build-depends']: + if dep not in reverse_deps_dict: + reverse_deps_dict[dep] = [chunk_name] + else: + reverse_deps_dict[dep].append(chunk_name) + + sort_order = list(chunk_dict.keys()) + sort_order.sort(key=unicode.lower) + + result = [] + satisfied_list = [] + repeat_count = 0 + while len(sort_order) > 0: + postponed_list = [] + + # Move any chunk into the result order that has all its + # dependencies satisfied in the result already. + for chunk_name in sort_order: + deps_satisfied = True + + chunk = chunk_dict[chunk_name] + for dep in chunk['build-depends']: + if dep not in satisfied_list: + deps_satisfied = False + if dep not in sort_order: + raise cliapp.AppException( + 'Invalid build-dependency for %s: %s' + % (chunk['name'], dep)) + break + + if deps_satisfied: + result.append(chunk) + satisfied_list.append(chunk_name) + else: + postponed_list.append(chunk_name) + + if len(postponed_list) == len(sort_order): + # This is not the smartest algorithm possible (but it works!) + repeat_count += 1 + if repeat_count > 10: + raise cliapp.AppException('Stuck in loop while sorting') + + assert(len(postponed_list) + len(result) == len(chunk_dict.keys())) + sort_order = postponed_list + + # Move chunks which are not build-depends of other chunks to the end. + targets = [c for c in chunk_dict.keys() if c not in reverse_deps_dict] + targets.sort(key=unicode.lower) + for chunk_name in targets: + result.remove(chunk_dict[chunk_name]) + result.append(chunk_dict[chunk_name]) + + return result + + @staticmethod + @contextlib.contextmanager + def _open_yaml(path): + with open(path, 'r') as f: + d = yaml.load(f) + yield d + with open(path, 'w') as f: + yaml.dump(d, f, default_flow_style=False) + + def cmd_set_system_artifact_depends(self, args): + '''Change the artifacts used by a System. + + Usage: MORPHOLOGY_FILE STRATUM_NAME ARTIFACTS + + ARTIFACTS is an English language string describing which artifacts + to include, since the primary use of this command is to assist + yarn tests. + + Example: edit-morph set-system-artifact-depends system.morph \ + build-essential "build-essential-minimal, + build-essential-runtime and build-essential-devel" + + ''' + + file_path = args[0] + stratum_name = args[1] + artifacts = re.split(r"\s+and\s+|,?\s*", args[2]) + with self._open_yaml(file_path) as d: + for spec in d["strata"]: + if spec.get("alias", spec["name"]) == stratum_name: + spec["artifacts"] = artifacts + + def cmd_set_stratum_match_rules(self, (file_path, match_rules)): + '''Set a stratum's match rules. + + Usage: FILE_PATH MATCH_RULES_YAML + + This sets the stratum's "products" field, which is used to + determine which chunk artifacts go into which stratum artifacts + the stratum produces. + + The match rules must be a string that yaml can parse. + + ''' + with self._open_yaml(file_path) as d: + d['products'] = yaml.load(match_rules) + + @classmethod + def _splice_cluster_system(cls, syslist, syspath): + sysname = syspath[0] + syspath = syspath[1:] + for system in syslist: + if sysname in system['deploy']: + break + else: + system = { + 'morph': None, + 'deploy': { + sysname: { + 'type': None, + 'location': None, + }, + }, + } + syslist.append(system) + if syspath: + cls._splice_cluster_system( + system.setdefault('subsystems', []), syspath) + + @classmethod + def _find_cluster_system(cls, syslist, syspath): + sysname = syspath[0] + syspath = syspath[1:] + for system in syslist: + if sysname in system['deploy']: + break + if syspath: + return cls._find_cluster_system(system['subsystems'], syspath) + return system + + def cmd_cluster_init(self, (cluster_file,)): + with open(cluster_file, 'w') as f: + d = { + 'name': os.path.splitext(os.path.basename(cluster_file))[0], + 'kind': 'cluster', + } + yaml.dump(d, f) + + def cmd_cluster_system_init(self, (cluster_file, system_path)): + syspath = system_path.split('.') + with self._open_yaml(cluster_file) as d: + self._splice_cluster_system(d.setdefault('systems', []), syspath) + + def cmd_cluster_system_set_morphology(self, + (cluster_file, system_path, morphology)): + + syspath = system_path.split('.') + with self._open_yaml(cluster_file) as d: + system = self._find_cluster_system(d['systems'], syspath) + system['morph'] = morphology + + def cmd_cluster_system_set_deploy_type(self, + (cluster_file, system_path, deploy_type)): + + syspath = system_path.split('.') + with self._open_yaml(cluster_file) as d: + system = self._find_cluster_system(d['systems'], syspath) + system['deploy'][syspath[-1]]['type'] = deploy_type + + def cmd_cluster_system_set_deploy_location(self, + (cluster_file, system_path, deploy_location)): + + syspath = system_path.split('.') + with self._open_yaml(cluster_file) as d: + system = self._find_cluster_system(d['systems'], syspath) + system['deploy'][syspath[-1]]['location'] = deploy_location + + def cmd_cluster_system_set_deploy_variable(self, + (cluster_file, system_path, key, val)): + + syspath = system_path.split('.') + with self._open_yaml(cluster_file) as d: + system = self._find_cluster_system(d['systems'], syspath) + system['deploy'][syspath[-1]][key] = val + +EditMorph().run() diff --git a/scripts/fix-committer-info b/scripts/fix-committer-info new file mode 100644 index 00000000..0bd85274 --- /dev/null +++ b/scripts/fix-committer-info @@ -0,0 +1,25 @@ +#!/bin/sh +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Fix git committer info. By hardcoding all of this info we make sure that +# all of the commits we make during testing have reproducible commit SHA1s. + +export GIT_AUTHOR_NAME=developer +export GIT_AUTHOR_EMAIL=developer@example.com +export GIT_AUTHOR_DATE="1343753514 +0000" +export GIT_COMMITTER_NAME=developer +export GIT_COMMITTER_EMAIL=developer@example.com +export GIT_COMMITTER_DATE="1343753514 +0000" diff --git a/scripts/git-daemon-wrap b/scripts/git-daemon-wrap new file mode 100755 index 00000000..528b7bed --- /dev/null +++ b/scripts/git-daemon-wrap @@ -0,0 +1,46 @@ +#!/usr/bin/python + +'Launch a Git Daemon on an ephemeral port, and report which port was used.' + +import argparse +import contextlib +import pipes +import socket +import subprocess +import sys + +# Parse arguments with bare argparse, since cliapp hates unknown options +class UnsupportedArgument(argparse.Action): + def __call__(self, parser, namespace, values, option_string=None): + sys.stderr.write('%s not supported\n' % option_string) + sys.exit(1) + +parser = argparse.ArgumentParser(description=__doc__) +for arg in ('user', 'group', 'detach', 'port', 'syslog', 'pid-file'): + parser.add_argument('--' + arg, action=UnsupportedArgument) +parser.add_argument('--listen', default='127.0.0.1') +parser.add_argument('--port-file', required=True, + help='Report which port the git daemon was bound to.') +options, args = parser.parse_known_args() + +with contextlib.closing(socket.socket()) as sock: + sock.bind((options.listen, 0)) + host, port = sock.getsockname() + with open(options.port_file, 'w') as f: + f.write('%s\n' % port) + sock.listen(1) + while True: + conn, addr = sock.accept() + with contextlib.closing(conn): + gitcmd = ['git', 'daemon', '--inetd'] + gitcmd.extend(args) + cmdstr = (' '.join(map(pipes.quote, gitcmd))) + sys.stderr.write('Running %s' % cmdstr) + ret = subprocess.call(args=gitcmd, stdin=conn, stdout=conn, + stderr=conn, close_fds=True) + if ret != 0: + sys.stderr.write('%s exited %d\n' % (cmdstr, ret)) + # git-daemon returns 255 when the repo doesn't exist + if ret not in (0, 255): + break +sys.exit(ret) diff --git a/scripts/list-tree b/scripts/list-tree new file mode 100755 index 00000000..a1e2e8cb --- /dev/null +++ b/scripts/list-tree @@ -0,0 +1,45 @@ +#!/bin/sh +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# List contents of a directory tree in a reproducible manner: only include +# details that we care about, and that won't be changing between test runs +# or test environments. + +set -eu + +shorttype(){ + case "$*" in + "directory") + echo d + ;; + "regular file"|"regular empty file") + echo f + ;; + "symbolic link") + echo l + ;; + *) + echo "$*" >&2 + echo U + ;; + esac +} + +export LC_ALL=C +cd "$1" +find | while read file; do + printf "%s %s\n" "$(shorttype $(stat -c %F $file))" "$file"; +done | sort diff --git a/scripts/python-check b/scripts/python-check new file mode 100644 index 00000000..ce3419d5 --- /dev/null +++ b/scripts/python-check @@ -0,0 +1,36 @@ +#!/bin/sh +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# When sourced by scripts, if the python version is too old +# fake the output and exit. + +if ! python --version 2>&1 | grep '^Python 2\.[78]' > /dev/null +then + outpath="$(dirname "$0")/$(basename "$0" .script).stdout" + errpath="$(dirname "$0")/$(basename "$0" .script).stderr" + exitpath="$(dirname "$0")/$(basename "$0" .script).exit" + if [ -r "$outpath" ]; then + cat "$outpath" + fi + if [ -r "$errpath" ]; then + cat "$errpath" >&2 + fi + if [ -r "$exitpath" ]; then + exit "$(cat "$exitpath")" + else + exit 0 + fi +fi diff --git a/scripts/review-gitmodules b/scripts/review-gitmodules new file mode 100755 index 00000000..89574833 --- /dev/null +++ b/scripts/review-gitmodules @@ -0,0 +1,121 @@ +#!/usr/bin/python +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Check every system morphology in a checked out system branch by editing +# every chunk and recursively checking gitmodules. + +import glob +import os +import re +import shutil +import tempfile + +import cliapp +import yaml + +class ReviewGitmodules(cliapp.Application): + + def process_args(self, args): + chunks = self.read_all_systems() + for chunk in chunks: + self.check_chunk(chunk) + + def merge_lists(self, old_list, new_list): + for entry in new_list: + if entry not in old_list: + old_list.append(entry) + + return old_list + + def read_all_systems(self): + chunks = [] + files = glob.glob('*.morph') + for entry in files: + with open(entry, 'r') as f: + morph = yaml.load(f) + if morph['kind'] == 'system': + found_chunks = self.read_all_strata(morph, files) + chunks = self.merge_lists(chunks, found_chunks) + + return chunks + + def read_all_strata(self, system, files): + chunks = [] + for stratum in system['strata']: + morph_file = stratum['morph']+'.morph' + if morph_file not in files: + raise cliapp.AppException('Morph %s not found in this system ' + 'branch. I am not clever enough to ' + 'find that, myself' % morph_file) + + with open(morph_file, 'r') as f: + stratum_morph = yaml.load(f) + + if stratum_morph['kind'] != 'stratum': + raise cliapp.AppException('Morph %s is not a stratum' + % morph_file) + + found_chunks = self.read_all_chunks(stratum_morph) + chunks = self.merge_lists(chunks, found_chunks) + + return chunks + + def read_all_chunks(self, stratum): + return stratum['chunks'] + + def check_chunk(self, chunk): + chunk_dir = tempfile.mkdtemp() + submodules_file = os.path.join(chunk_dir, '.gitmodules') + + expand_repo_output = cliapp.runcmd(['morph', 'expand-repo', + chunk['repo']]) + for line in expand_repo_output.splitlines(): + if line.startswith('pull:'): + pull_ref = line.split()[1] + break + + cliapp.runcmd(['git', 'clone', pull_ref, chunk_dir]) + cliapp.runcmd(['git', 'checkout', chunk['ref']], cwd=chunk_dir) + + if os.path.exists(submodules_file): + regex = re.compile(r''' + \[submodule\s"(?P<name>.*)"\]\s+ + path\s+=\s+(?P<path>\S+)\s+ + url\s+=\s+(?P<url>\S+) + ''', re.VERBOSE) + + self.output.write('Chunk %s has submodules\n' % chunk['name']) + with open(submodules_file, 'r') as f: + submodules_text = f.read() + + self.output.write('%s\n' % submodules_text) + submodules = regex.findall(submodules_text) + # Unfortunately, findall returns a list of tuples, not dicts + for submodule in submodules: + tree_data = cliapp.runcmd(['git', 'cat-file', '-p', + 'HEAD^{tree}'], cwd=chunk_dir) + + for line in tree_data.splitlines(): + words = line.split(None, 3) + if words[3] == submodule[2]: + submodule_ref = words[2] + self.check_chunk({'name':submodule[0], + 'repo':submodule[2], + 'ref':submodule_ref}) + + shutil.rmtree(chunk_dir) + +ReviewGitmodules().run() diff --git a/scripts/run-git-in b/scripts/run-git-in new file mode 100755 index 00000000..80b87d1a --- /dev/null +++ b/scripts/run-git-in @@ -0,0 +1,25 @@ +#!/bin/sh +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Run git in a given directory. + + +set -eu + +cd "$1" +shift +"$SRCDIR/scripts/cmd-filter" git "$@" diff --git a/scripts/setup-3rd-party-strata b/scripts/setup-3rd-party-strata new file mode 100644 index 00000000..fc263f96 --- /dev/null +++ b/scripts/setup-3rd-party-strata @@ -0,0 +1,135 @@ +#!/bin/sh +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Create strata outside the main morphologies repository, which is useful +# for the more complex workflow tests. + + +. "$SRCDIR/scripts/fix-committer-info" + +create_chunk() { + REPO="$1" + NAME="$2" + + mkdir "$1" + ln -s "$1" "$1.git" + cd "$1" + + cat <<EOF > "$1/$2.morph" +{ + "name": "$2", + "kind": "chunk", + "build-system": "dummy" +} +EOF + + git init --quiet + git add . + git commit --quiet -m "Initial commit" +} + +write_stratum_morph() { + REPO="$1" + NAME="$2" + +cat <<EOF > "$1/$2.morph" +{ + "name": "$2", + "kind": "stratum", + "chunks": [ + { + "name": "hello", + "repo": "test:$2-hello", + "ref": "master", + "build-mode": "test", + "build-depends": [] + } + ] +} +EOF +} + +# Create two more strata outside the test:morphs repository + +EXTERNAL_STRATA_REPO="$DATADIR/external-strata" +mkdir "$EXTERNAL_STRATA_REPO" +ln -s "$EXTERNAL_STRATA_REPO" "$EXTERNAL_STRATA_REPO".git +cd "$EXTERNAL_STRATA_REPO" + +git init --quiet . + +write_stratum_morph "$EXTERNAL_STRATA_REPO" "stratum2" +write_stratum_morph "$EXTERNAL_STRATA_REPO" "stratum3" + +git add . +git commit --quiet -m "Initial commit" + +# To make life harder, both chunks have the same name too + +create_chunk "$DATADIR/stratum2-hello" "hello" +create_chunk "$DATADIR/stratum3-hello" "hello" + +# Update hello-system to include them ... using a system branch! Since the +# strata refs are 'master' not 'me/add-external-strata' this does not cause +# problems with merging. + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs me/add-external-strata + +cd "$DATADIR/workspace/me/add-external-strata/test:morphs" + +cat <<EOF > "hello-system.morph" +{ + "name": "hello-system", + "kind": "system", + "arch": "x86_64", + "strata": [ + { + "morph": "hello-stratum", + "repo": "test:morphs", + "ref": "master" + }, + { + "morph": "stratum2", + "repo": "test:external-strata", + "ref": "master" + }, + { + "morph": "stratum3", + "repo": "test:external-strata", + "ref": "master" + } + ] +} +EOF +git commit --quiet --all -m "Add two more external strata" + +# Merge to master +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" checkout test:morphs master +cd master/test:morphs +"$SRCDIR/scripts/test-morph" merge me/add-external-strata + +# In reality the user would do: 'git push origin master' here, +# but since our remote repo is non-bare we must cheat a bit. +# We should consider a separate fixture for the workflow tests. +cd "$DATADIR/morphs" +git pull -q \ + "file://$DATADIR/workspace/master/test:morphs" master + +cd "$DATADIR/workspace" diff --git a/scripts/sparse-gunzip b/scripts/sparse-gunzip new file mode 100755 index 00000000..b6e1aa16 --- /dev/null +++ b/scripts/sparse-gunzip @@ -0,0 +1,6 @@ +#!/usr/bin/python +from morphlib.util import copyfileobj +import gzip, sys +infh = gzip.GzipFile(fileobj=sys.stdin) +copyfileobj(infh, sys.stdout) +infh.close() diff --git a/scripts/test-morph b/scripts/test-morph new file mode 100755 index 00000000..d8480d92 --- /dev/null +++ b/scripts/test-morph @@ -0,0 +1,57 @@ +#!/bin/sh +# Copyright (C) 2012,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Wrapper around morph for use by cmdtest tests. This does several things: +# +# * simpler command lines for running morph, so that each test does not +# need to add --no-default-config and other options every time +# * replace temporary filenames ($DATADIR) in the output with a known +# string ("TMP"), so that test output is deterministic + +set -eu + +# Set PATH to include the source directory. This is necessary for +# distributed builds, which invoke morph as a sub-process. +export PATH="$SRCDIR:$PATH" + +WARNING_IGNORES='-W ignore:(stratum|system)\s+morphology' +if [ "$1" = "--find-system-artifact" ]; then + shift + + python $WARNING_IGNORES \ + "$SRCDIR/morph" --no-default-config \ + --tarball-server= --cache-server= \ + --cachedir-min-space=0 --tempdir-min-space=0 \ + --config="$DATADIR/morph.conf" --verbose "$@" > $DATADIR/stdout + + ARTIFACT=$(grep "system \S\+-rootfs is cached at" "$DATADIR/stdout" | \ + sed -nre "s/^.*system \S+-rootfs is cached at (\S+)$/\1/p") + rm "$DATADIR/stdout" + + if [ ! -e "$ARTIFACT" ]; then + echo "Unable to find rootfs artifact: $ARTIFACT" 1>&2 + exit 1 + fi + + echo $ARTIFACT +else + "$SRCDIR/scripts/cmd-filter" \ + python $WARNING_IGNORES \ + "$SRCDIR/morph" --no-default-config \ + --cachedir-min-space=0 --tempdir-min-space=0 \ + --tarball-server= --cache-server= \ + --config="$DATADIR/morph.conf" "$@" +fi diff --git a/scripts/test-shell.c b/scripts/test-shell.c new file mode 100644 index 00000000..7975c188 --- /dev/null +++ b/scripts/test-shell.c @@ -0,0 +1,144 @@ +#include <fcntl.h> +#include <sys/types.h> +#include <sys/stat.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <getopt.h> + +#include <stdint.h> +#include <ftw.h> +#include <errno.h> + +char *readlinka(char const *path){ + size_t buflen = BUFSIZ; + char *buf = malloc(buflen); + ssize_t read; + while ((read = readlink(path, buf, buflen - 1)) >= buflen - 1) { + char *newbuf = realloc(buf, buflen * 2); + if (newbuf == NULL) { + goto failure; + } + buf = newbuf; + buflen = buflen * 2; + } + buf[read] = '\0'; + return buf; +failure: + free(buf); + return NULL; +} + +int copy_file_paths(char const *source_file, char const *target_file) { + int source_fd; + int target_fd; + int ret = -1; + struct stat st; + if ((source_fd = open(source_file, O_RDONLY)) == -1) { + return ret; + } + if (fstat(source_fd, &st) == -1) { + perror("stat"); + ret = -2; + goto cleanup_in; + } + if ((target_fd = open(target_file, O_WRONLY|O_CREAT, st.st_mode)) == -1) { + ret = -3; + goto cleanup_in; + } + ssize_t read; + while ((read = sendfile(target_fd, source_fd, NULL, BUFSIZ)) > 0); + if (read < 0) { + perror("sendfile"); + ret = -4; + } + ret = 0; +cleanup_all: + close(target_fd); +cleanup_in: + close(source_fd); + return ret; +} + +int copy_entry(const char *fpath, const struct stat *sb, int typeflag, + struct FTW *ftwbuf) { + int ret = 0; + char *target_path = NULL; + if (asprintf(&target_path, "%s/%s", getenv("DESTDIR"), fpath) == -1) { + return -1; + } + switch (typeflag) { + case FTW_F: + /* Copy file */ + if ((ret = copy_file_paths(fpath, target_path)) < 0) { + perror("Copy file"); + ret = -1; + } + break; + case FTW_D: + case FTW_DNR: + /* Copy directory */ + if (mkdir(target_path, sb->st_mode)) { + if (errno != EEXIST) { + perror("mkdir"); + ret = -1; + } + } + break; + case FTW_NS: + case FTW_SL: + case FTW_SLN: { + /* Create symlink */ + char *link_target = readlinka(fpath); + if (link_target == NULL) { + perror("readlink"); + ret = -1; + } + if (symlink(link_target, target_path) == -1) { + perror("symlink"); + ret = -1; + } + break; + } + } +cleanup: + free(target_path); + return ret; +} + +int main(int argc, char *argv[]) { + int ret = 1; + if (argc != 3 || strcmp(argv[1], "-c") != 0) { + fprintf(stderr, "Usage: %s -c COMMAND\n", argv[0]); + return 1; + } + size_t cmdlen = strlen(argv[2]); + FILE *cmdstream = fmemopen(argv[2], cmdlen, "r"); + { + ssize_t read; + size_t len = 0; + char *line = NULL; + + ret = 0; + while ((read = getline(&line, &len, cmdstream)) != -1) { + if (line[read - 1] == '\n') line[read - 1] = '\0'; + if (strcmp(line, "copy files") == 0) { + /* Recursively copy contents of current dir to DESTDIR */ + if (nftw(".", copy_entry, 20, FTW_PHYS)) { + ret = 1; + break; + } + } else if (strcmp(line, "false") == 0 || + strstr(line, "false ") == line) { + ret = 1; + break; + } else { + ret = 127; + break; + } + } + free(line); + } + return ret; +} diff --git a/scripts/yaml-extract b/scripts/yaml-extract new file mode 100755 index 00000000..6f55e62f --- /dev/null +++ b/scripts/yaml-extract @@ -0,0 +1,76 @@ +#!/usr/bin/python +# +# Extract field from YAML format morphologies, using a very simple +# query language. This is useful for black box testing. +# +# Usage: yaml-extract FILE PARAM... +# +# Where FILE is the name of the YAML morphology, and PARAM are a sequence +# of query parameters. +# +# The program reads in the YAML file, and then selects successively deeper +# parts of the object hieararchy in the file. If the object currently +# being looked at is a dictionary, PARAM is a field in the dictionary, +# and the next PARAM will look at the value stored with that key. +# If the current object is a list, PARAM can either be an integer list +# index, or a search key of the form KEY=VALUE, in which case the list +# is searched for the first member, which must be a dict, which has +# a key KEY that stores a value VALUE. +# +# Example: +# +# yaml-extract system.morph strata morph=core ref +# +# This would report the ref of the core stratum in a system. +# +# Note that this does not try to parse morphologies as morphologies, +# and so doesn't do special processing such as providing computed +# values for missing fields (e.g., the morph field if name is given). +# Construct your tests accordingly. + +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import sys +import yaml + + +with open(sys.argv[1]) as f: + obj = yaml.safe_load(f) +for thing in sys.argv[2:]: + if type(obj) == dict: + if thing not in obj: + raise Exception("Object does not contain %s" % thing) + obj = obj[thing] + elif type(obj) == list: + if '=' in thing: + # We need to search a list member dict with a given field. + key, value = thing.split('=', 1) + for item in obj: + if item.get(key) == value: + obj = item + break + else: + raise Exception( + "Couldn't find list item containing %s" % thing) + else: + # We can just index. + obj = obj[int(thing)] + else: + raise Exception("Can't handle %s with %s" % (repr(obj), repr(thing))) + +print obj + @@ -1,46 +1,170 @@ -#!/usr/bin/python +# Copyright (C) 2011 - 2014 Codethink Limited # -# Copyright (C) 2012 Codethink Limited -# # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; version 2 of the License. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +'''Setup.py for morph.''' + + from distutils.core import setup +from distutils.cmd import Command +from distutils.command.build import build +from distutils.command.clean import clean +import glob +import os +import os.path +import shutil +import stat +import subprocess + +import cliapp + +import morphlib + + +class GenerateResources(build): + + def run(self): + if not self.dry_run: + self.generate_manpages() + self.generate_version() + build.run(self) + + # Set exec permissions on deployment extensions. + for dirname, subdirs, basenames in os.walk('morphlib/exts'): + for basename in basenames: + orig = os.path.join(dirname, basename) + built = os.path.join('build/lib', dirname, basename) + st = os.lstat(orig) + bits = (st.st_mode & + (stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)) + if bits != 0: + st2 = os.lstat(built) + os.chmod(built, st2.st_mode | bits) + + def generate_manpages(self): + self.announce('building manpages') + for x in ['morph']: + with open('%s.1' % x, 'w') as f: + subprocess.check_call(['python', x, + '--generate-manpage=%s.1.in' % x, + '--output=%s.1' % x], stdout=f) + + def generate_version(self): + target_dir = os.path.join(self.build_lib, 'morphlib') + + self.mkpath(target_dir) + + def save_git_info(filename, *args): + path = os.path.join(target_dir, filename) + command = ['git'] + list(args) + + self.announce('generating %s with %s' % + (path, ' '.join(command))) + + with open(os.path.join(target_dir, filename), 'w') as f: + cwd = os.path.dirname(__file__) or '.' + p = subprocess.Popen(command, + cwd=cwd, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + o = p.communicate() + if p.returncode: + raise subprocess.CalledProcessError(p.returncode, command) + f.write(o[0].strip()) + + save_git_info('version', 'describe', '--abbrev=40', '--always', + '--dirty=-unreproducible', + '--match=DO-NOT-MATCH-ANY-TAGS') + save_git_info('commit', 'rev-parse', 'HEAD^{commit}') + save_git_info('tree', 'rev-parse', 'HEAD^{tree}') + save_git_info('ref', 'rev-parse', '--symbolic-full-name', 'HEAD') + +class Clean(clean): + + clean_files = [ + '.coverage', + 'build', + 'unittest-tempdir', + ] + clean_globs = [ + '*/*.py[co]', + ] + + def run(self): + clean.run(self) + itemses = ([self.clean_files] + + [glob.glob(x) for x in self.clean_globs]) + for items in itemses: + for filename in items: + if os.path.isdir(filename): + shutil.rmtree(filename) + elif os.path.exists(filename): + os.remove(filename) + + +class Check(Command): + + user_options = [ + ] + + def initialize_options(self): + pass + + def finalize_options(self): + pass + + def run(self): + subprocess.check_call(['python', '-m', 'CoverageTestRunner', + '--ignore-missing-from=without-test-modules', + 'morphlib', 'distbuild']) + os.remove('.coverage') -setup(name='morph-cache-server', - description='FIXME', - long_description='''\ -FIXME -''', +setup(name='morph', classifiers=[ - 'Development Status :: 2 - Pre-Alpha', - 'Environment :: Console', - 'Environment :: Web Environment', - 'Intended Audience :: Developers', - 'Intended Audience :: System Administrators', - 'License :: OSI Approved :: GNU General Public License (GPL)', - 'Operating System :: POSIX :: Linux', - 'Programming Language :: Python', - 'Topic :: Software Development :: Build Tools', - 'Topic :: Software Development :: Embedded Systems', - 'Topic :: System :: Archiving :: Packaging', - 'Topic :: System :: Software Distribution', + 'Development Status :: 2 - Pre-Alpha', + 'Environment :: Console', + 'Intended Audience :: Developers', + 'License :: OSI Approved :: GNU General Public License (GPL)', + 'Operating System :: POSIX :: Linux', + 'Programming Language :: Python', + 'Topic :: Software Development :: Build Tools', + 'Topic :: Software Development :: Embedded Systems', + 'Topic :: System :: Archiving :: Packaging', + 'Topic :: System :: Software Distribution', ], - author='Jannis Pohlmann', - author_email='jannis.pohlmann@codethink.co.uk', + author='Codethink Limited', + author_email='baserock-dev@baserock.org', url='http://www.baserock.org/', - scripts=['morph-cache-server'], - packages=['morphcacheserver'], - ) + scripts=['morph', 'distbuild-helper', 'morph-cache-server'], + packages=['morphlib', 'morphlib.plugins', 'distbuild', + 'morphcacheserver'], + package_data={ + 'morphlib': [ + 'xfer-hole', + 'recv-hole', + 'exts/*', + 'version', + 'commit', + 'tree', + 'ref', + ] + }, + data_files=[('share/man/man1', glob.glob('*.[1-8]'))], + cmdclass={ + 'build': GenerateResources, + 'check': Check, + 'clean': Clean, + }) diff --git a/source-stats b/source-stats new file mode 100755 index 00000000..811fdc3b --- /dev/null +++ b/source-stats @@ -0,0 +1,143 @@ +#!/usr/bin/python +# Copyright (C) 2012,2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import cliapp +import csv +import os +import shutil +import sys +import tarfile +import tempfile +import time + + +class SourceStats(cliapp.Application): + + '''Compute some basic statistics about Baserock components. + + * name of component + * total source lines, excluding blank lines + * number of commits over last 12 months + * lines added over 12 months + * lines removed over 12 months + + Usage: ./source-stat $HOME/baserock/gits/* + + ''' + + def add_settings(self): + self.settings.string(['gitsdir'], 'base directory for git repos') + + def setup(self): + self.writer = csv.writer(sys.stdout) + self.cols = ['name', 'lines', 'commits', 'added', 'deleted'] + self.writer.writerow(self.cols) + + def process_input(self, gitdir): + name = os.path.basename(gitdir) + stats = self.compute_stats(name, gitdir) + row = [stats[x] for x in self.cols] + self.writer.writerow(row) + sys.stdout.flush() + + def compute_stats(self, name, gitdir): + stats = { + 'name': name, + } + + t = time.time() - 365 * 86400 + tt = time.localtime(t) + start_date = time.strftime('%Y-%m-%d', tt) + + stats['branch'] = self.pick_branch(gitdir) + + self.get_sources(gitdir, stats['branch']) + + stats['lines'] = self.count_source_lines(gitdir) + + start, end = self.find_commit_range(gitdir, start_date) + stats['commits'] = self.count_commits(gitdir, start, end) + stats['added'], stats['deleted'] = self.diffstat(gitdir, start, end) + + return stats + + def pick_branch(self, gitdir): + out = self.runcmd(['git', 'branch', '-r'], cwd=gitdir) + lines = [x.split()[-1] for x in out.splitlines()] + + candidates = [ + 'origin/master', + 'origin/trunk', + 'origin/blead', + ] + + for x in candidates: + if x in lines: + return x + raise Exception('Cannot decide on branch in %s' % gitdir) + + def get_sources(self, gitdir, branch): + self.runcmd(['git', 'checkout', branch], cwd=gitdir) + + def count_source_lines(self, tempdir): + numlines = 0 + for dirname, subdirs, basenames in os.walk(tempdir): + if '.git' in subdirs: + subdirs.remove('.git') + + for basename in basenames: + filename = os.path.join(dirname, basename) + if os.path.isfile(filename) and not os.path.islink(filename): + with open(filename) as f: + for line in f: + if line.strip(): + numlines += 1 + + return numlines + + def find_commit_range(self, gitdir, start_date): + out = self.runcmd(['git', 'log', '--format=oneline', + '--since=%s' % start_date], + cwd=gitdir) + lines = out.splitlines() + if len(lines) < 2: + return 'HEAD', 'HEAD' + end = lines[0].split()[0] + start = lines[-1].split()[0] + return start, end + + def count_commits(self, gitdir, start, end): + out = self.runcmd(['git', 'log', '--format=oneline', + '%s..%s' % (start, end)], + cwd=gitdir) + return len(out.splitlines()) + + def diffstat(self, gitdir, start, end): + out = self.runcmd(['git', 'diff', '--numstat', start, end], + cwd=gitdir) + tuples = [line.split() for line in out.splitlines()] + + def toint(s): + try: + return int(s) + except ValueError: + return 0 + added = sum(toint(t[0]) for t in tuples) + deleted = sum(toint(t[1]) for t in tuples) + return added, deleted + +SourceStats().run() diff --git a/tests.branching/add-then-edit.script b/tests.branching/add-then-edit.script new file mode 100755 index 00000000..be3315d9 --- /dev/null +++ b/tests.branching/add-then-edit.script @@ -0,0 +1,51 @@ +#!/bin/sh +# +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test the workflow of adding a new chunk to a stratum then editing it + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs "me/add-then-edit" + +cd "me/add-then-edit" + +# add a chunk +cd test/morphs + +python -c 'import yaml +with open("hello-stratum.morph", "r") as f: + stratum = yaml.load(f) +stratum["chunks"].append({ + "build-depends": [], + "name": "goodbye", + "ref": "master", + "repo": "test:goodbye", +}) +with open("hello-stratum.morph", "w") as f: + yaml.dump(stratum, f) +' + +"$SRCDIR/scripts/test-morph" edit goodbye + +# check whether the stratum still contains the goodbye chunk +grep -qFe goodbye hello-stratum.morph + +# check whether edit has cloned the repository to the right branch +git --git-dir="../goodbye/.git" rev-parse --abbrev-ref HEAD diff --git a/tests.branching/add-then-edit.setup b/tests.branching/add-then-edit.setup new file mode 100755 index 00000000..bb58d05a --- /dev/null +++ b/tests.branching/add-then-edit.setup @@ -0,0 +1,37 @@ +#!/bin/sh +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +set -eu + +# Create goodbye chunk +mkdir "$DATADIR/goodbye" +cd "$DATADIR/goodbye" + +cat >goodbye <<'EOF' +#!/bin/sh +echo goodbye +EOF +chmod +x goodbye + +cat >goodbye.morph <<'EOF' +name: goodbye +kind: chunk +install-commands: +- install goodbye "$DESTDIR$PREFIX/bin/goodbye" +EOF +git init . +git add goodbye.morph goodbye +git commit -m "Initial commit" diff --git a/tests.branching/add-then-edit.stdout b/tests.branching/add-then-edit.stdout new file mode 100644 index 00000000..e0950ab5 --- /dev/null +++ b/tests.branching/add-then-edit.stdout @@ -0,0 +1 @@ +me/add-then-edit diff --git a/tests.branching/branch-cleans-up-on-failure.script b/tests.branching/branch-cleans-up-on-failure.script new file mode 100755 index 00000000..55666137 --- /dev/null +++ b/tests.branching/branch-cleans-up-on-failure.script @@ -0,0 +1,30 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## If a command fails, the state of the workspace should be as if the command +## was never run + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init + +# This will fail because we're trying to branch off a ref that doesn't exist +"$SRCDIR/scripts/test-morph" branch test:morphs foo/bar invalid-ref || true + +[ ! -d "$DATADIR/workspace/foo" ] diff --git a/tests.branching/branch-cleans-up-on-failure.stderr b/tests.branching/branch-cleans-up-on-failure.stderr new file mode 100644 index 00000000..37533408 --- /dev/null +++ b/tests.branching/branch-cleans-up-on-failure.stderr @@ -0,0 +1 @@ +ERROR: Ref invalid-ref is an invalid reference for repo file://TMP/morphs diff --git a/tests.branching/branch-creates-new-system-branch-not-from-master.script b/tests.branching/branch-creates-new-system-branch-not-from-master.script new file mode 100755 index 00000000..c561f191 --- /dev/null +++ b/tests.branching/branch-creates-new-system-branch-not-from-master.script @@ -0,0 +1,38 @@ +#!/bin/sh +# +# Copyright (C) 2012,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Make sure "morph branch" creates a new system branch. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init + +"$SRCDIR/scripts/test-morph" branch test:morphs newbranch alfred + +echo "File tree:" +"$SRCDIR/scripts/list-tree" . | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' + +echo "Current branches:" +"$SRCDIR/scripts/run-git-in" newbranch/test/morphs branch + +echo "Current origin:" +"$SRCDIR/scripts/run-git-in" newbranch/test/morphs remote show origin | + sed 's,\(TMP/workspace/\.morph/cache/gits/file_\).*_,\1,g' diff --git a/tests.branching/branch-creates-new-system-branch-not-from-master.stdout b/tests.branching/branch-creates-new-system-branch-not-from-master.stdout new file mode 100644 index 00000000..c61624b4 --- /dev/null +++ b/tests.branching/branch-creates-new-system-branch-not-from-master.stdout @@ -0,0 +1,27 @@ +File tree: +d . +d ./.morph +d ./newbranch +d ./newbranch/.morph-system-branch +d ./newbranch/test +d ./newbranch/test/morphs +d ./newbranch/test/morphs/.git +f ./newbranch/.morph-system-branch/config +f ./newbranch/test/morphs/hello-stratum.morph +f ./newbranch/test/morphs/hello-system.morph +f ./newbranch/test/morphs/this.is.alfred +Current branches: + alfred +* newbranch +Current origin: +* remote origin + Fetch URL: file://TMP/morphs + Push URL: file://TMP/morphs + HEAD branch: master + Remote branches: + alfred tracked + master tracked + Local branch configured for 'git pull': + alfred merges with remote alfred + Local ref configured for 'git push': + alfred pushes to alfred (up to date) diff --git a/tests.branching/branch-creates-new-system-branch.script b/tests.branching/branch-creates-new-system-branch.script new file mode 100755 index 00000000..784bed62 --- /dev/null +++ b/tests.branching/branch-creates-new-system-branch.script @@ -0,0 +1,38 @@ +#!/bin/sh +# +# Copyright (C) 2012,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Make sure "morph branch" creates a new system branch. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init + +"$SRCDIR/scripts/test-morph" branch test:morphs newbranch + +echo "File tree:" +"$SRCDIR/scripts/list-tree" . | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' + +echo "Current branches:" +"$SRCDIR/scripts/run-git-in" newbranch/test/morphs branch + +echo "Current origin:" +"$SRCDIR/scripts/run-git-in" newbranch/test/morphs remote show origin | + sed 's,\(TMP/workspace/\.morph/cache/gits/file_\).*_,\1,g' diff --git a/tests.branching/branch-creates-new-system-branch.stdout b/tests.branching/branch-creates-new-system-branch.stdout new file mode 100644 index 00000000..a7318378 --- /dev/null +++ b/tests.branching/branch-creates-new-system-branch.stdout @@ -0,0 +1,26 @@ +File tree: +d . +d ./.morph +d ./newbranch +d ./newbranch/.morph-system-branch +d ./newbranch/test +d ./newbranch/test/morphs +d ./newbranch/test/morphs/.git +f ./newbranch/.morph-system-branch/config +f ./newbranch/test/morphs/hello-stratum.morph +f ./newbranch/test/morphs/hello-system.morph +Current branches: + master +* newbranch +Current origin: +* remote origin + Fetch URL: file://TMP/morphs + Push URL: file://TMP/morphs + HEAD branch: master + Remote branches: + alfred tracked + master tracked + Local branch configured for 'git pull': + master merges with remote master + Local ref configured for 'git push': + master pushes to master (up to date) diff --git a/tests.branching/branch-fails-if-branch-exists.script b/tests.branching/branch-fails-if-branch-exists.script new file mode 100755 index 00000000..8a7da8ab --- /dev/null +++ b/tests.branching/branch-fails-if-branch-exists.script @@ -0,0 +1,41 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Make sure "morph branch" fails if the system branch already exists in the +## branch root (morphologies repo). + +set -eu + +cd "$DATADIR/morphs" +git checkout --quiet -b baserock/existing-branch + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init + +# We create a valid branch inside the same prefix first so we can check it +# doesn't get caught up in the deletion of the invalid branch directory + +"$SRCDIR/scripts/test-morph" branch test:morphs baserock/new-branch + +[ -d "$DATADIR/workspace/baserock/new-branch" ] + +"$SRCDIR/scripts/test-morph" branch test:morphs \ + baserock/existing-branch || true + +[ -d "$DATADIR/workspace/baserock/new-branch" ] +[ ! -d "$DATADIR/workspace/baserock/existing-branch" ] diff --git a/tests.branching/branch-fails-if-branch-exists.stderr b/tests.branching/branch-fails-if-branch-exists.stderr new file mode 100644 index 00000000..5b4bc943 --- /dev/null +++ b/tests.branching/branch-fails-if-branch-exists.stderr @@ -0,0 +1 @@ +ERROR: branch baserock/existing-branch already exists in repository test:morphs diff --git a/tests.branching/branch-when-branchdir-exists-locally.exit b/tests.branching/branch-when-branchdir-exists-locally.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.branching/branch-when-branchdir-exists-locally.exit @@ -0,0 +1 @@ +1 diff --git a/tests.branching/branch-when-branchdir-exists-locally.script b/tests.branching/branch-when-branchdir-exists-locally.script new file mode 100755 index 00000000..66a116be --- /dev/null +++ b/tests.branching/branch-when-branchdir-exists-locally.script @@ -0,0 +1,29 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Make sure "morph branch" fails when the system branch directory already +## exists. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init + +mkdir newbranch +"$SRCDIR/scripts/test-morph" branch test:morphs newbranch + diff --git a/tests.branching/branch-when-branchdir-exists-locally.stderr b/tests.branching/branch-when-branchdir-exists-locally.stderr new file mode 100644 index 00000000..e178cf2c --- /dev/null +++ b/tests.branching/branch-when-branchdir-exists-locally.stderr @@ -0,0 +1 @@ +ERROR: TMP/workspace/newbranch: File exists diff --git a/tests.branching/branch-works-anywhere.script b/tests.branching/branch-works-anywhere.script new file mode 100755 index 00000000..7f6156ce --- /dev/null +++ b/tests.branching/branch-works-anywhere.script @@ -0,0 +1,62 @@ +#!/bin/bash +# +# Copyright (C) 2012,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Make sure "morph branch" works anywhere in a workspace or system branch. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init + +# First, create a branch. +"$SRCDIR/scripts/test-morph" branch test:morphs branch1 + +echo "Workspace after creating the first branch:" +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' + +# Now, create a nother branch from the workspace. +"$SRCDIR/scripts/test-morph" branch test:morphs branch2 + +echo "Workspace after creating the second branch:" +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' + +# Now, enter the first branch and create a third branch, which +# should not be created in the working directory but in the +# workspace directory. +cd "$DATADIR/workspace/branch1" +"$SRCDIR/scripts/test-morph" branch test:morphs branch3 + +echo "Workspace after creating the third branch:" +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' + +# Now, go into the morphs repository of that third branch and +# create a fourth system branch from in there. This, too, should +# end up being created in the toplevel workspace directory. +cd "$DATADIR/workspace/branch3/test/morphs" +"$SRCDIR/scripts/test-morph" branch test:morphs branch4 + +echo "Workspace after creating the fourth branch:" +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' diff --git a/tests.branching/branch-works-anywhere.stdout b/tests.branching/branch-works-anywhere.stdout new file mode 100644 index 00000000..4e317902 --- /dev/null +++ b/tests.branching/branch-works-anywhere.stdout @@ -0,0 +1,92 @@ +Workspace after creating the first branch: +d . +d ./.morph +d ./branch1 +d ./branch1/.morph-system-branch +d ./branch1/test +d ./branch1/test/morphs +d ./branch1/test/morphs/.git +f ./branch1/.morph-system-branch/config +f ./branch1/test/morphs/hello-stratum.morph +f ./branch1/test/morphs/hello-system.morph +Workspace after creating the second branch: +d . +d ./.morph +d ./branch1 +d ./branch1/.morph-system-branch +d ./branch1/test +d ./branch1/test/morphs +d ./branch1/test/morphs/.git +d ./branch2 +d ./branch2/.morph-system-branch +d ./branch2/test +d ./branch2/test/morphs +d ./branch2/test/morphs/.git +f ./branch1/.morph-system-branch/config +f ./branch1/test/morphs/hello-stratum.morph +f ./branch1/test/morphs/hello-system.morph +f ./branch2/.morph-system-branch/config +f ./branch2/test/morphs/hello-stratum.morph +f ./branch2/test/morphs/hello-system.morph +Workspace after creating the third branch: +d . +d ./.morph +d ./branch1 +d ./branch1/.morph-system-branch +d ./branch1/test +d ./branch1/test/morphs +d ./branch1/test/morphs/.git +d ./branch2 +d ./branch2/.morph-system-branch +d ./branch2/test +d ./branch2/test/morphs +d ./branch2/test/morphs/.git +d ./branch3 +d ./branch3/.morph-system-branch +d ./branch3/test +d ./branch3/test/morphs +d ./branch3/test/morphs/.git +f ./branch1/.morph-system-branch/config +f ./branch1/test/morphs/hello-stratum.morph +f ./branch1/test/morphs/hello-system.morph +f ./branch2/.morph-system-branch/config +f ./branch2/test/morphs/hello-stratum.morph +f ./branch2/test/morphs/hello-system.morph +f ./branch3/.morph-system-branch/config +f ./branch3/test/morphs/hello-stratum.morph +f ./branch3/test/morphs/hello-system.morph +Workspace after creating the fourth branch: +d . +d ./.morph +d ./branch1 +d ./branch1/.morph-system-branch +d ./branch1/test +d ./branch1/test/morphs +d ./branch1/test/morphs/.git +d ./branch2 +d ./branch2/.morph-system-branch +d ./branch2/test +d ./branch2/test/morphs +d ./branch2/test/morphs/.git +d ./branch3 +d ./branch3/.morph-system-branch +d ./branch3/test +d ./branch3/test/morphs +d ./branch3/test/morphs/.git +d ./branch4 +d ./branch4/.morph-system-branch +d ./branch4/test +d ./branch4/test/morphs +d ./branch4/test/morphs/.git +f ./branch1/.morph-system-branch/config +f ./branch1/test/morphs/hello-stratum.morph +f ./branch1/test/morphs/hello-system.morph +f ./branch2/.morph-system-branch/config +f ./branch2/test/morphs/hello-stratum.morph +f ./branch2/test/morphs/hello-system.morph +f ./branch3/.morph-system-branch/config +f ./branch3/test/morphs/hello-stratum.morph +f ./branch3/test/morphs/hello-system.morph +f ./branch4/.morph-system-branch/config +f ./branch4/test/morphs/hello-stratum.morph +f ./branch4/test/morphs/hello-system.morph diff --git a/tests.branching/checkout-cleans-up-on-failure.script b/tests.branching/checkout-cleans-up-on-failure.script new file mode 100755 index 00000000..a0b0411b --- /dev/null +++ b/tests.branching/checkout-cleans-up-on-failure.script @@ -0,0 +1,29 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## If a command fails, the state of the workspace should be as if the command +## was never run + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init + +"$SRCDIR/scripts/test-morph" checkout test:morphs i/do/not/exist || true + +[ ! -d "$DATADIR/workspace/i" ] diff --git a/tests.branching/checkout-cleans-up-on-failure.stderr b/tests.branching/checkout-cleans-up-on-failure.stderr new file mode 100644 index 00000000..5b6a5645 --- /dev/null +++ b/tests.branching/checkout-cleans-up-on-failure.stderr @@ -0,0 +1 @@ +ERROR: Ref i/do/not/exist is an invalid reference for repo file://TMP/morphs diff --git a/tests.branching/checkout-existing-branch.script b/tests.branching/checkout-existing-branch.script new file mode 100755 index 00000000..b1740d9c --- /dev/null +++ b/tests.branching/checkout-existing-branch.script @@ -0,0 +1,33 @@ +#!/bin/sh +# +# Copyright (C) 2012,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that "morph checkout test:morphs master" works. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" checkout test:morphs master + +echo "File tree:" +"$SRCDIR/scripts/list-tree" . | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' + +echo "Current branches:" +"$SRCDIR/scripts/run-git-in" master/test/morphs branch diff --git a/tests.branching/checkout-existing-branch.stdout b/tests.branching/checkout-existing-branch.stdout new file mode 100644 index 00000000..a6026269 --- /dev/null +++ b/tests.branching/checkout-existing-branch.stdout @@ -0,0 +1,13 @@ +File tree: +d . +d ./.morph +d ./master +d ./master/.morph-system-branch +d ./master/test +d ./master/test/morphs +d ./master/test/morphs/.git +f ./master/.morph-system-branch/config +f ./master/test/morphs/hello-stratum.morph +f ./master/test/morphs/hello-system.morph +Current branches: +* master diff --git a/tests.branching/checkout-non-aliased-repos.script b/tests.branching/checkout-non-aliased-repos.script new file mode 100755 index 00000000..b52f6675 --- /dev/null +++ b/tests.branching/checkout-non-aliased-repos.script @@ -0,0 +1,52 @@ +#!/bin/bash +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that "morph checkout" works with repos that are not aliased. +## This test in particular verifies that URI schemes are stripped off +## and that the .git suffix is only removed at the end if it is actually +## present. + +set -eu + +REPO_WITH_SUFFIX="file://$DATADIR/morphs.git" +REPO_WITHOUT_SUFFIX="file://$DATADIR/morphs" + +TEMP_DIR=$(dirname "$DATADIR") + +cd "$DATADIR/workspace" + +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" checkout "$REPO_WITH_SUFFIX" master + +test -d "$DATADIR/workspace/master/$DATADIR/morphs" + +echo "Current branches of repo with suffix:" +"$SRCDIR/scripts/run-git-in" master/"${DATADIR:1}"/morphs branch + +cd "$DATADIR" +rm -rf "$DATADIR/workspace" +mkdir "$DATADIR/workspace" +cd "$DATADIR/workspace" + +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" checkout "$REPO_WITHOUT_SUFFIX" master + +test -d "$DATADIR/workspace/master/$DATADIR/morphs" + +echo "Current branches of repo without suffix:" +"$SRCDIR/scripts/run-git-in" master/"${DATADIR:1}"/morphs branch diff --git a/tests.branching/checkout-non-aliased-repos.stdout b/tests.branching/checkout-non-aliased-repos.stdout new file mode 100644 index 00000000..2d056c2f --- /dev/null +++ b/tests.branching/checkout-non-aliased-repos.stdout @@ -0,0 +1,4 @@ +Current branches of repo with suffix: +* master +Current branches of repo without suffix: +* master diff --git a/tests.branching/checkout-works-anywhere.script b/tests.branching/checkout-works-anywhere.script new file mode 100755 index 00000000..14d18842 --- /dev/null +++ b/tests.branching/checkout-works-anywhere.script @@ -0,0 +1,50 @@ +#!/bin/bash +# +# Copyright (C) 2012,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Make sure "morph checkout" works anywhere in a workspace or system branch. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init + +# First, check out the master branch from the workspace directory. +"$SRCDIR/scripts/test-morph" checkout test:morphs master + +echo "Workspace after checking out master from the workspace directory:" +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' + +# Reset the workspace. +cd "$DATADIR" +rm -rf workspace +mkdir workspace +cd workspace +"$SRCDIR/scripts/test-morph" init + +# This time, create a new branch and check out the master branch +# from within that branch. +"$SRCDIR/scripts/test-morph" branch test:morphs newbranch +cd newbranch/test/morphs +"$SRCDIR/scripts/test-morph" checkout test:morphs master + +echo "Workspace after checking out master from within a new branch:" +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' diff --git a/tests.branching/checkout-works-anywhere.stdout b/tests.branching/checkout-works-anywhere.stdout new file mode 100644 index 00000000..ed8b1567 --- /dev/null +++ b/tests.branching/checkout-works-anywhere.stdout @@ -0,0 +1,30 @@ +Workspace after checking out master from the workspace directory: +d . +d ./.morph +d ./master +d ./master/.morph-system-branch +d ./master/test +d ./master/test/morphs +d ./master/test/morphs/.git +f ./master/.morph-system-branch/config +f ./master/test/morphs/hello-stratum.morph +f ./master/test/morphs/hello-system.morph +Workspace after checking out master from within a new branch: +d . +d ./.morph +d ./master +d ./master/.morph-system-branch +d ./master/test +d ./master/test/morphs +d ./master/test/morphs/.git +d ./newbranch +d ./newbranch/.morph-system-branch +d ./newbranch/test +d ./newbranch/test/morphs +d ./newbranch/test/morphs/.git +f ./master/.morph-system-branch/config +f ./master/test/morphs/hello-stratum.morph +f ./master/test/morphs/hello-system.morph +f ./newbranch/.morph-system-branch/config +f ./newbranch/test/morphs/hello-stratum.morph +f ./newbranch/test/morphs/hello-system.morph diff --git a/tests.branching/edit-checkouts-existing-chunk.script b/tests.branching/edit-checkouts-existing-chunk.script new file mode 100755 index 00000000..df2a7d85 --- /dev/null +++ b/tests.branching/edit-checkouts-existing-chunk.script @@ -0,0 +1,37 @@ +#!/bin/sh +# +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that "morph edit" clones a chunk repository into a system branch. + +set -eu + +# Checkout the master system branch. +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" checkout test:morphs alfred + +# Edit the hello chunk in alfred. +cd "alfred" +"$SRCDIR/scripts/test-morph" edit hello + +echo "Current branches:" +"$SRCDIR/scripts/test-morph" foreach git branch + +echo +echo "Files in hello:" +ls "$DATADIR/workspace/alfred/test/hello" diff --git a/tests.branching/edit-checkouts-existing-chunk.stdout b/tests.branching/edit-checkouts-existing-chunk.stdout new file mode 100644 index 00000000..f6ac79c2 --- /dev/null +++ b/tests.branching/edit-checkouts-existing-chunk.stdout @@ -0,0 +1,11 @@ +Current branches: +test:hello +* alfred + master + +test:morphs +* alfred + + +Files in hello: +hello.morph diff --git a/tests.branching/edit-clones-chunk.script b/tests.branching/edit-clones-chunk.script new file mode 100755 index 00000000..a6313ca6 --- /dev/null +++ b/tests.branching/edit-clones-chunk.script @@ -0,0 +1,37 @@ +#!/bin/sh +# +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that "morph edit" clones a chunk repository into a system branch. + +set -eu + +# Create system branch. +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs newbranch + +# Edit chunk. +"$SRCDIR/scripts/test-morph" edit hello + +echo "Current branches:" +"$SRCDIR/scripts/test-morph" foreach git branch + +echo +echo "Current origins:" +"$SRCDIR/scripts/test-morph" foreach git remote show origin | \ + sed 's,\(TMP/workspace/\.morph/cache/gits/file_\).*_,\1,g' diff --git a/tests.branching/edit-clones-chunk.stdout b/tests.branching/edit-clones-chunk.stdout new file mode 100644 index 00000000..d0bcb565 --- /dev/null +++ b/tests.branching/edit-clones-chunk.stdout @@ -0,0 +1,39 @@ +Current branches: +test:hello + master +* newbranch + +test:morphs + master +* newbranch + + +Current origins: +test:hello +* remote origin + Fetch URL: file://TMP/hello + Push URL: file://TMP/hello + HEAD branch (remote HEAD is ambiguous, may be one of the following): + alfred + master + Remote branches: + alfred tracked + master tracked + Local branch configured for 'git pull': + master merges with remote master + Local ref configured for 'git push': + master pushes to master (up to date) + +test:morphs +* remote origin + Fetch URL: file://TMP/morphs + Push URL: file://TMP/morphs + HEAD branch: master + Remote branches: + alfred tracked + master tracked + Local branch configured for 'git pull': + master merges with remote master + Local ref configured for 'git push': + master pushes to master (up to date) + diff --git a/tests.branching/edit-handles-submodules.script b/tests.branching/edit-handles-submodules.script new file mode 100755 index 00000000..09592f74 --- /dev/null +++ b/tests.branching/edit-handles-submodules.script @@ -0,0 +1,33 @@ +#!/bin/sh +# +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## 'morph edit' should set up git URL rewriting correctly so that submodule +## commands function as usual, despite our prefixing and mirroring. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs newbranch + +# Submodules should be set up automatically +"$SRCDIR/scripts/test-morph" edit hello + +cd "$DATADIR/workspace/newbranch/test/hello" +[ -e foolib/README ] + diff --git a/tests.branching/edit-handles-submodules.setup b/tests.branching/edit-handles-submodules.setup new file mode 100755 index 00000000..cb61ad66 --- /dev/null +++ b/tests.branching/edit-handles-submodules.setup @@ -0,0 +1,40 @@ +#!/bin/sh +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +set -eu + +# Create a dummy submodule +mkdir "$DATADIR/foolib" +cd "$DATADIR/foolib" + +echo "Thanks" > README +git init . +git add README +git commit -m "Initial commit" + +# Use this in hello chunk +cd "$DATADIR/hello" +git submodule add "$DATADIR/foolib" foolib/ +git commit -m "Use Foolib submodule" + +# Rewrite the URL, as we would do in Trove +cat <<EOF > "$DATADIR/hello/.gitmodules" +[submodule "foolib"] + path = foolib + url = test:foolib +EOF +git add .gitmodules +git commit -m "Use Foolib from test: prefix" diff --git a/tests.branching/edit-updates-stratum.script b/tests.branching/edit-updates-stratum.script new file mode 100755 index 00000000..b60c46e7 --- /dev/null +++ b/tests.branching/edit-updates-stratum.script @@ -0,0 +1,32 @@ +#!/bin/sh +# +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that "morph edit" clones a chunk repository into a system branch. + +set -eu + +# Create system branch. +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs newbranch + +# Edit chunk. +"$SRCDIR/scripts/test-morph" edit hello + +# See what effect the editing had. +"$SRCDIR/scripts/run-git-in" "newbranch/test/morphs" diff diff --git a/tests.branching/edit-updates-stratum.stdout b/tests.branching/edit-updates-stratum.stdout new file mode 100644 index 00000000..ee9510b5 --- /dev/null +++ b/tests.branching/edit-updates-stratum.stdout @@ -0,0 +1,13 @@ +diff --git a/hello-stratum.morph b/hello-stratum.morph +index f335879..7bf9d37 100644 +--- a/hello-stratum.morph ++++ b/hello-stratum.morph +@@ -3,6 +3,7 @@ kind: stratum + chunks: + - name: hello + repo: test:hello +- ref: master ++ ref: newbranch ++ unpetrify-ref: master + build-depends: [] + build-mode: bootstrap diff --git a/tests.branching/edit-works-after-branch-root-was-renamed.script b/tests.branching/edit-works-after-branch-root-was-renamed.script new file mode 100755 index 00000000..e28ab7df --- /dev/null +++ b/tests.branching/edit-works-after-branch-root-was-renamed.script @@ -0,0 +1,42 @@ +#!/bin/sh +# +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that the branch root repository created by "morph branch" or +## "morph checkout" can be renamed and "morph edit" still finds the +## branch root repo and works. + +set -eu + +# FIXME: This test is disabled, because a) it's a corner case and b) Lars +# ran out of time to implement support for it. +cat "$SRCDIR/tests.branching/edit-works-after-branch-root-was-renamed.stdout" +exit 0 + +cd "$DATADIR/workspace" + +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" checkout test:morphs master + +cd "$DATADIR/workspace/master" +mv test:morphs my-renamed-morphs + +"$SRCDIR/scripts/test-morph" edit hello + +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" | grep -v '/\.git/' | + sed 's,/cache/gits/file_[^/]*_,/cache/gits/file_,' | + grep -v 'cache/gits/file_[^/]*/' diff --git a/tests.branching/edit-works-after-branch-root-was-renamed.stdout b/tests.branching/edit-works-after-branch-root-was-renamed.stdout new file mode 100644 index 00000000..f15fe30a --- /dev/null +++ b/tests.branching/edit-works-after-branch-root-was-renamed.stdout @@ -0,0 +1,12 @@ +d . +d ./.morph +d ./master +d ./master/.morph-system-branch +d ./master/my-renamed-morphs +d ./master/my-renamed-morphs/.git +d ./master/test:hello +d ./master/test:hello/.git +f ./master/.morph-system-branch/config +f ./master/my-renamed-morphs/hello-stratum.morph +f ./master/my-renamed-morphs/hello-system.morph +f ./master/test:hello/hello.morph diff --git a/tests.branching/foreach-handles-command-failure.exit b/tests.branching/foreach-handles-command-failure.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.branching/foreach-handles-command-failure.exit @@ -0,0 +1 @@ +1 diff --git a/tests.branching/foreach-handles-command-failure.script b/tests.branching/foreach-handles-command-failure.script new file mode 100755 index 00000000..4bc71c78 --- /dev/null +++ b/tests.branching/foreach-handles-command-failure.script @@ -0,0 +1,28 @@ +#!/bin/sh +# +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that "morph foreach" deals with failure in a grown-up way + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" checkout test:morphs master +"$SRCDIR/scripts/test-morph" edit hello + +"$SRCDIR/scripts/test-morph" foreach git remote update non-existant-remote diff --git a/tests.branching/foreach-handles-command-failure.stderr b/tests.branching/foreach-handles-command-failure.stderr new file mode 100644 index 00000000..c7b8316b --- /dev/null +++ b/tests.branching/foreach-handles-command-failure.stderr @@ -0,0 +1 @@ +ERROR: Command failed at repo test:hello: git remote update non-existant-remote diff --git a/tests.branching/foreach-handles-command-failure.stdout b/tests.branching/foreach-handles-command-failure.stdout new file mode 100644 index 00000000..d687996d --- /dev/null +++ b/tests.branching/foreach-handles-command-failure.stdout @@ -0,0 +1,2 @@ +test:hello +fatal: No such remote or remote group: non-existant-remote diff --git a/tests.branching/foreach-handles-full-urls.script b/tests.branching/foreach-handles-full-urls.script new file mode 100755 index 00000000..6e0b14ec --- /dev/null +++ b/tests.branching/foreach-handles-full-urls.script @@ -0,0 +1,28 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## morph foreach: should not break if we used a full URL for a repo + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" checkout file://$DATADIR/morphs master + +# This will fail if we get the directory name +"$SRCDIR/scripts/test-morph" foreach -- git status diff --git a/tests.branching/foreach-handles-full-urls.stdout b/tests.branching/foreach-handles-full-urls.stdout new file mode 100644 index 00000000..cee2f70a --- /dev/null +++ b/tests.branching/foreach-handles-full-urls.stdout @@ -0,0 +1,4 @@ +file://TMP/morphs +# On branch master +nothing to commit, working directory clean + diff --git a/tests.branching/init-cwd.script b/tests.branching/init-cwd.script new file mode 100755 index 00000000..10dd0cc7 --- /dev/null +++ b/tests.branching/init-cwd.script @@ -0,0 +1,26 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test that "morph init" works for the current working directory. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init . +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" + diff --git a/tests.branching/init-cwd.stdout b/tests.branching/init-cwd.stdout new file mode 100644 index 00000000..e7922ee1 --- /dev/null +++ b/tests.branching/init-cwd.stdout @@ -0,0 +1,2 @@ +d . +d ./.morph diff --git a/tests.branching/init-default.script b/tests.branching/init-default.script new file mode 100755 index 00000000..da67828f --- /dev/null +++ b/tests.branching/init-default.script @@ -0,0 +1,25 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test that "morph init" works without an explicit argument. + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" diff --git a/tests.branching/init-default.stdout b/tests.branching/init-default.stdout new file mode 100644 index 00000000..e7922ee1 --- /dev/null +++ b/tests.branching/init-default.stdout @@ -0,0 +1,2 @@ +d . +d ./.morph diff --git a/tests.branching/init-existing.script b/tests.branching/init-existing.script new file mode 100755 index 00000000..506e94bb --- /dev/null +++ b/tests.branching/init-existing.script @@ -0,0 +1,25 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test that "morph init" works when given the name of an existing, +## empty directory. + +set -eu + +"$SRCDIR/scripts/test-morph" init "$DATADIR/workspace" +"$SRCDIR/scripts/list-tree" "$DATADIR/workspace" diff --git a/tests.branching/init-existing.stdout b/tests.branching/init-existing.stdout new file mode 100644 index 00000000..e7922ee1 --- /dev/null +++ b/tests.branching/init-existing.stdout @@ -0,0 +1,2 @@ +d . +d ./.morph diff --git a/tests.branching/init-newdir.script b/tests.branching/init-newdir.script new file mode 100755 index 00000000..1f505d92 --- /dev/null +++ b/tests.branching/init-newdir.script @@ -0,0 +1,25 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test that "morph init" works if given a directory that +## does not exist yet. + +set -eu + +"$SRCDIR/scripts/test-morph" init "$DATADIR/foo" +"$SRCDIR/scripts/list-tree" "$DATADIR/foo" diff --git a/tests.branching/init-newdir.stdout b/tests.branching/init-newdir.stdout new file mode 100644 index 00000000..e7922ee1 --- /dev/null +++ b/tests.branching/init-newdir.stdout @@ -0,0 +1,2 @@ +d . +d ./.morph diff --git a/tests.branching/init-nonempty.exit b/tests.branching/init-nonempty.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.branching/init-nonempty.exit @@ -0,0 +1 @@ +1 diff --git a/tests.branching/init-nonempty.script b/tests.branching/init-nonempty.script new file mode 100755 index 00000000..c5c1947c --- /dev/null +++ b/tests.branching/init-nonempty.script @@ -0,0 +1,25 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test that "morph init" fails when given the name of an existing, +## non-empty directory. + +set -eu + +touch "$DATADIR/workspace/foo" +"$SRCDIR/scripts/test-morph" init "$DATADIR/workspace" diff --git a/tests.branching/init-nonempty.stderr b/tests.branching/init-nonempty.stderr new file mode 100644 index 00000000..bc0ef0e1 --- /dev/null +++ b/tests.branching/init-nonempty.stderr @@ -0,0 +1 @@ +ERROR: can only initialize empty directory as a workspace: TMP/workspace diff --git a/tests.branching/morph-repository-stored-in-cloned-repositories.script b/tests.branching/morph-repository-stored-in-cloned-repositories.script new file mode 100755 index 00000000..f60b16ae --- /dev/null +++ b/tests.branching/morph-repository-stored-in-cloned-repositories.script @@ -0,0 +1,49 @@ +#!/bin/sh +# +# Copyright (C) 2012,2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that morph branch/checkout/edit create repositories that have +## a "git config morph.repository" option set so that we can +## identify these repositories later even when the user has renamed or +## moved their local directories. + +set -eu + +cd "$DATADIR/workspace" + +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs newbranch + +echo "morph.repository in branch root repository:" +cd "$DATADIR/workspace/newbranch/test/morphs" +git config morph.repository +echo + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" checkout test:morphs master + +echo "morph.repository in branch root repository of a checkout:" +cd "$DATADIR/workspace/master/test/morphs" +git config morph.repository +echo + +cd "$DATADIR/workspace/master" +"$SRCDIR/scripts/test-morph" edit hello + +echo "morph.repository of an edited repository:" +cd "$DATADIR/workspace/master/test/hello" +git config morph.repository diff --git a/tests.branching/morph-repository-stored-in-cloned-repositories.stdout b/tests.branching/morph-repository-stored-in-cloned-repositories.stdout new file mode 100644 index 00000000..eadcdd19 --- /dev/null +++ b/tests.branching/morph-repository-stored-in-cloned-repositories.stdout @@ -0,0 +1,8 @@ +morph.repository in branch root repository: +test:morphs + +morph.repository in branch root repository of a checkout: +test:morphs + +morph.repository of an edited repository: +test:hello diff --git a/tests.branching/setup b/tests.branching/setup new file mode 100755 index 00000000..a2d23090 --- /dev/null +++ b/tests.branching/setup @@ -0,0 +1,99 @@ +#!/bin/bash +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Set up $DATADIR. +# +# - a morph.conf configuration file +# - an empty morph workspace directory +# - a git repository called "morphs" for fake system, stratum morphologies +# - a git repository calle "hello" for a dummy chunk + +set -eu + +source "$SRCDIR/scripts/fix-committer-info" + +# Create a morph configuration file +cat <<EOF > "$DATADIR/morph.conf" +[config] +repo-alias = test=file://$DATADIR/%s#file://$DATADIR/%s +cachedir = $DATADIR/cache +log = $DATADIR/morph.log +no-distcc = true +quiet = true +EOF + + +# Create an empty directory to be used as a morph workspace +mkdir "$DATADIR/workspace" + + +# Create a fake morphs repository +mkdir "$DATADIR/morphs" + +## Create a link to this repo that has a .git suffix +ln -s "$DATADIR/morphs" "$DATADIR/morphs.git" + +cat <<EOF > "$DATADIR/morphs/hello-system.morph" +name: hello-system +kind: system +arch: $("$SRCDIR/scripts/test-morph" print-architecture) +strata: +- morph: hello-stratum +EOF + +cat <<EOF > "$DATADIR/morphs/hello-stratum.morph" +name: hello-stratum +kind: stratum +chunks: +- name: hello + repo: test:hello + ref: master + build-depends: [] + build-mode: bootstrap +EOF + +scripts/run-git-in "$DATADIR/morphs" init +scripts/run-git-in "$DATADIR/morphs" add . +scripts/run-git-in "$DATADIR/morphs" commit -m initial + + +# Add an extra branch to the morphs repo. +scripts/run-git-in "$DATADIR/morphs" checkout -b alfred +touch "$DATADIR/morphs/this.is.alfred" +scripts/run-git-in "$DATADIR/morphs" add this.is.alfred +scripts/run-git-in "$DATADIR/morphs" commit --quiet -m 'mark as alfred' +scripts/run-git-in "$DATADIR/morphs" checkout master + + +# Create a dummy chunk repository +mkdir "$DATADIR/hello" + +cat <<EOF > "$DATADIR/hello/hello.morph" +name: hello +kind: chunk +build-system: dummy +EOF + +scripts/run-git-in "$DATADIR/hello" init +scripts/run-git-in "$DATADIR/hello" add . +scripts/run-git-in "$DATADIR/hello" commit -m initial + + +# Add an extra branch to the hello repo. +scripts/run-git-in "$DATADIR/hello" checkout -b alfred +scripts/run-git-in "$DATADIR/hello" checkout master + diff --git a/tests.branching/setup-second-chunk b/tests.branching/setup-second-chunk new file mode 100755 index 00000000..24604ab8 --- /dev/null +++ b/tests.branching/setup-second-chunk @@ -0,0 +1,62 @@ +#!/bin/sh +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Add a second chunk to hello-stratum. + +. "$SRCDIR/scripts/fix-committer-info" + +create_chunk() { + REPO="$1" + NAME="$2" + + mkdir "$1" + ln -s "$1" "$1.git" + cd "$1" + + cat <<EOF > "$1/$2.morph" +build-system: dummy +kind: chunk +name: $2 +EOF + + git init --quiet + git add . + git commit --quiet -m "Initial commit" +} + +create_chunk "$DATADIR/goodbye" "goodbye" + +cd "$DATADIR/morphs" +cat <<EOF > hello-stratum.morph +name: hello-stratum +kind: stratum +chunks: +- name: hello + repo: test:hello + ref: master + build-depends: [] + build-mode: bootstrap +- name: goodbye + repo: test:goodbye + ref: master + build-depends: [] + build-mode: bootstrap +EOF + +git commit -q --all -m "Add goodbye to hello-stratum" + +cd "$DATADIR/workspace" diff --git a/tests.branching/show-system-branch-fails-outside-workspace.exit b/tests.branching/show-system-branch-fails-outside-workspace.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.branching/show-system-branch-fails-outside-workspace.exit @@ -0,0 +1 @@ +1 diff --git a/tests.branching/show-system-branch-fails-outside-workspace.script b/tests.branching/show-system-branch-fails-outside-workspace.script new file mode 100755 index 00000000..d227d5b0 --- /dev/null +++ b/tests.branching/show-system-branch-fails-outside-workspace.script @@ -0,0 +1,33 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Check that 'morph show-system-branch' fails when being run +## outside a workspace. + +set -eu + +# Create a workspace and branch. +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs testbranch + +# Leave the workspace. +cd .. + +# Try to show the current branch. +"$SRCDIR/scripts/test-morph" show-system-branch diff --git a/tests.branching/show-system-branch-fails-outside-workspace.stderr b/tests.branching/show-system-branch-fails-outside-workspace.stderr new file mode 100644 index 00000000..041c64b3 --- /dev/null +++ b/tests.branching/show-system-branch-fails-outside-workspace.stderr @@ -0,0 +1,2 @@ +ERROR: Can't find the workspace directory. +Morph must be built and deployed within the system branch checkout within the workspace directory. diff --git a/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.exit b/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.exit @@ -0,0 +1 @@ +1 diff --git a/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.script b/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.script new file mode 100755 index 00000000..12e23147 --- /dev/null +++ b/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.script @@ -0,0 +1,32 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Check that 'morph show-system-branch' fails when the system branch +## is not obvious. + +set -eu + +# Create a workspace and two system branches +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs first/branch +"$SRCDIR/scripts/test-morph" branch test:morphs second/branch + +# Try to find out the branch from the workspace directory. +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" show-system-branch diff --git a/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.stderr b/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.stderr new file mode 100644 index 00000000..7a0a1b00 --- /dev/null +++ b/tests.branching/show-system-branch-fails-when-branch-is-ambiguous.stderr @@ -0,0 +1,2 @@ +ERROR: Can't find the system branch directory. +Morph must be built and deployed within the system branch checkout. diff --git a/tests.branching/show-system-branch-works-anywhere-with-a-single-branch.script b/tests.branching/show-system-branch-works-anywhere-with-a-single-branch.script new file mode 100755 index 00000000..800a8e5b --- /dev/null +++ b/tests.branching/show-system-branch-works-anywhere-with-a-single-branch.script @@ -0,0 +1,31 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Check that 'morph show-system-branch' works even outside a branch +## if there only is one in the workspcae. + +set -eu + +# Create a workspace and a system branch. +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs first/branch + +# Show the branch even when outside the branch. +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" show-system-branch diff --git a/tests.branching/show-system-branch-works-anywhere-with-a-single-branch.stdout b/tests.branching/show-system-branch-works-anywhere-with-a-single-branch.stdout new file mode 100644 index 00000000..b934ad8e --- /dev/null +++ b/tests.branching/show-system-branch-works-anywhere-with-a-single-branch.stdout @@ -0,0 +1 @@ +first/branch diff --git a/tests.branching/show-system-branch-works-in-different-directories-in-a-branch.script b/tests.branching/show-system-branch-works-in-different-directories-in-a-branch.script new file mode 100755 index 00000000..d89e671c --- /dev/null +++ b/tests.branching/show-system-branch-works-in-different-directories-in-a-branch.script @@ -0,0 +1,57 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Check that 'morph show-system-branch' shows the name of the +## current system branch correctly from various working directories. + +set -eu + +# Create a workspace and two system branches. +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init +"$SRCDIR/scripts/test-morph" branch test:morphs first/branch +"$SRCDIR/scripts/test-morph" branch test:morphs second/branch + +# Create a few subdirectories in the first branch. +mkdir -p "$DATADIR/workspace/first/branch/foo" +mkdir -p "$DATADIR/workspace/first/branch/bar" +mkdir -p "$DATADIR/workspace/first/branch/foo/bar/baz" + +# Show the first branch when partially inside the branch. +cd "$DATADIR/workspace/first" +"$SRCDIR/scripts/test-morph" show-system-branch + +# Show the first branch when inside the main branch directory. +cd "$DATADIR/workspace/first/branch" +"$SRCDIR/scripts/test-morph" show-system-branch + +# Show the first branch when somewhere inside the branch. +cd "$DATADIR/workspace/first/branch/foo" +"$SRCDIR/scripts/test-morph" show-system-branch + +# Show the first branch when somewhere else inside the branch. +cd "$DATADIR/workspace/first/branch/foo/bar/baz" +"$SRCDIR/scripts/test-morph" show-system-branch + +# Show the second branch when partially inside the branch. +cd "$DATADIR/workspace/second" +"$SRCDIR/scripts/test-morph" show-system-branch + +# Show the second branch when inside the main branch directory. +cd "$DATADIR/workspace/second/branch" +"$SRCDIR/scripts/test-morph" show-system-branch diff --git a/tests.branching/show-system-branch-works-in-different-directories-in-a-branch.stdout b/tests.branching/show-system-branch-works-in-different-directories-in-a-branch.stdout new file mode 100644 index 00000000..f9cc3aec --- /dev/null +++ b/tests.branching/show-system-branch-works-in-different-directories-in-a-branch.stdout @@ -0,0 +1,6 @@ +first/branch +first/branch +first/branch +first/branch +second/branch +second/branch diff --git a/tests.branching/status-in-clean-branch.script b/tests.branching/status-in-clean-branch.script new file mode 100755 index 00000000..335db9f9 --- /dev/null +++ b/tests.branching/status-in-clean-branch.script @@ -0,0 +1,27 @@ +#!/bin/sh +# +# Copyright (C) 2011, 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## 'morph status' within a branch + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init . +"$SRCDIR/scripts/test-morph" branch test:morphs test + +"$SRCDIR/scripts/test-morph" status diff --git a/tests.branching/status-in-clean-branch.stdout b/tests.branching/status-in-clean-branch.stdout new file mode 100644 index 00000000..684ab9c9 --- /dev/null +++ b/tests.branching/status-in-clean-branch.stdout @@ -0,0 +1,3 @@ +On branch test, root test:morphs + +No repos have outstanding changes. diff --git a/tests.branching/status-in-dirty-branch.script b/tests.branching/status-in-dirty-branch.script new file mode 100755 index 00000000..37fca97b --- /dev/null +++ b/tests.branching/status-in-dirty-branch.script @@ -0,0 +1,48 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## 'morph status' within a branch + +set -eu + +# FIXME: This is disabled, since a) we haven't decided if we really +# want to support system and stratum morphologies in different git +# repos, and b) the rewritten "morph edit" thus doesn't support it, +# since writing the code is not necessarily simple if one wants to +# cover all corner cases. +cat "$SRCDIR/tests.branching/status-in-dirty-branch.stdout" +exit 0 + +. "$SRCDIR/scripts/setup-3rd-party-strata" + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" branch test:morphs branch1 + +# Make the branch have some interesting changes and pitfalls +cd branch1 +"$SRCDIR/scripts/test-morph" edit hello + +cd test/stratum2-hello +git checkout -q master + +cd .. +mkdir red-herring +cd red-herring +git init -q . + +"$SRCDIR/scripts/test-morph" status diff --git a/tests.branching/status-in-dirty-branch.stdout b/tests.branching/status-in-dirty-branch.stdout new file mode 100644 index 00000000..bee47eaa --- /dev/null +++ b/tests.branching/status-in-dirty-branch.stdout @@ -0,0 +1,5 @@ +On branch branch1, root test:morphs + test:morphs: uncommitted changes + TMP/workspace/branch1/red-herring: not part of system branch + test:external-strata: uncommitted changes + test:stratum2-hello: unexpected ref checked out "master" diff --git a/tests.branching/status-in-workspace.script b/tests.branching/status-in-workspace.script new file mode 100755 index 00000000..e998c097 --- /dev/null +++ b/tests.branching/status-in-workspace.script @@ -0,0 +1,30 @@ +#!/bin/sh +# +# Copyright (C) 2011, 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## 'morph status' within a workspace + +set -eu + +cd "$DATADIR/workspace" +"$SRCDIR/scripts/test-morph" init . +"$SRCDIR/scripts/test-morph" checkout test:morphs master +"$SRCDIR/scripts/test-morph" branch test:morphs a/b/c/d/e/foo +"$SRCDIR/scripts/test-morph" branch test:morphs a/b/c/d/e/bar +mkdir a/b/c/red-herring + +"$SRCDIR/scripts/test-morph" status diff --git a/tests.branching/status-in-workspace.stdout b/tests.branching/status-in-workspace.stdout new file mode 100644 index 00000000..15958736 --- /dev/null +++ b/tests.branching/status-in-workspace.stdout @@ -0,0 +1,4 @@ +System branches in current workspace: + a/b/c/d/e/bar + a/b/c/d/e/foo + master diff --git a/tests.branching/teardown b/tests.branching/teardown new file mode 100755 index 00000000..94928416 --- /dev/null +++ b/tests.branching/teardown @@ -0,0 +1,22 @@ +#!/bin/sh +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Clean up $DATADIR. + +set -eu + +find "$DATADIR" -mindepth 1 -delete diff --git a/tests.branching/workspace-not-found.exit b/tests.branching/workspace-not-found.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.branching/workspace-not-found.exit @@ -0,0 +1 @@ +1 diff --git a/tests.branching/workspace-not-found.script b/tests.branching/workspace-not-found.script new file mode 100755 index 00000000..9e9b5d75 --- /dev/null +++ b/tests.branching/workspace-not-found.script @@ -0,0 +1,23 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Handle being run from outside workspace directory. + +scripts/test-morph init "$DATADIR/workspace" +cd "$DATADIR" +"$SRCDIR/scripts/test-morph" workspace diff --git a/tests.branching/workspace-not-found.stderr b/tests.branching/workspace-not-found.stderr new file mode 100644 index 00000000..041c64b3 --- /dev/null +++ b/tests.branching/workspace-not-found.stderr @@ -0,0 +1,2 @@ +ERROR: Can't find the workspace directory. +Morph must be built and deployed within the system branch checkout within the workspace directory. diff --git a/tests.branching/workspace.script b/tests.branching/workspace.script new file mode 100755 index 00000000..e717873c --- /dev/null +++ b/tests.branching/workspace.script @@ -0,0 +1,24 @@ +#!/bin/sh +# +# Copyright (C) 2012 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## morph init: Create a workspace. + +scripts/test-morph init "$DATADIR/workspace" +mkdir -p "$DATADIR/workspace/a/b/c" +cd "$DATADIR/workspace/a/b/c" +"$SRCDIR/scripts/test-morph" workspace diff --git a/tests.branching/workspace.stdout b/tests.branching/workspace.stdout new file mode 100644 index 00000000..14c44f7d --- /dev/null +++ b/tests.branching/workspace.stdout @@ -0,0 +1 @@ +TMP/workspace diff --git a/tests.build/ambiguous-refs.script b/tests.build/ambiguous-refs.script new file mode 100755 index 00000000..e1eae59d --- /dev/null +++ b/tests.build/ambiguous-refs.script @@ -0,0 +1,32 @@ +#!/bin/sh +# +# Copyright (C) 2011-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Guard against a bug that occurs if 'git show-ref' is used to resolve refs +## instead of 'git rev-parse --verify': show-ref returns a list of partial +## matches sorted alphabetically, so any code using it may resolve refs + +set -eu + +# Create a ref that will show up in 'git show-ref' before the real master ref +cd "$DATADIR/morphs-repo" +git checkout -q -b alpha/master +git rm -q hello-system.morph +git commit -q -m "This ref will not build correctly" + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system diff --git a/tests.build/build-chunk-failures-dump-log.exit b/tests.build/build-chunk-failures-dump-log.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.build/build-chunk-failures-dump-log.exit @@ -0,0 +1 @@ +1 diff --git a/tests.build/build-chunk-failures-dump-log.script b/tests.build/build-chunk-failures-dump-log.script new file mode 100755 index 00000000..645fd59a --- /dev/null +++ b/tests.build/build-chunk-failures-dump-log.script @@ -0,0 +1,39 @@ +#!/bin/bash +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test building a chunk that fails. + +set -eu + +# Make 'hello' chunk fail to build +chunkrepo="$DATADIR/chunk-repo" +cd "$chunkrepo" +git checkout --quiet farrokh +cat <<EOF >hello.morph +name: hello +kind: chunk +build-system: dummy +build-commands: + - echo The next command will fail + - "false" +EOF +git add hello.morph +git commit --quiet -m "Make morphology fail to build." + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system 2>/dev/null diff --git a/tests.build/build-chunk-failures-dump-log.stdout b/tests.build/build-chunk-failures-dump-log.stdout new file mode 100644 index 00000000..7a13c12a --- /dev/null +++ b/tests.build/build-chunk-failures-dump-log.stdout @@ -0,0 +1,8 @@ +build failed +# configure +# # echo dummy configure +dummy configure +# build +# # echo The next command will fail +The next command will fail +# # false diff --git a/tests.build/build-chunk-writes-log.script b/tests.build/build-chunk-writes-log.script new file mode 100755 index 00000000..5bfb2ae3 --- /dev/null +++ b/tests.build/build-chunk-writes-log.script @@ -0,0 +1,38 @@ +#!/bin/sh +# +# Copyright (C) 2011-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Build log should be saved when a chunk is built. + +set -eu + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +SOURCES="$DATADIR/cached-sources" +find "$DATADIR/cache/artifacts" -name '*.chunk.*' | + sed 's|\.chunk\..*||' | sort -u >"$SOURCES" + +found=false +# list of sources in cache is not piped because while loop changes variable +while read source; do + [ -e "$source".build-log ] || continue + found=true + break +done <"$SOURCES" +"$found" + diff --git a/tests.build/build-stratum-with-submodules.script b/tests.build/build-stratum-with-submodules.script new file mode 100755 index 00000000..015db3f2 --- /dev/null +++ b/tests.build/build-stratum-with-submodules.script @@ -0,0 +1,67 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test build a stratum that uses a chunk which needs a submodule. + +set -eu + +# tests/setup creates a chunk-repo. We now create a new chunk, which +# uses chunk-repo as a submodule. + +parent="$DATADIR/parent-repo" +mkdir "$parent" +cat <<EOF > "$parent/parent.morph" +name: parent +kind: chunk +build-system: manual +build-commands: + - test -f le-sub/README +EOF + +"$SRCDIR/scripts/run-git-in" "$parent" init --quiet +"$SRCDIR/scripts/run-git-in" "$parent" add . +"$SRCDIR/scripts/run-git-in" "$parent" \ + submodule --quiet add -b farrokh "$DATADIR/chunk-repo" le-sub > /dev/null +"$SRCDIR/scripts/run-git-in" "$parent" commit --quiet -m initial + + +# Modify the stratum to refer to the parent, not the submodule. + +morphs="$DATADIR/morphs-repo" +cat <<EOF > "$morphs/hello-stratum.morph" +name: hello-stratum +kind: stratum +chunks: + - name: parent + repo: test:parent-repo + ref: master + build-depends: [] + build-mode: test +EOF +"$SRCDIR/scripts/run-git-in" "$morphs" add hello-stratum.morph +"$SRCDIR/scripts/run-git-in" "$morphs" commit --quiet -m 'foo' + + +# Now build and verify we got a stratum. + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +system=$(ls "$DATADIR/cache/artifacts/"*hello-system-rootfs) +tar tf $system | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -v '^baserock/' + diff --git a/tests.build/build-stratum-with-submodules.stdout b/tests.build/build-stratum-with-submodules.stdout new file mode 100644 index 00000000..d4d03e13 --- /dev/null +++ b/tests.build/build-stratum-with-submodules.stdout @@ -0,0 +1,3 @@ +./ +etc/ +etc/os-release diff --git a/tests.build/build-system-autotools-fails-if-autogen-fails.exit b/tests.build/build-system-autotools-fails-if-autogen-fails.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.build/build-system-autotools-fails-if-autogen-fails.exit @@ -0,0 +1 @@ +1 diff --git a/tests.build/build-system-autotools-fails-if-autogen-fails.script b/tests.build/build-system-autotools-fails-if-autogen-fails.script new file mode 100755 index 00000000..d7fdd055 --- /dev/null +++ b/tests.build/build-system-autotools-fails-if-autogen-fails.script @@ -0,0 +1,41 @@ +#!/bin/sh +# +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test that the autotools build system fails if it runs autogen.sh and that +## fails. + +set -eu + +cd "$DATADIR/chunk-repo" +git checkout -q farrokh + +cat <<EOF > autogen.sh +#!/bin/sh +echo "in failing autogen.sh" +exit 1 +EOF +chmod a+x autogen.sh + +git add autogen.sh +git rm -q hello.morph +git commit -q -m "Convert hello to a broken autotools project" + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system \ + >/dev/null 2> /dev/null + diff --git a/tests.build/build-system-autotools.script b/tests.build/build-system-autotools.script new file mode 100755 index 00000000..2ea53174 --- /dev/null +++ b/tests.build/build-system-autotools.script @@ -0,0 +1,54 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Convert the hello-chunk project to something autotools-like, then +## build it. + +set -eu + +chunkrepo="$DATADIR/chunk-repo" +cd "$chunkrepo" + +git checkout --quiet farrokh + +cat <<'EOF' >Makefile +all: hello + +install: all + install -d "$(DESTDIR)/etc" + install -d "$(DESTDIR)/bin" + install hello "$(DESTDIR)/bin/hello" +EOF +git add Makefile + +cat <<EOF > hello.morph +name: hello +kind: chunk +build-system: autotools +configure-commands: [] +EOF +git add hello.morph +git commit --quiet -m "Convert hello to an autotools project" + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(bin|etc)' diff --git a/tests.build/build-system-autotools.stdout b/tests.build/build-system-autotools.stdout new file mode 100644 index 00000000..683441c9 --- /dev/null +++ b/tests.build/build-system-autotools.stdout @@ -0,0 +1,3 @@ +bin/ +bin/hello +etc/ diff --git a/tests.build/build-system-cmake.script b/tests.build/build-system-cmake.script new file mode 100755 index 00000000..570a9af7 --- /dev/null +++ b/tests.build/build-system-cmake.script @@ -0,0 +1,56 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Convert the hello-chunk project to something cmake-like, then +## build it. + +set -eu + +chunkrepo="$DATADIR/chunk-repo" +cd "$chunkrepo" + +git checkout --quiet farrokh + +cat <<'EOF' >CMakeLists.txt +cmake_minimum_required(VERSION 2.8) +project(hello) + +set(hello_SOURCES hello.c) +add_executable(hello ${hello_SOURCES}) +install(TARGETS hello RUNTIME DESTINATION ${CMAKE_INSTALL_PREFIX}/bin) +EOF + +git add CMakeLists.txt + +cat <<EOF > hello.morph +name: hello +kind: chunk +build-system: cmake +install-commands: + - make DESTDIR="\$DESTDIR" install +EOF +git add hello.morph +git commit --quiet -m "Convert hello to a cmake project" + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(usr/)?(bin|etc)' diff --git a/tests.build/build-system-cmake.stdout b/tests.build/build-system-cmake.stdout new file mode 100644 index 00000000..3410b113 --- /dev/null +++ b/tests.build/build-system-cmake.stdout @@ -0,0 +1,2 @@ +usr/bin/ +usr/bin/hello diff --git a/tests.build/build-system-cpan.script b/tests.build/build-system-cpan.script new file mode 100755 index 00000000..735dac84 --- /dev/null +++ b/tests.build/build-system-cpan.script @@ -0,0 +1,78 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Convert the hello-chunk project to perl with CPAN and build. + +set -eu + +chunkrepo="$DATADIR/chunk-repo" +cd "$chunkrepo" + +git checkout --quiet farrokh + +git rm --quiet hello.c + +cat <<EOF >hello +#!/usr/bin/perl +print "hello, world\n" +EOF +git add hello + +cat <<EOF >Makefile.PL +use strict; +use warnings; +use ExtUtils::MakeMaker; +WriteMakefile( + EXE_FILES => ['hello'], +) +EOF +git add Makefile.PL + +cat <<EOF >hello.morph +name: hello +kind: chunk +build-system: cpan +EOF +git add hello.morph + +git commit --quiet -m 'convert hello into a perl cpan project' + +# Set 'prefix' of hello to something custom +cd "$DATADIR/morphs-repo" +cat <<EOF > hello-stratum.morph +name: hello-stratum +kind: stratum +chunks: + - name: hello + repo: test:chunk-repo + ref: farrokh + build-depends: [] + build-mode: test + prefix: / +EOF +git add hello-stratum.morph +git commit -q -m "Set custom install prefix for hello" + + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -F 'bin/hello' diff --git a/tests.build/build-system-cpan.stdout b/tests.build/build-system-cpan.stdout new file mode 100644 index 00000000..180e949b --- /dev/null +++ b/tests.build/build-system-cpan.stdout @@ -0,0 +1 @@ +bin/hello diff --git a/tests.build/build-system-python-distutils.script b/tests.build/build-system-python-distutils.script new file mode 100755 index 00000000..9a751491 --- /dev/null +++ b/tests.build/build-system-python-distutils.script @@ -0,0 +1,81 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Convert the hello-chunk project to python with distutils and build. + +set -eu + +chunkrepo="$DATADIR/chunk-repo" +cd "$chunkrepo" + +git checkout --quiet farrokh + +git rm --quiet hello.c +cat <<EOF >hello +#!/usr/bin/python +print "hello, world" +EOF +git add hello + +cat <<EOF >setup.py +#!/usr/bin/python +from distutils.core import setup +setup(name='hello', + scripts=['hello']) +EOF +git add setup.py + +cat <<EOF >hello.morph +name: hello +kind: chunk +build-system: python-distutils +EOF +git add hello.morph + +git commit --quiet -m 'convert hello into a python project' + + +# Set 'prefix' of hello to something custom +cd "$DATADIR/morphs-repo" +cat <<EOF > hello-stratum.morph +name: hello-stratum +kind: stratum +chunks: + - name: hello + repo: test:chunk-repo + ref: farrokh + build-depends: [] + build-mode: test + prefix: "" +EOF +git add hello-stratum.morph +git commit -q -m "Set custom install prefix for hello" + + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(bin|lib)' | +sed -e 's:^local/::' \ + -e 's:lib/python2.[6-9]/:lib/python2.x/:' \ + -e 's:/hello-0\.0\.0[^/]*\.egg-info$:/hello.egg-info/:' \ + -e 's:[^/]*-packages:packages:' \ + -e '/^$/d' diff --git a/tests.build/build-system-python-distutils.stdout b/tests.build/build-system-python-distutils.stdout new file mode 100644 index 00000000..4d4c3a1e --- /dev/null +++ b/tests.build/build-system-python-distutils.stdout @@ -0,0 +1,6 @@ +bin/ +bin/hello +lib/ +lib/python2.x/ +lib/python2.x/packages/ +lib/python2.x/packages/hello.egg-info/ diff --git a/tests.build/build-system-qmake.script b/tests.build/build-system-qmake.script new file mode 100755 index 00000000..b3861936 --- /dev/null +++ b/tests.build/build-system-qmake.script @@ -0,0 +1,66 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Convert the hello-chunk project to something qmake-like, then +## build it. + +set -eu + +if ! command -v qmake > /dev/null ; then + # There is no qmake, so skip this test. + cat "$SRCDIR/tests.build/build-system-qmake.stdout" + exit 0 +fi + +chunkrepo="$DATADIR/chunk-repo" +cd "$chunkrepo" + +git checkout --quiet farrokh + +cat <<'EOF' >hello.pro +TEMPLATE = app +TARGET = hello +DEPENDPATH += . +INCLUDEPATH += . + +SOURCES += hello.c +hello.path = /usr/bin +hello.files = hello +INSTALLS += hello +EOF +git add hello.pro + +cat <<EOF > hello.morph +name: hello +kind: chunk +build-system: qmake +install-commands: + - make INSTALL_ROOT="\$DESTDIR" install +EOF +git add hello.morph +git commit --quiet -m "Convert hello to an qmake project" + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + echo "$chunk:" | sed 's/[^.]*//' + tar -tf "$chunk" | LC_ALL=C sort | sed '/^\.\/./s:^\./::' + echo +done diff --git a/tests.build/build-system-qmake.stdout b/tests.build/build-system-qmake.stdout new file mode 100644 index 00000000..ccf80a86 --- /dev/null +++ b/tests.build/build-system-qmake.stdout @@ -0,0 +1,8 @@ +.chunk.hello: +./ +baserock/ +baserock/hello.meta +usr/ +usr/bin/ +usr/bin/hello + diff --git a/tests.build/build-system.script b/tests.build/build-system.script new file mode 100755 index 00000000..56d80735 --- /dev/null +++ b/tests.build/build-system.script @@ -0,0 +1,27 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test building a simple system. + +set -eu + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +system=$(ls "$DATADIR/cache/artifacts/"*hello-system-rootfs) +tar tf $system | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -v '^baserock/' diff --git a/tests.build/build-system.stdout b/tests.build/build-system.stdout new file mode 100644 index 00000000..4d0fac2f --- /dev/null +++ b/tests.build/build-system.stdout @@ -0,0 +1,5 @@ +./ +bin/ +bin/hello +etc/ +etc/os-release diff --git a/tests.build/cross-bootstrap-only-to-supported-archs.exit b/tests.build/cross-bootstrap-only-to-supported-archs.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.build/cross-bootstrap-only-to-supported-archs.exit @@ -0,0 +1 @@ +1 diff --git a/tests.build/cross-bootstrap-only-to-supported-archs.script b/tests.build/cross-bootstrap-only-to-supported-archs.script new file mode 100755 index 00000000..872acd9f --- /dev/null +++ b/tests.build/cross-bootstrap-only-to-supported-archs.script @@ -0,0 +1,25 @@ +#!/bin/bash +# +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Test that "morph cross-bootstrap" works only for the architectures that +# Morph explicitly supports. + +set -eu + +"$SRCDIR/scripts/test-morph" cross-bootstrap \ + unknown-archicture test:morphs-repo master hello-system -v diff --git a/tests.build/cross-bootstrap-only-to-supported-archs.stderr b/tests.build/cross-bootstrap-only-to-supported-archs.stderr new file mode 100644 index 00000000..61c0fe0d --- /dev/null +++ b/tests.build/cross-bootstrap-only-to-supported-archs.stderr @@ -0,0 +1 @@ +ERROR: Unsupported architecture "unknown-archicture" diff --git a/tests.build/cross-bootstrap.script b/tests.build/cross-bootstrap.script new file mode 100755 index 00000000..51d9ef1e --- /dev/null +++ b/tests.build/cross-bootstrap.script @@ -0,0 +1,28 @@ +#!/bin/bash +# +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Test "morph cross-bootstrap", up to the point of the tarball it generates +# for the target + +set -eu + +"$SRCDIR/tests.build/setup-build-essential" + +"$SRCDIR/scripts/test-morph" cross-bootstrap \ + $("$SRCDIR/scripts/test-morph" print-architecture) \ + test:morphs-repo master hello-system diff --git a/tests.build/empty-stratum.exit b/tests.build/empty-stratum.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.build/empty-stratum.exit @@ -0,0 +1 @@ +1 diff --git a/tests.build/empty-stratum.script b/tests.build/empty-stratum.script new file mode 100755 index 00000000..19c36558 --- /dev/null +++ b/tests.build/empty-stratum.script @@ -0,0 +1,36 @@ +#!/bin/sh +# +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +set -eu + +morphsrepo="$DATADIR/morphs-repo" +cd "$morphsrepo" + +git checkout --quiet -b empty-stratum + +# Create empty stratum to test S4585 +cat <<EOF > hello-stratum.morph +name: hello-stratum +kind: stratum +EOF +sed -i 's/master/empty-stratum/' hello-system.morph +git add hello-stratum.morph hello-system.morph + +git commit --quiet -m "add empty stratum" + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo empty-stratum hello-system diff --git a/tests.build/empty-stratum.stderr b/tests.build/empty-stratum.stderr new file mode 100644 index 00000000..6a4ecb05 --- /dev/null +++ b/tests.build/empty-stratum.stderr @@ -0,0 +1 @@ +ERROR: Stratum hello-stratum has no chunks in string diff --git a/tests.build/missing-ref.exit b/tests.build/missing-ref.exit new file mode 100644 index 00000000..d00491fd --- /dev/null +++ b/tests.build/missing-ref.exit @@ -0,0 +1 @@ +1 diff --git a/tests.build/missing-ref.script b/tests.build/missing-ref.script new file mode 100755 index 00000000..a18ce2d1 --- /dev/null +++ b/tests.build/missing-ref.script @@ -0,0 +1,23 @@ +#!/bin/sh +# +# Copyright (C) 2011-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test building with a bad reference. + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo non-existent-branch hello-system + diff --git a/tests.build/missing-ref.stderr b/tests.build/missing-ref.stderr new file mode 100644 index 00000000..5fa5456b --- /dev/null +++ b/tests.build/missing-ref.stderr @@ -0,0 +1 @@ +ERROR: Ref non-existent-branch is an invalid reference for repo file://TMP/morphs-repo diff --git a/tests.build/morphless-chunks.script b/tests.build/morphless-chunks.script new file mode 100755 index 00000000..9a8b41dd --- /dev/null +++ b/tests.build/morphless-chunks.script @@ -0,0 +1,48 @@ +#!/bin/sh +# +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Try to build a morphless chunk. + +set -eu + +# Make 'hello' chunk into an auto-detectable chunk. + +cd "$DATADIR/chunk-repo" +git checkout -q farrokh + +touch configure +chmod +x configure +# FIXME: If we leave the file empty, busybox sh on ARMv7 fails to execute it. +echo '#!/bin/sh' > configure + +cat << EOF > Makefile +all install: +EOF + +git rm -q hello.morph +git add Makefile configure +git commit -q -m "Convert hello into an autodetectable chunk" + + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | cat >/dev/null # No files get installed apart from metadata diff --git a/tests.build/morphless-chunks.stdout b/tests.build/morphless-chunks.stdout new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/tests.build/morphless-chunks.stdout diff --git a/tests.build/only-build-systems.script b/tests.build/only-build-systems.script new file mode 100755 index 00000000..699be942 --- /dev/null +++ b/tests.build/only-build-systems.script @@ -0,0 +1,29 @@ +#!/bin/sh +# +# Copyright (C) 2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Morph should refuse build a chunk out of the context. Only +## system and stratum morphologies can be built. + +set -eu + +! "$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-stratum + +! "$SRCDIR/scripts/test-morph" build-morphology \ + test:chunk-repo farrokh hello + diff --git a/tests.build/only-build-systems.stderr b/tests.build/only-build-systems.stderr new file mode 100644 index 00000000..ba7339d2 --- /dev/null +++ b/tests.build/only-build-systems.stderr @@ -0,0 +1,2 @@ +ERROR: Building a stratum directly is not supported +ERROR: Building a chunk directly is not supported diff --git a/tests.build/prefix.script b/tests.build/prefix.script new file mode 100755 index 00000000..209c1a54 --- /dev/null +++ b/tests.build/prefix.script @@ -0,0 +1,73 @@ +#!/bin/sh +# +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Honour 'prefix' attribute for chunks within stratum morphs + +set -eu + +# Create two chunks which print out PATH and PREFIX from their environment. +cd "$DATADIR/chunk-repo" +git checkout -q master +cat <<\EOF > xyzzy.morph +name: xyzzy +kind: chunk +configure-commands: + - "echo First chunk: prefix $PREFIX" +EOF + +cat <<\EOF > plugh.morph +name: plugh +kind: chunk +configure-commands: + - "echo Second chunk: prefix $PREFIX" + - "echo Path: $(echo $PATH | grep -o '/plover')" +EOF + +git add xyzzy.morph +git add plugh.morph +git commit -q -m "Add chunks" + +# Change stratum to include those two chunks, and use a custom install prefix +cd "$DATADIR/morphs-repo" +cat <<EOF > hello-stratum.morph +name: hello-stratum +kind: stratum +chunks: + - name: xyzzy + repo: test:chunk-repo + ref: master + build-depends: [] + build-mode: test + prefix: /plover + - name: plugh + repo: test:chunk-repo + ref: master + build-mode: test + build-depends: + - xyzzy +EOF +git add hello-stratum.morph +git commit -q -m "Update stratum" + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +cd "$DATADIR/cache/artifacts" +first_chunk=$(ls -1 *.chunk.xyzzy-* | head -n1 | cut -c -64) +second_chunk=$(ls -1 *.chunk.plugh-* | head -n1 | cut -c -64) +cat $first_chunk.build-log $second_chunk.build-log diff --git a/tests.build/prefix.stdout b/tests.build/prefix.stdout new file mode 100644 index 00000000..80c18fae --- /dev/null +++ b/tests.build/prefix.stdout @@ -0,0 +1,8 @@ +# configure +# # echo First chunk: prefix $PREFIX +First chunk: prefix /plover +# configure +# # echo Second chunk: prefix $PREFIX +Second chunk: prefix /usr +# # echo Path: $(echo $PATH | grep -o '/plover') +Path: /plover diff --git a/tests.build/rebuild-cached-stratum.script b/tests.build/rebuild-cached-stratum.script new file mode 100755 index 00000000..0014e545 --- /dev/null +++ b/tests.build/rebuild-cached-stratum.script @@ -0,0 +1,59 @@ +#!/bin/sh +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Does a cached stratum get rebuilt if its chunk changes? +## This tests a bug that is currently in morph, where the stratum does +## not get rebuilt in that case. Later on, the test will guard against +## regressions. + +set -eu + +cache="$DATADIR/cache/artifacts" + +# Make a branch in the chunk repo where we can make our own modifications. +(cd "$DATADIR/chunk-repo" && + git checkout --quiet farrokh && + git checkout --quiet -b rebuild-cached-stratum) + +# Make a branch in the morphs repo and modify the stratum to refer to +# the new chunk branch. +(cd "$DATADIR/morphs-repo" && + git checkout --quiet -b rebuild-cached-stratum && + sed -i 's/farrokh/rebuild-cached-stratum/' hello-stratum.morph && + sed -i 's/master/rebuild-cached-stratum/' hello-system.morph && + git commit --quiet -m "rebuild-cached-stratum" -a) + +# Build the first time. +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo rebuild-cached-stratum hello-system +echo "first build:" +(cd "$cache" && ls *.chunk.* *hello-stratum-* | sed 's/^[^.]*\./ /' | + LC_ALL=C sort -u) + +# Change the chunk. +(cd "$DATADIR/chunk-repo" && + echo >> hello.c && + git commit --quiet -am change) + +# Rebuild. +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo rebuild-cached-stratum hello-system +echo "second build:" +(cd "$cache" && ls *.chunk.* *hello-stratum-* | sed 's/^[^.]*\./ /' | + LC_ALL=C sort -u) + diff --git a/tests.build/rebuild-cached-stratum.stdout b/tests.build/rebuild-cached-stratum.stdout new file mode 100644 index 00000000..9c53ee60 --- /dev/null +++ b/tests.build/rebuild-cached-stratum.stdout @@ -0,0 +1,22 @@ +first build: + chunk.hello-bins + chunk.hello-devel + chunk.hello-doc + chunk.hello-libs + chunk.hello-locale + chunk.hello-misc + stratum.hello-stratum-devel + stratum.hello-stratum-devel.meta + stratum.hello-stratum-runtime + stratum.hello-stratum-runtime.meta +second build: + chunk.hello-bins + chunk.hello-devel + chunk.hello-doc + chunk.hello-libs + chunk.hello-locale + chunk.hello-misc + stratum.hello-stratum-devel + stratum.hello-stratum-devel.meta + stratum.hello-stratum-runtime + stratum.hello-stratum-runtime.meta diff --git a/tests.build/setup b/tests.build/setup new file mode 100755 index 00000000..833f132d --- /dev/null +++ b/tests.build/setup @@ -0,0 +1,118 @@ +#!/bin/sh +# +# Create git repositories for tests. The chunk repository will contain a +# simple "hello, world" C program, and two branches ("master", "farrokh"), +# with the master branch containing just a README. The two branches are there +# so that we can test building a branch that hasn't been checked out. +# The branches are different so that we know that if the wrong branch +# is uses, the build will fail. +# +# The stratum repository contains a single branch, "master", with a +# stratum and a system morphology that include the chunk above. +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +set -eu + +# The $DATADIR should be empty at the beginnig of each test. +find "$DATADIR" -mindepth 1 -delete + +# Create chunk repository. + +chunkrepo="$DATADIR/chunk-repo" +mkdir "$chunkrepo" +cd "$chunkrepo" +git init --quiet + +cat <<EOF > README +This is a sample README. +EOF +git add README +git commit --quiet -m "add README" + +git checkout --quiet -b farrokh + +cat <<EOF > hello.c +#include <stdio.h> +int main(void) +{ + puts("hello, world"); + return 0; +} +EOF +git add hello.c + +cat <<EOF > hello.morph +name: hello +kind: chunk +build-system: dummy +build-commands: + - gcc -o hello hello.c +install-commands: + - install -d "\$DESTDIR"/etc + - install -d "\$DESTDIR"/bin + - install hello "\$DESTDIR"/bin/hello +EOF +git add hello.morph + +git commit --quiet -m "add a hello world program and morph" + +git checkout --quiet master + + + +# Create morph repository. + +morphsrepo="$DATADIR/morphs-repo" +mkdir "$morphsrepo" +cd "$morphsrepo" +git init --quiet + +cat <<EOF > hello-stratum.morph +name: hello-stratum +kind: stratum +chunks: + - name: hello + repo: test:chunk-repo + ref: farrokh + build-mode: test + build-depends: [] +EOF +git add hello-stratum.morph + +cat <<EOF > hello-system.morph +name: hello-system +kind: system +arch: $("$SRCDIR/scripts/test-morph" print-architecture) +strata: + - morph: hello-stratum +EOF +git add hello-system.morph + +git commit --quiet -m "add morphs" + + +# Create a morph configuration file. +cat <<EOF > "$DATADIR/morph.conf" +[config] +repo-alias = test=file://$DATADIR/%s#file://$DATADIR/%s +cachedir = $DATADIR/cache +log = $DATADIR/morph.log +no-distcc = true +quiet = true +EOF + diff --git a/tests.build/setup-build-essential b/tests.build/setup-build-essential new file mode 100755 index 00000000..9ffb7774 --- /dev/null +++ b/tests.build/setup-build-essential @@ -0,0 +1,107 @@ +#!/bin/sh +# +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Set up a stratum which resembles Baserock's 'build-essential' slightly. Used +# for testing 'morph cross-bootstrap' and the 'bootstrap' build mode. + +mkdir -p "$DATADIR/cc-repo" +cd "$DATADIR/cc-repo" + +cat <<EOF > "morph-test-cc" +#!/bin/sh +echo "I'm a compiler!" +EOF +chmod +x morph-test-cc + +cat <<EOF > "stage1-cc.morph" +name: stage1-cc +kind: chunk +install-commands: + - install -d "\$DESTDIR\$PREFIX/bin" + - install -m 755 morph-test-cc "\$DESTDIR\$PREFIX/bin/morph-test-cc" +EOF + +cat <<EOF > "cc.morph" +name: cc +kind: chunk +configure-commands: + - [ -e ../tools/bin/morph-test-cc ] +install-commands: + - install -d "\$DESTDIR\$PREFIX/bin" + - install -m 755 morph-test-cc "\$DESTDIR\$PREFIX/bin/morph-test-cc" +EOF + +git init -q +git add morph-test-cc cc.morph stage1-cc.morph +git commit -q -m "Create compiler chunk" + +# Require 'cc' in hello-chunk. We should have the second version available +# but *not* the first one. +cd "$DATADIR/chunk-repo" +git checkout -q farrokh +cat <<EOF > "hello.morph" +name: hello +kind: chunk +configure-commands: + - [ ! -e ../tools/bin/morph-test-cc ] + - [ -e ../usr/bin/morph-test-cc ] +build-commands: + - ../usr/bin/morph-test-cc > hello +install-commands: + - install -d "\$DESTDIR\$PREFIX/bin" + - install hello "\$DESTDIR\$PREFIX/bin/hello" +EOF +git add hello.morph +git commit -q -m "Make 'hello' require our mock compiler" + +# Add 'build-essential' stratum and make hello-stratum depend upon it. Only +# the *second* 'cc' chunk should make it into the build-essential stratum +# artifact, and neither should make it into the system. +cd "$DATADIR/morphs-repo" +cat <<EOF > "build-essential.morph" +name: build-essential +kind: stratum +chunks: + - name: stage1-cc + repo: test:cc-repo + ref: master + build-depends: [] + build-mode: bootstrap + prefix: /tools + - name: cc + repo: test:cc-repo + ref: master + build-depends: + - stage1-cc + build-mode: test +EOF + +cat <<EOF > "hello-stratum.morph" +name: hello-stratum +kind: stratum +build-depends: + - morph: build-essential +chunks: + - name: hello + repo: test:chunk-repo + ref: farrokh + build-depends: [] + build-mode: test +EOF + +git add build-essential.morph hello-stratum.morph hello-system.morph +git commit -q -m "Add fake build-essential stratum" diff --git a/tests.build/uses-tempdir.script b/tests.build/uses-tempdir.script new file mode 100755 index 00000000..80c06d56 --- /dev/null +++ b/tests.build/uses-tempdir.script @@ -0,0 +1,28 @@ +#!/bin/sh +# +# Copyright (C) 2011-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test that temporary directories are created in the dir specified +## by --tempdir rather than specified in the environment by TMPDIR. + +set -eu +export TMPDIR +TMPDIR="$DATADIR"/unwritable-tmp +install -m 000 -d "$TMPDIR" +mkdir "$DATADIR"/tmp +"$SRCDIR/scripts/test-morph" build-morphology --tempdir "$DATADIR"/tmp \ + test:morphs-repo master hello-system diff --git a/tests/setup b/tests/setup new file mode 100755 index 00000000..6ebab880 --- /dev/null +++ b/tests/setup @@ -0,0 +1,43 @@ +#!/bin/sh +# +# Create git repositories for tests. The chunk repository will contain a +# simple "hello, world" C program, and two branches ("master", "farrokh"), +# with the master branch containing just a README. The two branches are there +# so that we can test building a branch that hasn't been checked out. +# The branches are different so that we know that if the wrong branch +# is uses, the build will fail. +# +# The stratum repository contains a single branch, "master", with a +# stratum and a system morphology that include the chunk above. +# +# Copyright (C) 2011-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +set -eu + +# The $DATADIR should be empty at the beginnig of each test. +find "$DATADIR" -mindepth 1 -delete + +# Create a morph configuration file. +cat <<EOF > "$DATADIR/morph.conf" +[config] +repo-alias = test=file://$DATADIR/%s#file://$DATADIR/%s +cachedir = $DATADIR/cache +log = $DATADIR/morph.log +no-distcc = true +quiet = true +EOF + diff --git a/tests/show-dependencies.script b/tests/show-dependencies.script new file mode 100755 index 00000000..15b69e25 --- /dev/null +++ b/tests/show-dependencies.script @@ -0,0 +1,25 @@ +#!/bin/bash +# +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Test "show-dependencies" subcommand. + +set -eu + +"$SRCDIR/scripts/test-morph" \ + show-dependencies test:test-repo master xfce-system | + sed 's/test://' diff --git a/tests/show-dependencies.setup b/tests/show-dependencies.setup new file mode 100755 index 00000000..74d10c2b --- /dev/null +++ b/tests/show-dependencies.setup @@ -0,0 +1,250 @@ +#!/bin/bash +# +# Copyright (C) 2012-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +set -e + +source "$SRCDIR/scripts/fix-committer-info" + +# Create a repository +repo="$DATADIR/test-repo" +mkdir "$repo" +cd "$repo" +git init --quiet + +# Add a single source file to simulate compiling +cat <<EOF > hello.c +#include <stdio.h> +int main(void) +{ + puts("hello, world"); + return 0; +} +EOF +git add hello.c + +# Define a couple of chunk morphologies for the GTK stack +gtkcomponents=(freetype fontconfig cairo pango glib gdk-pixbuf gtk + dbus-glib dbus) +for component in "${gtkcomponents[@]}" +do + cat <<EOF > $component.morph +name: $component +kind: chunk +build-commands: + - gcc -o hello hello.c +install-commands: + - install -d "\$DESTDIR"/etc + - install -d "\$DESTDIR"/bin + - install hello "\$DESTDIR"/bin/$component +EOF + git add $component.morph +done +git commit --quiet -m "add .c source file and GTK chunk morphologies" + +# Define a stratum for the GTK stack +cat <<EOF > gtk-stack.morph +name: gtk-stack +kind: stratum +build-depends: [] +chunks: + - name: freetype + repo: test:test-repo + ref: master + build-depends: [] + build-mode: bootstrap + - name: fontconfig + repo: test:test-repo + ref: master + build-depends: [] + build-mode: bootstrap + - name: cairo + repo: test:test-repo + ref: master + build-depends: [] + build-mode: bootstrap + - name: pango + repo: test:test-repo + ref: master + build-depends: + - freetype + - fontconfig + - name: glib + repo: test:test-repo + ref: master + build-depends: [] + build-mode: bootstrap + - name: gdk-pixbuf + repo: test:test-repo + ref: master + build-depends: + - glib + - name: gtk + repo: test:test-repo + ref: master + build-depends: + - cairo + - gdk-pixbuf + - glib + - pango + - name: dbus + repo: test:test-repo + ref: master + build-depends: [] + build-mode: bootstrap + - name: dbus-glib + repo: test:test-repo + ref: master + build-depends: + - dbus + - glib +EOF +git add gtk-stack.morph +git commit --quiet -m "add gtk-stack.morph stratum" + +# Add a single source file to simulate compiling +cat <<EOF > hello.c +#include <stdio.h> +int main(void) +{ + puts("hello, world"); + return 0; +} +EOF +git add hello.c + +# Define a couple of chunk morphologies for the GTK stack +xfcecomponents=(xfce4-dev-tools libxfce4util libxfce4ui exo xfconf garcon + thunar tumbler xfce4-panel xfce4-settings xfce4-session + xfwm4 xfdesktop xfce4-appfinder gtk-xfce-engine) +for component in "${xfcecomponents[@]}" +do + cat <<EOF > $component.morph +name: $component +kind: chunk +build-commands: + - gcc -o hello hello.c +install-commands: + - install -d "\$DESTDIR"/etc + - install -d "\$DESTDIR"/bin + - install hello "\$DESTDIR"/bin/$component +EOF + git add $component.morph +done +git commit --quiet -m "add .c source file and GTK chunk morphologies" + +# Define a stratum for the Xfce core +cat <<EOF > xfce-core.morph +name: xfce-core +kind: stratum +build-depends: + - morph: gtk-stack +chunks: + - name: libxfce4util + repo: test:test-repo + ref: master + build-depends: [] + - name: xfconf + repo: test:test-repo + ref: master + build-depends: + - libxfce4util + - name: libxfce4ui + repo: test:test-repo + ref: master + build-depends: + - xfconf + - name: exo + repo: test:test-repo + ref: master + build-depends: + - libxfce4util + - name: garcon + repo: test:test-repo + ref: master + build-depends: + - libxfce4util + - name: thunar + repo: test:test-repo + ref: master + build-depends: + - libxfce4ui + - exo + - name: tumbler + repo: test:test-repo + ref: master + build-depends: [] + - name: xfce4-panel + repo: test:test-repo + ref: master + build-depends: + - libxfce4ui + - exo + - garcon + - name: xfce4-settings + repo: test:test-repo + ref: master + build-depends: + - libxfce4ui + - exo + - xfconf + - name: xfce4-session + repo: test:test-repo + ref: master + build-depends: + - libxfce4ui + - exo + - xfconf + - name: xfwm4 + repo: test:test-repo + ref: master + build-depends: + - libxfce4ui + - xfconf + - name: xfdesktop + repo: test:test-repo + ref: master + build-depends: + - libxfce4ui + - xfconf + - name: xfce4-appfinder + repo: test:test-repo + ref: master + build-depends: + - libxfce4ui + - garcon + - xfconf + - name: gtk-xfce-engine + repo: test:test-repo + ref: master + build-depends: + - libxfce4ui + - garcon + - xfconf +EOF +git add xfce-core.morph +git commit --quiet -m "add xfce-core.morph stratum" + +cat <<EOF > xfce-system.morph +name: xfce-system +kind: system +arch: $("$SRCDIR/scripts/test-morph" print-architecture) +strata: + - morph: xfce-core + build-mode: bootstrap +EOF +git add xfce-system.morph +git commit --quiet -m "add xfce-system" diff --git a/tests/show-dependencies.stdout b/tests/show-dependencies.stdout new file mode 100644 index 00000000..833b7245 --- /dev/null +++ b/tests/show-dependencies.stdout @@ -0,0 +1,1680 @@ +dependency graph for test-repo|master|xfce-system.morph: + test-repo|master|xfce-system.morph|xfce-system|xfce-system-rootfs + -> test-repo|master|xfce-core.morph|xfce-core-devel|xfce-core-devel + -> test-repo|master|xfce-core.morph|xfce-core-runtime|xfce-core-runtime + test-repo|master|xfce-core.morph|xfce-core-devel|xfce-core-devel + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-devel + -> test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|thunar.morph|thunar|thunar-devel + -> test-repo|master|thunar.morph|thunar|thunar-doc + -> test-repo|master|tumbler.morph|tumbler|tumbler-devel + -> test-repo|master|tumbler.morph|tumbler|tumbler-doc + -> test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-devel + -> test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-doc + -> test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-devel + -> test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-doc + -> test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-devel + -> test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-doc + -> test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-devel + -> test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-devel + -> test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-doc + -> test-repo|master|xfwm4.morph|xfwm4|xfwm4-devel + -> test-repo|master|xfwm4.morph|xfwm4|xfwm4-doc + test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-doc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-devel + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-doc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-devel + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfwm4.morph|xfwm4|xfwm4-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfwm4.morph|xfwm4|xfwm4-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-doc + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-devel + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-doc + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-devel + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-doc + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-devel + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|tumbler.morph|tumbler|tumbler-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|tumbler.morph|tumbler|tumbler-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|thunar.morph|thunar|thunar-doc + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|thunar.morph|thunar|thunar-devel + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|xfce-core.morph|xfce-core-runtime|xfce-core-runtime + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-bins + -> test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-libs + -> test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-locale + -> test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-misc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + -> test-repo|master|thunar.morph|thunar|thunar-bins + -> test-repo|master|thunar.morph|thunar|thunar-libs + -> test-repo|master|thunar.morph|thunar|thunar-locale + -> test-repo|master|thunar.morph|thunar|thunar-misc + -> test-repo|master|tumbler.morph|tumbler|tumbler-bins + -> test-repo|master|tumbler.morph|tumbler|tumbler-libs + -> test-repo|master|tumbler.morph|tumbler|tumbler-locale + -> test-repo|master|tumbler.morph|tumbler|tumbler-misc + -> test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-bins + -> test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-libs + -> test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-locale + -> test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-misc + -> test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-bins + -> test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-libs + -> test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-locale + -> test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-misc + -> test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-bins + -> test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-libs + -> test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-locale + -> test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-misc + -> test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-bins + -> test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-libs + -> test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-locale + -> test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + -> test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-bins + -> test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-libs + -> test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-locale + -> test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-misc + -> test-repo|master|xfwm4.morph|xfwm4|xfwm4-bins + -> test-repo|master|xfwm4.morph|xfwm4|xfwm4-libs + -> test-repo|master|xfwm4.morph|xfwm4|xfwm4-locale + -> test-repo|master|xfwm4.morph|xfwm4|xfwm4-misc + test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-locale + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-libs + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|gtk-xfce-engine.morph|gtk-xfce-engine|gtk-xfce-engine-bins + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-locale + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-libs + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-appfinder.morph|xfce4-appfinder|xfce4-appfinder-bins + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-locale + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-libs + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfdesktop.morph|xfdesktop|xfdesktop-bins + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfwm4.morph|xfwm4|xfwm4-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfwm4.morph|xfwm4|xfwm4-locale + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfwm4.morph|xfwm4|xfwm4-libs + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfwm4.morph|xfwm4|xfwm4-bins + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-misc + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-locale + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-libs + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-session.morph|xfce4-session|xfce4-session-bins + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-misc + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-locale + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-libs + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-settings.morph|xfce4-settings|xfce4-settings-bins + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-misc + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-locale + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-libs + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|xfce4-panel.morph|xfce4-panel|xfce4-panel-bins + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|garcon.morph|garcon|garcon-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|garcon.morph|garcon|garcon-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|tumbler.morph|tumbler|tumbler-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|tumbler.morph|tumbler|tumbler-locale + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|tumbler.morph|tumbler|tumbler-libs + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|tumbler.morph|tumbler|tumbler-bins + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|thunar.morph|thunar|thunar-misc + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|thunar.morph|thunar|thunar-locale + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|thunar.morph|thunar|thunar-libs + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|thunar.morph|thunar|thunar-bins + -> test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + test-repo|master|exo.morph|exo|exo-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|exo.morph|exo|exo-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|garcon.morph|garcon|garcon-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|garcon.morph|garcon|garcon-locale + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|garcon.morph|garcon|garcon-libs + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|garcon.morph|garcon|garcon-bins + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|exo.morph|exo|exo-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|exo.morph|exo|exo-locale + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|exo.morph|exo|exo-libs + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|exo.morph|exo|exo-bins + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-locale + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-libs + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|libxfce4ui.morph|libxfce4ui|libxfce4ui-bins + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|xfconf.morph|xfconf|xfconf-misc + test-repo|master|xfconf.morph|xfconf|xfconf-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|xfconf.morph|xfconf|xfconf-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|xfconf.morph|xfconf|xfconf-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|xfconf.morph|xfconf|xfconf-locale + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|xfconf.morph|xfconf|xfconf-libs + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|xfconf.morph|xfconf|xfconf-bins + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-doc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-misc + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-locale + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-libs + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|libxfce4util.morph|libxfce4util|libxfce4util-bins + -> test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + test-repo|master|gtk-stack.morph|gtk-stack-runtime|gtk-stack-runtime + -> test-repo|master|cairo.morph|cairo|cairo-bins + -> test-repo|master|cairo.morph|cairo|cairo-libs + -> test-repo|master|cairo.morph|cairo|cairo-locale + -> test-repo|master|cairo.morph|cairo|cairo-misc + -> test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-bins + -> test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-libs + -> test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-locale + -> test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-misc + -> test-repo|master|dbus.morph|dbus|dbus-bins + -> test-repo|master|dbus.morph|dbus|dbus-libs + -> test-repo|master|dbus.morph|dbus|dbus-locale + -> test-repo|master|dbus.morph|dbus|dbus-misc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-bins + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-libs + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-locale + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-misc + -> test-repo|master|freetype.morph|freetype|freetype-bins + -> test-repo|master|freetype.morph|freetype|freetype-libs + -> test-repo|master|freetype.morph|freetype|freetype-locale + -> test-repo|master|freetype.morph|freetype|freetype-misc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-bins + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-libs + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-locale + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + -> test-repo|master|gtk.morph|gtk|gtk-bins + -> test-repo|master|gtk.morph|gtk|gtk-libs + -> test-repo|master|gtk.morph|gtk|gtk-locale + -> test-repo|master|gtk.morph|gtk|gtk-misc + -> test-repo|master|pango.morph|pango|pango-bins + -> test-repo|master|pango.morph|pango|pango-libs + -> test-repo|master|pango.morph|pango|pango-locale + -> test-repo|master|pango.morph|pango|pango-misc + test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-misc + -> test-repo|master|dbus.morph|dbus|dbus-bins + -> test-repo|master|dbus.morph|dbus|dbus-devel + -> test-repo|master|dbus.morph|dbus|dbus-doc + -> test-repo|master|dbus.morph|dbus|dbus-libs + -> test-repo|master|dbus.morph|dbus|dbus-locale + -> test-repo|master|dbus.morph|dbus|dbus-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-locale + -> test-repo|master|dbus.morph|dbus|dbus-bins + -> test-repo|master|dbus.morph|dbus|dbus-devel + -> test-repo|master|dbus.morph|dbus|dbus-doc + -> test-repo|master|dbus.morph|dbus|dbus-libs + -> test-repo|master|dbus.morph|dbus|dbus-locale + -> test-repo|master|dbus.morph|dbus|dbus-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-libs + -> test-repo|master|dbus.morph|dbus|dbus-bins + -> test-repo|master|dbus.morph|dbus|dbus-devel + -> test-repo|master|dbus.morph|dbus|dbus-doc + -> test-repo|master|dbus.morph|dbus|dbus-libs + -> test-repo|master|dbus.morph|dbus|dbus-locale + -> test-repo|master|dbus.morph|dbus|dbus-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-bins + -> test-repo|master|dbus.morph|dbus|dbus-bins + -> test-repo|master|dbus.morph|dbus|dbus-devel + -> test-repo|master|dbus.morph|dbus|dbus-doc + -> test-repo|master|dbus.morph|dbus|dbus-libs + -> test-repo|master|dbus.morph|dbus|dbus-locale + -> test-repo|master|dbus.morph|dbus|dbus-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|gtk.morph|gtk|gtk-misc + -> test-repo|master|cairo.morph|cairo|cairo-bins + -> test-repo|master|cairo.morph|cairo|cairo-devel + -> test-repo|master|cairo.morph|cairo|cairo-doc + -> test-repo|master|cairo.morph|cairo|cairo-libs + -> test-repo|master|cairo.morph|cairo|cairo-locale + -> test-repo|master|cairo.morph|cairo|cairo-misc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-bins + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-devel + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-doc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-libs + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-locale + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + -> test-repo|master|pango.morph|pango|pango-bins + -> test-repo|master|pango.morph|pango|pango-devel + -> test-repo|master|pango.morph|pango|pango-doc + -> test-repo|master|pango.morph|pango|pango-libs + -> test-repo|master|pango.morph|pango|pango-locale + -> test-repo|master|pango.morph|pango|pango-misc + test-repo|master|gtk.morph|gtk|gtk-locale + -> test-repo|master|cairo.morph|cairo|cairo-bins + -> test-repo|master|cairo.morph|cairo|cairo-devel + -> test-repo|master|cairo.morph|cairo|cairo-doc + -> test-repo|master|cairo.morph|cairo|cairo-libs + -> test-repo|master|cairo.morph|cairo|cairo-locale + -> test-repo|master|cairo.morph|cairo|cairo-misc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-bins + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-devel + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-doc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-libs + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-locale + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + -> test-repo|master|pango.morph|pango|pango-bins + -> test-repo|master|pango.morph|pango|pango-devel + -> test-repo|master|pango.morph|pango|pango-doc + -> test-repo|master|pango.morph|pango|pango-libs + -> test-repo|master|pango.morph|pango|pango-locale + -> test-repo|master|pango.morph|pango|pango-misc + test-repo|master|gtk.morph|gtk|gtk-libs + -> test-repo|master|cairo.morph|cairo|cairo-bins + -> test-repo|master|cairo.morph|cairo|cairo-devel + -> test-repo|master|cairo.morph|cairo|cairo-doc + -> test-repo|master|cairo.morph|cairo|cairo-libs + -> test-repo|master|cairo.morph|cairo|cairo-locale + -> test-repo|master|cairo.morph|cairo|cairo-misc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-bins + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-devel + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-doc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-libs + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-locale + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + -> test-repo|master|pango.morph|pango|pango-bins + -> test-repo|master|pango.morph|pango|pango-devel + -> test-repo|master|pango.morph|pango|pango-doc + -> test-repo|master|pango.morph|pango|pango-libs + -> test-repo|master|pango.morph|pango|pango-locale + -> test-repo|master|pango.morph|pango|pango-misc + test-repo|master|gtk.morph|gtk|gtk-bins + -> test-repo|master|cairo.morph|cairo|cairo-bins + -> test-repo|master|cairo.morph|cairo|cairo-devel + -> test-repo|master|cairo.morph|cairo|cairo-doc + -> test-repo|master|cairo.morph|cairo|cairo-libs + -> test-repo|master|cairo.morph|cairo|cairo-locale + -> test-repo|master|cairo.morph|cairo|cairo-misc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-bins + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-devel + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-doc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-libs + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-locale + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + -> test-repo|master|pango.morph|pango|pango-bins + -> test-repo|master|pango.morph|pango|pango-devel + -> test-repo|master|pango.morph|pango|pango-doc + -> test-repo|master|pango.morph|pango|pango-libs + -> test-repo|master|pango.morph|pango|pango-locale + -> test-repo|master|pango.morph|pango|pango-misc + test-repo|master|gtk-stack.morph|gtk-stack-devel|gtk-stack-devel + -> test-repo|master|cairo.morph|cairo|cairo-devel + -> test-repo|master|cairo.morph|cairo|cairo-doc + -> test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-devel + -> test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-doc + -> test-repo|master|dbus.morph|dbus|dbus-devel + -> test-repo|master|dbus.morph|dbus|dbus-doc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-devel + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-doc + -> test-repo|master|freetype.morph|freetype|freetype-devel + -> test-repo|master|freetype.morph|freetype|freetype-doc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-devel + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-doc + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|gtk.morph|gtk|gtk-devel + -> test-repo|master|gtk.morph|gtk|gtk-doc + -> test-repo|master|pango.morph|pango|pango-devel + -> test-repo|master|pango.morph|pango|pango-doc + test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-doc + -> test-repo|master|dbus.morph|dbus|dbus-bins + -> test-repo|master|dbus.morph|dbus|dbus-devel + -> test-repo|master|dbus.morph|dbus|dbus-doc + -> test-repo|master|dbus.morph|dbus|dbus-libs + -> test-repo|master|dbus.morph|dbus|dbus-locale + -> test-repo|master|dbus.morph|dbus|dbus-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|dbus-glib.morph|dbus-glib|dbus-glib-devel + -> test-repo|master|dbus.morph|dbus|dbus-bins + -> test-repo|master|dbus.morph|dbus|dbus-devel + -> test-repo|master|dbus.morph|dbus|dbus-doc + -> test-repo|master|dbus.morph|dbus|dbus-libs + -> test-repo|master|dbus.morph|dbus|dbus-locale + -> test-repo|master|dbus.morph|dbus|dbus-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|dbus.morph|dbus|dbus-misc + test-repo|master|dbus.morph|dbus|dbus-locale + test-repo|master|dbus.morph|dbus|dbus-libs + test-repo|master|dbus.morph|dbus|dbus-bins + test-repo|master|dbus.morph|dbus|dbus-doc + test-repo|master|dbus.morph|dbus|dbus-devel + test-repo|master|gtk.morph|gtk|gtk-doc + -> test-repo|master|cairo.morph|cairo|cairo-bins + -> test-repo|master|cairo.morph|cairo|cairo-devel + -> test-repo|master|cairo.morph|cairo|cairo-doc + -> test-repo|master|cairo.morph|cairo|cairo-libs + -> test-repo|master|cairo.morph|cairo|cairo-locale + -> test-repo|master|cairo.morph|cairo|cairo-misc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-bins + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-devel + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-doc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-libs + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-locale + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + -> test-repo|master|pango.morph|pango|pango-bins + -> test-repo|master|pango.morph|pango|pango-devel + -> test-repo|master|pango.morph|pango|pango-doc + -> test-repo|master|pango.morph|pango|pango-libs + -> test-repo|master|pango.morph|pango|pango-locale + -> test-repo|master|pango.morph|pango|pango-misc + test-repo|master|gtk.morph|gtk|gtk-devel + -> test-repo|master|cairo.morph|cairo|cairo-bins + -> test-repo|master|cairo.morph|cairo|cairo-devel + -> test-repo|master|cairo.morph|cairo|cairo-doc + -> test-repo|master|cairo.morph|cairo|cairo-libs + -> test-repo|master|cairo.morph|cairo|cairo-locale + -> test-repo|master|cairo.morph|cairo|cairo-misc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-bins + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-devel + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-doc + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-libs + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-locale + -> test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + -> test-repo|master|pango.morph|pango|pango-bins + -> test-repo|master|pango.morph|pango|pango-devel + -> test-repo|master|pango.morph|pango|pango-doc + -> test-repo|master|pango.morph|pango|pango-libs + -> test-repo|master|pango.morph|pango|pango-locale + -> test-repo|master|pango.morph|pango|pango-misc + test-repo|master|pango.morph|pango|pango-misc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-bins + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-devel + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-doc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-libs + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-locale + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-misc + -> test-repo|master|freetype.morph|freetype|freetype-bins + -> test-repo|master|freetype.morph|freetype|freetype-devel + -> test-repo|master|freetype.morph|freetype|freetype-doc + -> test-repo|master|freetype.morph|freetype|freetype-libs + -> test-repo|master|freetype.morph|freetype|freetype-locale + -> test-repo|master|freetype.morph|freetype|freetype-misc + test-repo|master|pango.morph|pango|pango-locale + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-bins + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-devel + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-doc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-libs + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-locale + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-misc + -> test-repo|master|freetype.morph|freetype|freetype-bins + -> test-repo|master|freetype.morph|freetype|freetype-devel + -> test-repo|master|freetype.morph|freetype|freetype-doc + -> test-repo|master|freetype.morph|freetype|freetype-libs + -> test-repo|master|freetype.morph|freetype|freetype-locale + -> test-repo|master|freetype.morph|freetype|freetype-misc + test-repo|master|pango.morph|pango|pango-libs + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-bins + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-devel + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-doc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-libs + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-locale + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-misc + -> test-repo|master|freetype.morph|freetype|freetype-bins + -> test-repo|master|freetype.morph|freetype|freetype-devel + -> test-repo|master|freetype.morph|freetype|freetype-doc + -> test-repo|master|freetype.morph|freetype|freetype-libs + -> test-repo|master|freetype.morph|freetype|freetype-locale + -> test-repo|master|freetype.morph|freetype|freetype-misc + test-repo|master|pango.morph|pango|pango-bins + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-bins + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-devel + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-doc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-libs + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-locale + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-misc + -> test-repo|master|freetype.morph|freetype|freetype-bins + -> test-repo|master|freetype.morph|freetype|freetype-devel + -> test-repo|master|freetype.morph|freetype|freetype-doc + -> test-repo|master|freetype.morph|freetype|freetype-libs + -> test-repo|master|freetype.morph|freetype|freetype-locale + -> test-repo|master|freetype.morph|freetype|freetype-misc + test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-misc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-locale + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-libs + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-bins + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|cairo.morph|cairo|cairo-misc + test-repo|master|cairo.morph|cairo|cairo-locale + test-repo|master|cairo.morph|cairo|cairo-libs + test-repo|master|cairo.morph|cairo|cairo-bins + test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-doc + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|gdk-pixbuf.morph|gdk-pixbuf|gdk-pixbuf-devel + -> test-repo|master|glib.morph|glib|glib-bins + -> test-repo|master|glib.morph|glib|glib-devel + -> test-repo|master|glib.morph|glib|glib-doc + -> test-repo|master|glib.morph|glib|glib-libs + -> test-repo|master|glib.morph|glib|glib-locale + -> test-repo|master|glib.morph|glib|glib-misc + test-repo|master|glib.morph|glib|glib-misc + test-repo|master|glib.morph|glib|glib-locale + test-repo|master|glib.morph|glib|glib-libs + test-repo|master|glib.morph|glib|glib-bins + test-repo|master|glib.morph|glib|glib-doc + test-repo|master|glib.morph|glib|glib-devel + test-repo|master|pango.morph|pango|pango-doc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-bins + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-devel + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-doc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-libs + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-locale + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-misc + -> test-repo|master|freetype.morph|freetype|freetype-bins + -> test-repo|master|freetype.morph|freetype|freetype-devel + -> test-repo|master|freetype.morph|freetype|freetype-doc + -> test-repo|master|freetype.morph|freetype|freetype-libs + -> test-repo|master|freetype.morph|freetype|freetype-locale + -> test-repo|master|freetype.morph|freetype|freetype-misc + test-repo|master|pango.morph|pango|pango-devel + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-bins + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-devel + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-doc + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-libs + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-locale + -> test-repo|master|fontconfig.morph|fontconfig|fontconfig-misc + -> test-repo|master|freetype.morph|freetype|freetype-bins + -> test-repo|master|freetype.morph|freetype|freetype-devel + -> test-repo|master|freetype.morph|freetype|freetype-doc + -> test-repo|master|freetype.morph|freetype|freetype-libs + -> test-repo|master|freetype.morph|freetype|freetype-locale + -> test-repo|master|freetype.morph|freetype|freetype-misc + test-repo|master|fontconfig.morph|fontconfig|fontconfig-misc + test-repo|master|fontconfig.morph|fontconfig|fontconfig-locale + test-repo|master|fontconfig.morph|fontconfig|fontconfig-libs + test-repo|master|fontconfig.morph|fontconfig|fontconfig-bins + test-repo|master|freetype.morph|freetype|freetype-misc + test-repo|master|freetype.morph|freetype|freetype-locale + test-repo|master|freetype.morph|freetype|freetype-libs + test-repo|master|freetype.morph|freetype|freetype-bins + test-repo|master|cairo.morph|cairo|cairo-doc + test-repo|master|cairo.morph|cairo|cairo-devel + test-repo|master|fontconfig.morph|fontconfig|fontconfig-doc + test-repo|master|fontconfig.morph|fontconfig|fontconfig-devel + test-repo|master|freetype.morph|freetype|freetype-doc + test-repo|master|freetype.morph|freetype|freetype-devel diff --git a/tests/trove-id.script b/tests/trove-id.script new file mode 100755 index 00000000..998bde44 --- /dev/null +++ b/tests/trove-id.script @@ -0,0 +1,100 @@ +#!/bin/sh +# +# Copyright (C) 2012-2013 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +## Verify that trove-id (and by corollary trove-host) work properly. + +set -eu + +RAWDUMP="$DATADIR/raw-configdump" +PROCESSEDDUMP="$DATADIR/processed-configdump" + +# Step 1, gather all the raw and processed repo-alias entries + +"$SRCDIR/scripts/test-morph" \ + --trove-host="TROVEHOST" \ + --trove-id="fudge" \ + --trove-id="github" \ + --dump-config > "$RAWDUMP" +env MORPH_DUMP_PROCESSED_CONFIG=1 "$SRCDIR/scripts/test-morph" \ + --trove-host="TROVEHOST" \ + --trove-id="fudge" \ + --trove-id="github" \ + > "$PROCESSEDDUMP" + +RAW_ALIAS=$(grep repo-alias "$RAWDUMP" | cut -d\ -f3-) +PROCESSED_ALIAS=$(grep repo-alias "$PROCESSEDDUMP" | cut -d\ -f3-) + +find_alias () { + ALIASES="$1" + WHICH="$2" + for alias in $ALIASES; do + alias=$(echo $alias | sed -e's/,$//') + prefix=$(echo $alias | cut -d= -f1) + if test "x$WHICH" = "x$prefix"; then + echo $alias + exit 0 + fi + done +} + +# Step 2, all raw aliases should be in processed aliases unchanged. As part of +# this, we're also validating that the 'github' prefix we pass in does not +# affect the alias output since it is overridden by repo-alias. + +for raw_alias in $RAW_ALIAS; do + raw_alias=$(echo $raw_alias | sed -e's/,$//') + raw_prefix=$(echo $raw_alias | cut -d= -f1) + processed_alias=$(find_alias "$PROCESSED_ALIAS" "$raw_prefix") + if test "x$raw_alias" != "x$processed_alias"; then + echo >&2 "Raw $raw_alias not in processed aliases" + fi +done + +# Step 3, all aliases in the processed aliases which do not come from the raw +# aliases should contain the trove host. + +for processed_alias in $PROCESSED_ALIAS; do + processed_alias=$(echo $processed_alias | sed -e's/,$//') + processed_prefix=$(echo $processed_alias | cut -d= -f1) + raw_alias=$(find_alias "$RAW_ALIAS" "$processed_prefix") + if test "x$raw_alias" = "x"; then + grep_out=$(echo "$processed_alias" | grep TROVEHOST) + if test "x$grep_out" = "x"; then + echo >&2 "Processed $processed_alias does not mention TROVEHOST" + fi + fi +done + +# Step 4, validate that the processed aliases do contain a baserock and an +# upstream alias since those are implicit in morph's behaviour. + +for prefix in baserock upstream; do + processed_alias=$(find_alias "$PROCESSED_ALIAS" "$prefix") + if test "x$processed_alias" = "x"; then + echo >&2 "Processed aliases lack $prefix prefix" + fi +done + +# Step 5, validate that the fudge prefix has been correctly expanded as though +# it were fudge=fudge#ssh#ssh + +fudge_alias=$(find_alias "$PROCESSED_ALIAS" "fudge") +desired_fudge="fudge=ssh://git@TROVEHOST/fudge/%s#ssh://git@TROVEHOST/fudge/%s" +if test "x$fudge_alias" != "x$desired_fudge"; then + echo >&2 "Fudge alias was '$fudge_alias' where we wanted '$desired_fudge'" +fi diff --git a/without-test-modules b/without-test-modules new file mode 100644 index 00000000..55e5291d --- /dev/null +++ b/without-test-modules @@ -0,0 +1,54 @@ +morphlib/__init__.py +morphlib/artifactcachereference.py +morphlib/artifactsplitrule.py +morphlib/builddependencygraph.py +morphlib/tester.py +morphlib/git.py +morphlib/app.py +morphlib/mountableimage.py +morphlib/extensions.py +morphlib/extractedtarball.py +morphlib/plugins/artifact_inspection_plugin.py +morphlib/plugins/cross-bootstrap_plugin.py +morphlib/plugins/hello_plugin.py +morphlib/plugins/graphing_plugin.py +morphlib/plugins/syslinux-disk-systembuilder_plugin.py +morphlib/plugins/disk-systembuilder_plugin.py +morphlib/plugins/tarball-systembuilder_plugin.py +morphlib/plugins/show_dependencies_plugin.py +morphlib/plugins/branch_and_merge_plugin.py +morphlib/buildcommand.py +morphlib/plugins/build_plugin.py +morphlib/gitversion.py +morphlib/plugins/expand_repo_plugin.py +morphlib/plugins/deploy_plugin.py +morphlib/plugins/__init__.py +morphlib/writeexts.py +morphlib/plugins/list_artifacts_plugin.py +morphlib/plugins/trovectl_plugin.py +morphlib/plugins/gc_plugin.py +morphlib/plugins/print_architecture_plugin.py +morphlib/plugins/add_binary_plugin.py +morphlib/plugins/push_pull_plugin.py +morphlib/plugins/distbuild_plugin.py +distbuild/__init__.py +distbuild/build_controller.py +distbuild/connection_machine.py +distbuild/distbuild_socket.py +distbuild/eventsrc.py +distbuild/helper_router.py +distbuild/idgen.py +distbuild/initiator.py +distbuild/initiator_connection.py +distbuild/jm.py +distbuild/json_router.py +distbuild/mainloop.py +distbuild/protocol.py +distbuild/proxy_event_source.py +distbuild/sockbuf.py +distbuild/socketsrc.py +distbuild/sockserv.py +distbuild/timer_event_source.py +distbuild/worker_build_scheduler.py +# Not unit tested, since it needs a full system branch +morphlib/buildbranch.py diff --git a/yarns/architecture.yarn b/yarns/architecture.yarn new file mode 100644 index 00000000..07274ec3 --- /dev/null +++ b/yarns/architecture.yarn @@ -0,0 +1,36 @@ +Morph Cross-Building Tests +========================== + + SCENARIO building a system for a different architecture + GIVEN a workspace + AND a git server + AND a system called base-system-testarch.morph for the test architecture in the git server + WHEN the user checks out the system branch called master + AND the user attempts to build the system base-system-testarch.morph in branch master + THEN morph failed + AND the build error message includes the string "Are you trying to cross-build?" + FINALLY the git server is shut down + + +Morph Cross-Bootstrap Tests +=========================== + + SCENARIO cross-bootstraping a system for a different architecture + GIVEN a workspace + AND a git server + AND a system called base-system-testarch.morph for the test architecture in the git server + WHEN the user checks out the system branch called master + THEN the user cross-bootstraps the system base-system-testarch.morph in branch master of repo test:morphs to the arch testarch + FINALLY the git server is shut down + +Architecture validation Tests +============================= + + SCENARIO building a system with no architecture + GIVEN a workspace + AND a git server + AND a system called base-system-noarch.morph with no architecture in the git server + WHEN the user checks out the system branch called master + AND the user attempts to build the system base-system-testarch.morph in branch master + THEN morph failed + FINALLY the git server is shut down diff --git a/yarns/branches-workspaces.yarn b/yarns/branches-workspaces.yarn new file mode 100644 index 00000000..34aa97e0 --- /dev/null +++ b/yarns/branches-workspaces.yarn @@ -0,0 +1,469 @@ +Morph black box tests for system branches and workspaces +======================================================== + +Morph implements **system branches**, which are checked out and +manipulated by the user in **workspaces**. See +FIXME for more information. + +Workspace creation +------------------ + +The first thing a user needs to do is create a workspace. + + SCENARIO create and initialise a new workspace + GIVEN no workspace + WHEN the user initialises a workspace + THEN an empty workspace exists + +The workspace directory may exist, if it is empty. + + SCENARIO initialise an empty workspace directory + GIVEN an empty workspace directory + WHEN the user initialises a workspace + THEN an empty workspace exists + +However, the directory must really be empty. It must not be +an empty, but initialised workspace. + + SCENARIO initialise an existing, empty workspace directory + GIVEN no workspace + WHEN the user initialises a workspace + AND the user attempts to initialise a workspace + THEN morph failed + +Likewise, if the directory exists, and is non-empty, but isn't an +existing workspace, initialising it should fail. + + SCENARIO initialise a non-empty workspace directory + GIVEN a non-empty workspace directory + WHEN the user attempts to initialise a workspace + THEN morph failed + +Checking out system branches +----------------------------------------- + +Once we have a workspace, we can check out a system branch. + + SCENARIO check out an existing system branch + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + THEN the system branch master is checked out + +Edit is probably not the best name for is, but we can use `morph edit` +to investigate chunks in existing branches. + + WHEN the user edits the chunk test-chunk in branch master + THEN the edited chunk test:test-chunk has git branch master + FINALLY the git server is shut down + +Checking out a system branch should fail, if the branch doesn't exist. + + SCENARIO checking out a system branch that doesn't exist + GIVEN a workspace + AND a git server + WHEN the user attempts to check out the system branch called foo + THEN morph failed + FINALLY the git server is shut down + +Branching system branches +----------------------------------------- + +We can, instead, create a new system branch, off master. + + SCENARIO branch off master + GIVEN a workspace + AND a git server + WHEN the user creates a system branch called foo + THEN the system branch foo is checked out + FINALLY the git server is shut down + +We can also branch off another system branch. However, we need to first +push the other branch to the git server, since Morph is not smart enough +to check for that locally. + + SCENARIO branch off non-master + GIVEN a workspace + AND a git server + WHEN the user creates a system branch called foo + AND the user pushes the system branch called foo to the git server + AND the user creates a system branch called bar, based on foo + THEN the system branch bar is checked out + FINALLY the git server is shut down + +Query commands in workspaces +---------------------------- + +`morph workspace` writes out the fully qualified path to the workspace +directory, regardless of where the user is. There's a few cases. + + SCENARIO morph workspace works at root of empty workspace + GIVEN a workspace + WHEN the user reports the workspace from the directory . + THEN the workspace is reported correctly + +Also check it in the root of a system branch checkout, and inside +a git checkout inside that. + + SCENARIO morph workspace works in system branch checkouts + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user reports the workspace from the directory master + THEN the workspace is reported correctly + +We leak a little bit of the implementation here, to keep things simple: +the (mocked) git server the implementation sets up has the `test:morphs` +repository, which is the system branch root repository. + + WHEN the user reports the workspace from the directory master/test/morphs + THEN the workspace is reported correctly + FINALLY the git server is shut down + +However, running it outside a workspace should fail. + + SCENARIO morph fails outside workspace + GIVEN no workspace + WHEN the user attempts to report the workspace from a non-workspace directory + THEN morph failed + +`morph show-system-branch` should report the name of the system +branch, when run anywhere in the system branch checkout. As a special +case, if there is only one system branch checkout at or below the +current working directory, it will find it and report it correctly. + + SCENARIO morph reports system branch + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user reports the system branch from the directory master + THEN the system branch is reported as master + + WHEN the user reports the system branch from the directory master/test/morphs + THEN the system branch is reported as master + + WHEN the user reports the system branch from the directory . + THEN the system branch is reported as master + FINALLY the git server is shut down + +However, if there's two system branches checked out below the +current directory, things should fail. + + SCENARIO morph fails to report system branch with two checked out + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user creates a system branch called foo + AND the user attempts to report the system branch from the directory . + THEN morph failed + FINALLY the git server is shut down + +`morph show-branch-root` reports the path of the system branch root +repository. It can be run inside a checkout, or somewhere outside a +checkout, where exactly one checkout exists below. + + SCENARIO morph reports system branch root repository + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user reports the system branch root repository from the directory master + THEN the system branch root repository is reported as workspace/master/test/morphs + + WHEN the user reports the system branch root repository from the directory . + THEN the system branch root repository is reported as workspace/master/test/morphs + FINALLY the git server is shut down + +However, it fails if run outside a checkout and there's no system +branches checked out. + + SCENARIO morph fails to report system branch with none checked out + GIVEN a workspace + AND a git server + WHEN the user attempts to report the system branch root repository from the directory . + THEN morph failed + FINALLY the git server is shut down + +Editing components +------------------ + +`morph edit` can edit refs for a chunk, and check out the chunk's +repository. + +First of all, we verify that that when we create a system branch, +all the refs are unchanged. + + SCENARIO morph branch does not edit refs + GIVEN a workspace + AND a git server + WHEN the user creates a system branch called foo + +Edit the chunk. We make use of special knowledge here: `test:test-chunk` +is a chunk repository created in the mocked git server, for testing +purposes. + + WHEN the user edits the chunk test-chunk in branch foo + THEN in branch foo, stratum strata/core.morph refs test-chunk in foo + AND the edited chunk test:test-chunk has git branch foo + +Editing a morphology should not cause it to start having repo or ref +fields when referring to strata, when it didn't before. + + AND in branch foo, system systems/test-system.morph refers to core without repo + AND in branch foo, system systems/test-system.morph refers to core without ref + FINALLY the git server is shut down + +Temporary Build Branch behaviour +-------------------------------- + +Morph always builds from committed changes, but it's not always convenient +to commit and push changes, so `morph build` can create temporary build +branches when necessary. + +### What gets included in temporary build branches ### + + SCENARIO morph builds the branches of edited chunks you checked-out + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user edits the chunk test-chunk in branch master + +If we make an uncommitted change to an edited chunk, then a temporary +build branch is made to include that change. + + WHEN the user makes changes to test-chunk in branch master + AND the user builds systems/test-system.morph of the master branch + THEN the changes to test-chunk in branch master are included in the temporary build branch + +### When branches are created ### + +It's convenient to have Temporary Build Branches, but we don't always +need them, and they get in the way when we don't need them, so we need +to be careful about when to make them. + + SCENARIO morph makes temporary build branches for uncommitted changes when necessary + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + +The user hasn't made any changes yet, so attempts to build require no +temporary build branches. + + GIVEN the workspace contains no temporary build branches + AND we can build with local branches + WHEN the user builds systems/test-system.morph of the master branch + THEN the morphs repository in the workspace for master has no temporary build branches + +Similarly, if we need to build from pushed branches, such as when we're +distbuilding, we don't need temporary build branches yet, since we have +no local changes. + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we must build from pushed branches + WHEN the user builds systems/test-system.morph of the master branch + THEN the morphs repository in the workspace for master has no temporary build branches + AND no temporary build branches were pushed to the morphs repository + +If we actually want to be able to push our changes for review, we need to +use a different branch from master, since we require code to be reviewed +then merged, rather than pushing directly to master. + + WHEN the user creates a system branch called baserock/test + +When we start making changes we do need temporary build branches, since +the chunk specifiers in the strata now need to refer to the local changes +to the repository. + + WHEN the user edits the chunk test-chunk in branch baserock/test + +If we don't need to build from pushed branches then we have temporary +build branches only in the local clones of the repositories. + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we can build with local branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has temporary build branches + AND no temporary build branches were pushed to the morphs repository + +If we do need to build from pushed changes, then the temporary build +branch needs to be pushed. + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we must build from pushed branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has temporary build branches + AND temporary build branches were pushed to the morphs repository + +NOTE: We're not checking whether the test-chunk repo has changes since +it's currently an implementation detail that it does, but it would +be possible to build without a temporary build branch for the chunk +repository. + +Now that we have the chunk repository available, we can make our changes. + + WHEN the user makes changes to test-chunk in branch baserock/test + +When we have uncommitted changes to chunk repositories, we need +temporary build branches locally for local builds. + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we can build with local branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has temporary build branches + AND the test-chunk repository in the workspace for baserock/test has temporary build branches + AND no temporary build branches were pushed to the morphs repository + AND no temporary build branches were pushed to the test-chunk repository + +As before, we also need temporary build branches to have been pushed + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we must build from pushed branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has temporary build branches + AND the test-chunk repository in the workspace for baserock/test has temporary build branches + AND temporary build branches were pushed to the morphs repository + AND temporary build branches were pushed to the test-chunk repository + +Now that we've made our changes, we can commit them. + + WHEN the user commits changes to morphs in branch baserock/test + AND the user commits changes to test-chunk in branch baserock/test + +For local builds we should be able to use these committed changes, +provided the ref in the morphology matches the committed ref in the +chunk repository. + +However, since we do not currently do this integrity check, as it requires +extra tracking between edited morphologies and the local repositories, +it's easier to just require a temporary build branch. + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we can build with local branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has temporary build branches + AND the test-chunk repository in the workspace for baserock/test has temporary build branches + AND no temporary build branches were pushed to the morphs repository + AND no temporary build branches were pushed to the test-chunk repository + +For distributed building, it being committed locally is not sufficient, +as remote workers need to be able to access the changes, and dist-build +workers tunneling into the developer's machine and using those +repositories would be madness, so we require temporary build branches +to be pushed. + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we must build from pushed branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has temporary build branches + AND the test-chunk repository in the workspace for baserock/test has temporary build branches + AND temporary build branches were pushed to the morphs repository + AND temporary build branches were pushed to the test-chunk repository + +We can now push our committed changes. + + WHEN the user pushes the system branch called baserock/test to the git server + +We now don't need temporary build branches for local builds. + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we can build with local branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has no temporary build branches + AND the test-chunk repository in the workspace for baserock/test has no temporary build branches + AND no temporary build branches were pushed to the morphs repository + AND no temporary build branches were pushed to the test-chunk repository + +Nor do we need temporary build branches for distributed builds. + + GIVEN the workspace contains no temporary build branches + AND the git server contains no temporary build branches + AND we must build from pushed branches + WHEN the user builds systems/test-system.morph of the baserock/test branch + THEN the morphs repository in the workspace for baserock/test has no temporary build branches + AND the test-chunk repository in the workspace for baserock/test has no temporary build branches + AND no temporary build branches were pushed to the morphs repository + AND no temporary build branches were pushed to the test-chunk repository + FINALLY the git server is shut down + + +### Temporary Build Branch implementations ### + + IMPLEMENTS WHEN the user makes changes to test-chunk in branch (\S+) + chunkdir="$(slashify_colons "test:test-chunk")" + cd "$DATADIR/workspace/$MATCH_1/$chunkdir" + sed -i -e 's/Hello/Goodbye/g' usr/libexec/test-bin + + IMPLEMENTS THEN the changes to test-chunk in branch (\S+) are included in the temporary build branch + build_ref_prefix=baserock/builds/ + chunkdir="$(slashify_colons "test:test-chunk")" + cd "$DATADIR/workspace/$MATCH_1/$chunkdir" + testbin=usr/libexec/test-bin + eval "$(git for-each-ref --count=1 --shell --sort=committerdate \ + --format='git cat-file -p %(refname):$testbin | diff $testbin -' \ + "$build_ref_prefix")" + + IMPLEMENTS WHEN the user commits changes to (\S+) in branch (\S+) + chunkdir="$(slashify_colons "test:$MATCH_1")" + cd "$DATADIR/workspace/$MATCH_2/$chunkdir" + git commit -a -m 'Commit local changes' + + +Status of system branch checkout +-------------------------------- + +`morph status` shows the status of all git repositories in a +system branch checkout: only the ones that exist locally, not all the +repositories referenced in the system branch. + + SCENARIO morph status reports changes correctly + GIVEN a workspace + AND a git server + WHEN the user creates a system branch called foo + THEN morph reports no outstanding changes in foo + + WHEN the user edits the chunk test-chunk in branch foo + THEN morph reports changes in foo in test:morphs only + + WHEN creating file foo in test/test-chunk in branch foo + THEN morph reports changes in foo in test:morphs only + + WHEN adding file foo in test:test-chunk in branch foo to git + THEN morph reports changes in foo in test:morphs and test:test-chunk + + WHEN committing changes in test:morphs in branch foo + THEN morph reports changes in foo in test:test-chunk only + + WHEN committing changes in test:test-chunk in branch foo + THEN morph reports no outstanding changes in foo + FINALLY the git server is shut down + +`morph foreach` +-------------- + +`morph foreach` runs a shell command in each of the git repos in a system +branch checkout. + + SCENARIO morph foreach runs command in each git repo + GIVEN a workspace + AND a git server + WHEN the user creates a system branch called foo + AND the user edits the chunk test-chunk in branch foo + AND running shell command in each repo in foo + THEN morph ran command in test/morphs in foo + AND morph ran command in test/test-chunk in foo + FINALLY the git server is shut down + +Generating a manifest works + + SCENARIO morph generates a manifest + GIVEN a workspace + AND a system artifact + WHEN morph generates a manifest + THEN the manifest is generated diff --git a/yarns/building.yarn b/yarns/building.yarn new file mode 100644 index 00000000..c708b5bb --- /dev/null +++ b/yarns/building.yarn @@ -0,0 +1,10 @@ +Morph Building Tests +====================== + + SCENARIO attempting to build a system morphology which has never been committed + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user creates an uncommitted system morphology called systems/base-system.morph for our architecture in system branch master + THEN morph build the system systems/base-system.morph of the branch master + FINALLY the git server is shut down diff --git a/yarns/deployment.yarn b/yarns/deployment.yarn new file mode 100644 index 00000000..0782c7c1 --- /dev/null +++ b/yarns/deployment.yarn @@ -0,0 +1,330 @@ +Morph Deployment Tests +====================== + + SCENARIO deploying a non-cluster morphology + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user attempts to deploy the system systems/test-system.morph in branch master + THEN morph failed + AND the deploy error message includes the string "morph deployment commands are only supported for cluster morphologies" + FINALLY the git server is shut down + + SCENARIO deploying a cluster morphology as a tarfile + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + GIVEN a cluster called test-cluster.morph in system branch master + AND a system in cluster test-cluster.morph in branch master called test-system + AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch master has deployment type: tar + AND system test-system in cluster test-cluster.morph in branch master has deployment location: test.tar + WHEN the user builds the system systems/test-system.morph in branch master + AND the user attempts to deploy the cluster test-cluster.morph in branch master + THEN morph succeeded + FINALLY the git server is shut down + +Some deployment types support upgrades, but some do not and Morph needs to make +this clear. + + SCENARIO attempting to upgrade a tarfile deployment + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + GIVEN a cluster called test-cluster.morph in system branch master + AND a system in cluster test-cluster.morph in branch master called test-system + AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch master has deployment type: tar + AND system test-system in cluster test-cluster.morph in branch master has deployment location: test.tar + WHEN the user builds the system systems/test-system.morph in branch master + AND the user attempts to upgrade the cluster test-cluster.morph in branch master + THEN morph failed + FINALLY the git server is shut down + +The rawdisk write extension supports both initial deployment and subsequent +upgrades. Note that the rawdisk upgrade code needs bringing up to date to use +the new Baserock OS version manager tool. Also, the test deploys an identical +base OS as an upgrade. While pointless, this is permitted and does exercise +the same code paths as a real upgrade. + + SCENARIO deploying a cluster morphology as rawdisk and then upgrading it + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + GIVEN a cluster called test-cluster.morph in system branch master + AND a system in cluster test-cluster.morph in branch master called test-system + AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch master has deployment type: rawdisk + AND system test-system in cluster test-cluster.morph in branch master has deployment location: test.tar + WHEN the user builds the system systems/test-system.morph in branch master + AND the user attempts to deploy the cluster test-cluster.morph in branch master with options test-system.DISK_SIZE=20M test-system.VERSION_LABEL=test1 + THEN morph succeeded + WHEN the user attempts to upgrade the cluster test-cluster.morph in branch master with options test-system.VERSION_LABEL=test2 + THEN morph succeeded + FINALLY the git server is shut down + +Nested deployments +================== + +For the use-cases of: + +1. Installer CD/USB +2. NFS/VM host +3. System with multiple containerised applications +4. System with a toolchain targetting the sysroot of another target +5. Any nested combination of the above + +It is convenient to be able to deploy one system inside another. + + SCENARIO deploying a cluster morphology with nested systems + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + GIVEN a cluster called test-cluster.morph in system branch master + AND a system in cluster test-cluster.morph in branch master called test-system + AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch master has deployment type: tar + +After the usual setup, we also add a subsystem to the cluster. + + GIVEN a subsystem in cluster test-cluster.morph in branch master called test-system.sysroot + AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master has deployment type: sysroot + +We specify the location as a file path, this is relative to the parent +system's extracted rootfs, before it is configured. + + AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master has deployment location: var/lib/sysroots/test-system + WHEN the user builds the system systems/test-system.morph in branch master + AND the user attempts to deploy the cluster test-cluster.morph in branch master with options test-system.location="$DATADIR/test.tar" + THEN morph succeeded + +Morph succeeding alone is not sufficient to check whether it actually +worked, since if it ignored the subsystems field, or got the location +wrong for the subsystem. To actually test it, we have to check that our +deployed system contains the other. Since the baserock directory is in +every system, we can check for that. + + AND tarball test.tar contains var/lib/sysroots/test-system/baserock + FINALLY the git server is shut down + +Initramfs deployments +===================== + +There's a few ways of creating an initramfs. We could: +1. Build a sysroot and: + 1. Have a chunk turn that into a cpio archive, written into /boot. + 2. Embed it in the Linux kernel image, having the initramfs as part + of the BSP. +2. Deploy an existing system as a cpio archive + 1. As a stand-alone system, without a rootfs + 2. Nested inside another system + +1.1 and 1.2 require system engineering work, so won't be mentioned here. + + SCENARIO deploying a system with an initramfs + ASSUMING there is space for 5 512M disk images + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + GIVEN a cluster called C.morph in system branch master + AND a system in cluster C.morph in branch master called S + +2.2 needs a nested system that is deployed with the initramfs write +extension. + + GIVEN a subsystem in cluster C.morph in branch master called S.I + AND subsystem S.I in cluster C.morph in branch master builds systems/test-system.morph + AND subsystem S.I in cluster C.morph in branch master has deployment type: initramfs + +The nested system needs to be placed somewhere in the parent. The +traditional place for an initramfs is `/boot`. + + AND subsystem S.I in cluster C.morph in branch master has deployment location: boot/initramfs.gz + +1.1 and 2.2 need the write extension to configure the boot-loader to +use the produced initramfs. Only write extensions that involve creating a disk image care, so we'll use `rawdisk.write`. + + GIVEN system S in cluster C.morph in branch master builds systems/test-system.morph + AND system S in cluster C.morph in branch master has deployment type: rawdisk + AND system S in cluster C.morph in branch master has deployment location: test.img + AND system S in cluster C.morph in branch master has deployment variable: DISK_SIZE=512M + +Initramfs support is triggered by the `INITRAMFS_PATH` variable. It could have been made automatic, triggering the behaviour if `/boot/initramfs.gz` exists, but: + +1. There are a bunch of possible names, some of which imply different formats. +2. If we decide on one specific name, how do we pick. +3. If we allow multiple possible names, how do we handle multiple being possible. +4. We may need to pick a non-standard name: e.g. We have a deployment + where the system loads a kernel and initramfs from a disk, then boots + the target in KVM, so the bootloader we want to use for the guest is + `initramfs.gz`, while the host's initramfs is `hyp-initramfs.gz`. +5. We may have the initramfs come from a chunk the system built, but + for speed, we want this particular deployment not to use an initramfs, + even though we have a generic image that may support one. + +For all these reasons, despite there being redundancy in some cases, +we're going to set `INITRAMFS_PATH` to the same as the nested deployment's +location. + + GIVEN system S in cluster C.morph in branch master has deployment variable: INITRAMFS_PATH=boot/initramfs.gz + +Fully testing that the system is bootable requires a lot more time, +infrastructure and dependencies, so we're just going to build it and +inspect the result of the deployment. + + WHEN the user builds the system systems/test-system.morph in branch master + AND the user attempts to deploy the cluster C.morph in branch master + THEN morph succeeded + AND file workspace/master/test/morphs/test.img exists + +If the initramfs write extension works, the rootfs image should contain +`boot/initramfs.gz`. + + WHEN disk image workspace/master/test/morphs/test.img is mounted at mnt + THEN file mnt/systems/default/run/boot/initramfs.gz exists + +If the `rawdisk` write extension worked, then the bootloader config file +will mention the initramfs, and the UUID of the disk. + + AND file mnt/extlinux.conf matches initramfs + AND file mnt/extlinux.conf matches root=UUID= + FINALLY mnt is unmounted + AND the git server is shut down + +Partial deployments +=================== + +Deploy part of a cluster +------------------------ + +Starting from the well-defined position of having a cluster morphology +with only one definition. + + SCENARIO partially deploying a cluster morphology + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user builds the system systems/test-system.morph in branch master + GIVEN a cluster called test-cluster.morph in system branch master + AND a system in cluster test-cluster.morph in branch master called test-system + AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch master has deployment type: tar + AND system test-system in cluster test-cluster.morph in branch master has deployment location: test-system.tar + +It is useful to group related deployments together, so we support adding +another deployment to the same cluster morphology. + + GIVEN a system in cluster test-cluster.morph in branch master called second-system + AND system second-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system second-system in cluster test-cluster.morph in branch master has deployment type: tar + AND system second-system in cluster test-cluster.morph in branch master has deployment location: second-system.tar + +When we don't tell `morph deploy` which system we want to deploy, all +of the systems in the cluster are deployed. Here a successful deployment +will have morph exit sucessfully and in the case of tarball deployments, +the tarballs for both the systems will be created. + + WHEN the user attempts to deploy the cluster test-cluster.morph in branch master + THEN morph succeeded + AND file workspace/master/test/morphs/test-system.tar exists + AND file workspace/master/test/morphs/second-system.tar exists + +However, we don't need to deploy every system defined in a cluster at +once. This is useful for cases such as having a cluster morphology for +deploying a whole distbuild network, and re-deploying only nodes that +have failed. + + GIVEN the files workspace/master/test/morphs/test-system.tar and workspace/master/test/morphs/second-system.tar are removed + WHEN the user attempts to deploy test-system from cluster test-cluster.morph in branch master + +A successful deployment will have morph exit successfully, and in the +case of tarball deployments, only the tarball for the system we asked +for will be created. + + THEN morph succeeded + AND file workspace/master/test/morphs/test-system.tar exists + AND file workspace/master/test/morphs/second-system.tar does not exist + +Cluster morphs can contain "nested systems", i.e. systems which have +subsystems to deploy as part of them. + +We need to add a subsystem to the cluster to test this. + + GIVEN a subsystem in cluster test-cluster.morph in branch master called test-system.sysroot + AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master has deployment type: sysroot + +We specify the location as a file path, this is relative to the parent +system's extracted rootfs, before it is configured. + + AND subsystem test-system.sysroot in cluster test-cluster.morph in branch master has deployment location: var/lib/sysroots/test-system + +The system which contains a nested system is deployed the same as +before, we don't need to mention the nested deployment. + + AND the file workspace/master/test/morphs/test-system.tar is removed + WHEN the user attempts to deploy test-system from cluster test-cluster.morph in branch master + THEN morph succeeded + AND file workspace/master/test/morphs/test-system.tar exists + AND tarball workspace/master/test/morphs/test-system.tar contains var/lib/sysroots/test-system/baserock + +Morph will abort deployment if the system to deploy that is specified +on the command line is not defined in the morphology. + + WHEN the user attempts to deploy not-a-system from cluster test-cluster.morph in branch master + THEN morph failed + +It is not valid to deploy a nested system on its own. If it becomes +desirable to deploy a system that is identical to a system that already +exists but is nested in another, it should be redefined as a top-level +deployment. + + WHEN the user attempts to deploy test-system.sysroot from cluster test-cluster.morph in branch master + THEN morph failed + FINALLY the git server is shut down + +Deploying branch-from-image produced systems +============================================ + +We have this nifty subcommand called branch-from-image, which can be +used to build the same thing as an existing image. + +There's no special requirements for making the image reproducible. + + SCENARIO reproducing systems + GIVEN a workspace + AND a git server + WHEN the user checks out the system branch called master + AND the user builds the system systems/test-system.morph in branch master + GIVEN a cluster called test-cluster.morph in system branch master + AND a system in cluster test-cluster.morph in branch master called test-system + AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch master has deployment type: sysroot + AND system test-system in cluster test-cluster.morph in branch master has deployment location: test-system + WHEN the user attempts to deploy the cluster test-cluster.morph in branch master + THEN morph succeeded + AND file workspace/master/test/morphs/test-system exists + +To reproduce an existing image, do a checkout with the extracted root +filesystem's /baserock directory as the `--metadata-dir` argument. + + WHEN the user attempts to check out the system branch from workspace/master/test/morphs/test-system called mybranch + THEN morph succeeded + AND the system branch mybranch is checked out + +After it is checked-out, the system can be rebuilt. + + WHEN the user attempts to build the system systems/test-system.morph in branch mybranch + THEN morph succeeded + +Once it is rebuilt, it can be deployed. + + GIVEN a cluster called test-cluster.morph in system branch mybranch + AND a system in cluster test-cluster.morph in branch mybranch called test-system + AND system test-system in cluster test-cluster.morph in branch mybranch builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch mybranch has deployment type: tar + AND system test-system in cluster test-cluster.morph in branch mybranch has deployment location: test-system.tar + WHEN the user attempts to deploy the cluster test-cluster.morph in branch mybranch + THEN morph succeeded + AND file workspace/mybranch/test/morphs/test-system.tar exists diff --git a/yarns/fstab-configure.yarn b/yarns/fstab-configure.yarn new file mode 100644 index 00000000..cd7e7438 --- /dev/null +++ b/yarns/fstab-configure.yarn @@ -0,0 +1,62 @@ +`fstab.configure` +================= + +The `fstab.configure` extension appends text to the `/etc/fstab` from +environment variables beginning with `FSTAB_`. It also sets the +ownership and permissions of the file. + +The first thing to test is that the extension doesn't write anything +if not requested to do so, but does create the file if it doesn't +exist. + + SCENARIO fstab.configure does nothing by default + GIVEN a directory called tree/etc + WHEN fstab.configure is run against tree + THEN file tree/etc/fstab exists + AND file tree/etc/fstab has permissions -rw-r--r-- + AND file tree/etc/fstab is owned by uid 0 + AND file tree/etc/fstab is owned by gid 0 + AND file tree/etc/fstab is empty + +Append a something to the file, and verify the contents are exactly +correct. + + SCENARIO fstab.configure appends requested lines + GIVEN a directory called tree/etc + AND an environment variable FSTAB_FOO containing "foo" + WHEN fstab.configure is run against tree + THEN file tree/etc/fstab exists + AND file tree/etc/fstab has permissions -rw-r--r-- + AND file tree/etc/fstab is owned by uid 0 + AND file tree/etc/fstab is owned by gid 0 + AND file tree/etc/fstab contains "foo\n" + +Append something to an existing file, with wrong ownership and +permission. + + SCENARIO fstab.configure appends to existing file + GIVEN a directory called tree/etc + AND a file called tree/etc/fstab containing "# comment\n" + AND tree/etc/fstab is owned by uid 1 + AND tree/etc/fstab is owned by gid 1 + AND tree/etc/fstab has permissions 0600 + AND an environment variable FSTAB_FOO containing "foo" + WHEN fstab.configure is run against tree + THEN file tree/etc/fstab exists + AND file tree/etc/fstab has permissions -rw-r--r-- + AND file tree/etc/fstab is owned by uid 0 + AND file tree/etc/fstab is owned by gid 0 + AND file tree/etc/fstab contains "# comment\nfoo\n" + +Implement running `fstab.configure` +----------------------------------- + +When we actually run `fstab.configure`, we source `$DATADIR/env` to +get the desired environment variables. + + IMPLEMENTS WHEN fstab.configure is run against (\S+) + if [ -e "$DATADIR/env" ] + then + . "$DATADIR/env" + fi + "$SRCDIR/morphlib/exts/fstab.configure" "$DATADIR/$MATCH_1" diff --git a/yarns/implementations.yarn b/yarns/implementations.yarn new file mode 100644 index 00000000..86c3a9c4 --- /dev/null +++ b/yarns/implementations.yarn @@ -0,0 +1,971 @@ +IMPLEMENTS implementations +========================== + +Implementation sections for workspaces +-------------------------------------- + +We'll use `$DATADIR/workspace` as the workspace directory that is used. + + IMPLEMENTS GIVEN no workspace + true + + IMPLEMENTS GIVEN an empty workspace directory + mkdir "$DATADIR/workspace" + + IMPLEMENTS GIVEN a non-empty workspace directory + mkdir "$DATADIR/workspace" + touch "$DATADIR/workspace/random-file" + +We run `morph init` in two different ways: either the simple way, +letting yarn catch errors, or in a way that catches the error so +we can test it later in a THEN step. + + IMPLEMENTS WHEN the user (attempts to initialise|initialises) a workspace + set init "$DATADIR/workspace" + if [ $MATCH_1 == "initialises" ]; then run_morph "$@" + else attempt_morph "$@"; fi + + IMPLEMENTS THEN morph failed + case $(cat "$DATADIR/morph-exit") in + 0) die "Morph should have failed, but didn't. Unexpected success!" ;; + esac + + IMPLEMENTS THEN morph succeeded + case $(cat "$DATADIR/morph-exit") in + 0) echo "Morph succeeded!" + ;; + *) die "Morph should have succeeded, but didn't. Unexpected failure!" + ;; + esac + +We need to check that a workspace creation worked. This requires the +directory to exist, and its `.morph` subdirectory to exist, and nothing +else. + + IMPLEMENTS THEN an empty workspace exists + is_dir "$DATADIR/workspace" + is_dir "$DATADIR/workspace/.morph" + assert_equal $(ls -A "$DATADIR/workspace" | wc -l) 1 + +Tests for things other than `morph init` just want to have a workspace +created. + + IMPLEMENTS GIVEN a workspace + run_morph init "$DATADIR/workspace" + +Implementation sections related to a simulated Trove +---------------------------------------------------- + +Morph needs access to a Trove, i.e., a git server, in order to do certain +kinds of stuff. We simulate this by creating a set of git repositories +locally, which we'll tell Morph to access using `file:` URLs. Specifically, +we'll create a repository to hold system and stratum morphologies, and +another to hold a chunk. + + IMPLEMENTS GIVEN a git server + + # Create a directory for all the git repositories. + mkdir "$DATADIR/gits" + + + # Create the bootstrap chunk repositories + mkdir "$DATADIR/gits/bootstrap-chunk" + cd "$DATADIR/gits/bootstrap-chunk" + git init . + git checkout -b bootstrap + cp "$SRCDIR/scripts/test-shell.c" sh.c + install /dev/stdin <<'EOF' configure + #!/bin/true + EOF + printf >Makefile ' + CFLAGS = -D_GNU_SOURCE -static + + all: sh + + install: sh + \tinstall -D -m755 sh $(DESTDIR)/bin/sh' + git add . + git commit -m "Add bootstrap shell" + + git checkout --orphan master HEAD + # Commit a pre-built test-shell, as a compiler is too heavy to bootstrap + make sh + mkdir bin + mv sh bin/sh + git rm -f Makefile sh.c configure + git add bin/sh + git commit -m "Build bootstrap shell with bootstrap shell" + + # Create the test chunk repository. + + mkdir "$DATADIR/gits/test-chunk" + cd "$DATADIR/gits/test-chunk" + git init . + + # To verify that chunk splitting works, we have a chunk that installs + # dummy files in all the places that different kinds of files are + # usually installed. e.g. executables in `/bin` and `/usr/bin` + + PREFIX=/usr + DESTDIR=. + # It's important that we can test whether executables get + # installed, so we install an empty script into `/usr/bin/test` and + # `/usr/sbin/test`. + + # `install -D` will create the leading components for us, and install + # defaults to creating the file with its executable bit set. + + # `install` needs a source file to install, but since we only care + # that the file exists, rather than its contents, we can use /dev/null + # as the source. + + + for bindir in bin sbin; do + install -D /dev/null "$DESTDIR/$PREFIX/$bindir/test" + done + + # We need shared libraries too, sometimes they're libraries to support + # the executables that a chunk provides, sometimes for other chunks. + + # Libraries can be found in a variety of places, hence why we install + # them into lib, lib32 and lib64. + + # Shared libraries' file names start with lib and end with `.so` + # for shared-object, with version numbers optionally suffixed. + + for libdir in lib lib32 lib64; do + dirpath="$DESTDIR/$PREFIX/$libdir" + install -D /dev/null "$dirpath/libtest.so" + ln -s libtest.so "$dirpath/libtest.so.0" + ln -s libtest.so.0 "$dirpath/libtest.so.0.0" + ln -s libtest.so.0.0 "$dirpath/libtest.so.0.0.0" + done + + # Shared objects aren't the only kind of library, some executable + # binaries count as libraries, such as git's plumbing commands. + + # In some distributions they go into /lib, in others, and the default + # autotools configuration, they go into /libexec. + + install -D /dev/stdin "$DESTDIR/$PREFIX/libexec/test-bin" <<'EOF' + #!/bin/sh + echo Hello World + EOF + + # As well as run-time libraries, there's development files. For C + # this is headers, which describe the API of the libraries, which + # then use the shared objects, and other files which are needed + # to build the executables, but aren't needed to run them, such as + # static libraries. + + # Header files go into `include` and end with `.h`. They are not + # executable, so the install command changes the permissions with the + # `-m` option. + install -D -m 644 /dev/stdin <<'EOF' "$DESTDIR/$PREFIX/include/test.h" + int foo(void); + EOF + + # `pkg-config` is a standard way to locate libraries and get the + # compiler flags needed to build with the library. It's also used + # for other configuration for packages that don't install binaries, + # so as well as being found in `lib/pkgconfig`, it can be found in + # `share/pkgconfig`, so we install dummy files to both. + + for pkgdir in lib lib32 lib64 share; do + install -D -m 644 /dev/stdin <<EOF \ + "$DESTDIR/$PREFIX/$pkgdir/pkgconfig/test.pc" + prefix=$PREFIX + includedir=\${prefix}/include + Name: test + Cflags: -I{includedir} + EOF + done + + # Static libraries can be used to build static binaries, which don't + # require their dependencies to be installed. They are typically in + # the form of `.a` archive and `.la` libtool archives. + + for libdir in lib lib32 lib64; do + for libname in libtest.a libtest.la; do + install -D -m 644 /dev/null "$DESTDIR/$PREFIX/$libdir/$libname" + done + done + + # Packages may also install documentation, this comes in a variety + # of formats, but info pages, man pages and html documentation are + # the most common. + + for docfile in info/test.info.gz man/man3/test.3.gz doc/test/doc.html; do + install -D -m 644 /dev/null "$DESTDIR/$PREFIX/share/$docfile" + done + + # Locale covers translations, timezones, keyboard layouts etc. in + # all manner of strange file formats and locations. + + # Locale provides various translations for specific messages. + + install -D -m 644 /dev/null \ + "$DESTDIR/$PREFIX/share/locale/en_GB/LC_MESSAGES/test.mo" + + # Internationalisation (i18n) includes character maps and other data + # such as currency. + + for localefile in i18n/locales/en_GB charmaps/UTF-8.gz; do + install -D -m 644 /dev/null "$DESTDIR/$PREFIX/share/$localefile" + done + + # Timezones are another kind of localisation. + + install -D -m 644 /dev/null "$DESTDIR/$PREFIX/share/zoneinfo/UTC" + + # We also need a catch rule for everything that doesn't fit into + # the above categories, so to test that, we create some files that + # don't belong in one. + + for cfgfile in test.conf README; do + install -D -m 644 /dev/null "$DESTDIR/etc/test.d/$cfgfile" + done + + git add . + git commit --allow-empty -m Initial. + + # Create a repo for the morphologies. + + mkdir "$DATADIR/gits/morphs" + cd "$DATADIR/gits/morphs" + git init . + arch=$(run_morph print-architecture) + install -m644 -D /dev/stdin << EOF "systems/test-system.morph" + name: test-system + kind: system + arch: $arch + strata: + - name: build-essential + morph: strata/build-essential.morph + - name: core + morph: strata/core.morph + EOF + + install -m644 -D /dev/stdin << EOF "strata/build-essential.morph" + name: build-essential + kind: stratum + chunks: + - name: stage1-chunk + repo: test:bootstrap-chunk + ref: $(run_in "$DATADIR/gits/bootstrap-chunk" git rev-parse bootstrap) + unpetrify-ref: nootstrap + build-mode: bootstrap + build-depends: [] + - name: stage2-chunk + morph: stage2-chunk.morph + repo: test:bootstrap-chunk + ref: $(run_in "$DATADIR/gits/bootstrap-chunk" git rev-parse master) + unpetrify-ref: master + build-depends: + - stage1-chunk + EOF + install -m644 -D /dev/stdin << EOF "strata/core.morph" + name: core + kind: stratum + build-depends: + - morph: strata/build-essential.morph + chunks: + - name: test-chunk + morph: test-chunk.morph + repo: test:test-chunk + unpetrify-ref: master + ref: $(run_in "$DATADIR/gits/test-chunk" git rev-parse master) + build-depends: [] + EOF + + install -m644 -D /dev/stdin << 'EOF' "test-chunk.morph" + name: test-chunk + kind: chunk + build-system: manual + # `install-commands` is a list of shell commands to run. Commands + # may be on multiple lines, and indeed anything programmatic will + # benefit from doing so. Arguably we could have just one command, + # but it's split into multiple so that morph can inform us which + # command failed without us having to include a lot of status + # information in the command and look at the error message. + install-commands: + - copy files + EOF + + install -m644 -D /dev/stdin << 'EOF' "stage2-chunk.morph" + name: test-chunk + kind: chunk + build-system: manual + install-commands: + - copy files + EOF + + git add . + git commit -m Initial. + git tag -a "test-tag" -m "Tagging test-tag" + + # Start a git daemon to serve our git repositories + port_file="$DATADIR/git-daemon-port" + pid_file="$DATADIR/git-daemon-pid" + mkfifo "$port_file" + # git-daemon needs --foo=bar style arguments so we do that for consistency + start-stop-daemon --start --pidfile="$pid_file" --background \ + --make-pidfile --verbose \ + --startas="$SRCDIR/scripts/git-daemon-wrap" -- \ + --port-file="$port_file" \ + --export-all --verbose --base-path="$DATADIR/gits" \ + --enable=receive-pack #allow push + GIT_DAEMON_PORT="$(cat "$port_file")" + + # Create the Morph configuration file so we can access the repos + # using test:foo URL aliases. + + cat << EOF > "$DATADIR/morph.conf" + [config] + repo-alias = test=git://127.0.0.1:$GIT_DAEMON_PORT/%s#git://127.0.0.1:$GIT_DAEMON_PORT/%s + cachedir = $DATADIR/cache + tempdir = $DATADIR/tmp + trove-host= [] + EOF + + mkdir "$DATADIR/cache" + mkdir "$DATADIR/tmp" + +Some resources are cleaned up by yarn, forked processes aren't one of +these, so need to shut down the git daemon after we finish. + + IMPLEMENTS FINALLY the git server is shut down + pid_file="$DATADIR/git-daemon-pid" + if [ -e "$pid_file" ]; then + start-stop-daemon --stop --pidfile "$pid_file" --oknodo + fi + +We need a consistent value for the architecture in some tests, so we +have a morphology using the test architecture. + + IMPLEMENTS GIVEN a system called (\S+) for the test architecture in the git server + name="$(basename "${MATCH_1%.*}")" + cat << EOF > "$DATADIR/gits/morphs/$MATCH_1" + arch: testarch + configuration-extensions: [] + description: A system called $name for test architecture + kind: system + name: $name + strata: + - name: build-essential + morph: strata/build-essential.morph + - name: core + morph: strata/core.morph + EOF + + run_in "$DATADIR/gits/morphs" git add "strata/build-essential.morph" + run_in "$DATADIR/gits/morphs" git add "strata/core.morph" + run_in "$DATADIR/gits/morphs" git add "$MATCH_1" + run_in "$DATADIR/gits/morphs" git commit -m "Added $MATCH_1 and strata morphologies." + +You need an architecture to build a system, we don't default to the host architecture. + + IMPLEMENTS GIVEN a system called (\S+) with no architecture in the git server + name="$(basename "${MATCH_1%.*}")" + cat << EOF > "$DATADIR/gits/morphs/$MATCH_1" + configuration-extensions: [] + description: A system called $name for test architecture + kind: system + name: $name + strata: + - name: build-essential + morph: strata/build-essential.morph + - name: core + morph: strata/core.morph + EOF + + run_in "$DATADIR/gits/morphs" git add "$MATCH_1" + run_in "$DATADIR/gits/morphs" git commit -m "Added $MATCH_1." + + +Implementation sections for system branch operations +---------------------------------------------------- + +Checkout out an existing system branch. We parameterise this so the +same phrase can be used to check out any system branch. + + IMPLEMENTS WHEN the user (attempts to check|checks) out the system (branch|tag) called (\S+) + cd "$DATADIR/workspace" + set checkout test:morphs "$MATCH_3" + if [ $MATCH_1 == "checks" ]; then run_morph "$@" + else attempt_morph "$@"; fi + +Attempt to check out a system branch from a root that has no systems. + + IMPLEMENTS WHEN the user attempts to check out from a repository with no systems + cd "$DATADIR/workspace" + attempt_morph checkout test:test-chunk master + + IMPLEMENTS WHEN the user attempts to check out the system branch from (\S+) called (\S+) + cd "$DATADIR/workspace" + attempt_morph branch-from-image --metadata-dir "$DATADIR/$MATCH_1/baserock" "$MATCH_2" + +We also need to verify that a system branch has been checked out. + + IMPLEMENTS THEN the system branch (\S+) is checked out + is_dir "$DATADIR/workspace/$MATCH_1/test/morphs" + is_file "$DATADIR/workspace/$MATCH_1/test/morphs/systems/test-system.morph" + is_file "$DATADIR/workspace/$MATCH_1/test/morphs/strata/core.morph" + +We can create a new branch, off master. + + IMPLEMENTS WHEN the user (attempts to create|creates) a system branch called (\S+) + cd "$DATADIR/workspace" + set branch test:morphs "$MATCH_2" + if [ $MATCH_1 == "creates"]; then run morph "$@" + else attempt_morph "$@"; fi + +We can create a new branch, off another system branch. + + IMPLEMENTS WHEN the user creates a system branch called (\S+), based on (\S+) + cd "$DATADIR/workspace" + run_morph branch test:morphs "$MATCH_1" "$MATCH_2" + +Attempt to branch a system branch from a root that had no systems. + + IMPLEMENTS WHEN the user attempts to branch a repository with no systems + cd "$DATADIR/workspace" + attempt_morph branch test:test-chunk foo + +Pushing all changes in a system branch checkout to the git server. + + IMPLEMENTS WHEN the user pushes the system branch called (\S+) to the git server + cd "$DATADIR/workspace/$MATCH_1/" + run_morph foreach -- sh -c 'git push -u origin HEAD 2>&1' + +Report workspace path. + + IMPLEMENTS WHEN the user reports the workspace from the directory (\S+) + cd "$DATADIR/workspace/$MATCH_1" + run_morph workspace > "$DATADIR/workspace-reported" + + IMPLEMENTS THEN the workspace is reported correctly + assert_equal $(cat "$DATADIR/workspace-reported") "$DATADIR/workspace" + + IMPLEMENTS WHEN the user attempts to report the workspace from a non-workspace directory + cd "$DATADIR" + attempt_morph workspace + +Report system branch name: + + IMPLEMENTS WHEN the user (attempts to report|reports) the system branch from the directory (\S+) + cd "$DATADIR/workspace/$MATCH_2" + set $DATADIR/system-branch.reported + if [ $MATCH_1 == reports ]; then run_morph show-system-branch > "$@" + else attempt_morph show-system-branch > "$@"; fi + + IMPLEMENTS THEN the system branch is reported as (.*) + echo "$MATCH_1" > "$DATADIR/system-branch.actual" + diff -u "$DATADIR/system-branch.actual" "$DATADIR/system-branch.reported" + +Report system branch root repository. + + IMPLEMENTS WHEN the user (attempts to report|reports) the system branch root repository from the directory (.*) + cd "$DATADIR/workspace/$MATCH_2" + set $DATADIR/branch-root.reported + if [ $MATCH_1 == "reports" ]; then run_morph show-branch-root > "$@" + else attempt_morph show-branch-root > "$@"; fi + + IMPLEMENTS THEN the system branch root repository is reported as (.*) + echo "$DATADIR/$MATCH_1" > "$DATADIR/branch-root.actual" + diff -u "$DATADIR/branch-root.actual" "$DATADIR/branch-root.reported" + +Editing morphologies with `morph edit`. + + IMPLEMENTS THEN in branch (\S+), stratum (\S+) refs (\S+) in (\S+) + "$SRCDIR/scripts/yaml-extract" \ + "$DATADIR/workspace/$MATCH_1/test/morphs/$MATCH_2" \ + chunks name="$MATCH_3" ref > "$DATADIR/ref.actual" + echo "$MATCH_4" > "$DATADIR/ref.wanted" + diff -u "$DATADIR/ref.wanted" "$DATADIR/ref.actual" + + IMPLEMENTS THEN in branch (\S+), (system|stratum) (\S+) refers to (\S+) without (\S+) + if [ $MATCH_2 == system ]; then field=strata; else field=build-depends; fi + "$SRCDIR/scripts/yaml-extract" \ + "$DATADIR/workspace/$MATCH_1/test/morphs/$MATCH_3" \ + "$field" name="$MATCH_4" "$MATCH_5" 2>&1 | + grep -qFe "Object does not contain $MATCH_5" + + IMPLEMENTS WHEN the user edits the chunk (\S+) in branch (\S+) + cd "$DATADIR/workspace/$MATCH_2/test/morphs" + run_morph edit "$MATCH_1" + + IMPLEMENTS THEN the edited chunk (\S+) has git branch (\S+) + ls -l "$DATADIR/workspace/$MATCH_2" + chunkdir="$(slashify_colons "$MATCH_1")" + cd "$DATADIR/workspace/$MATCH_2/$chunkdir" + git rev-parse --abbrev-ref HEAD > "$DATADIR/git-branch.actual" + echo "$MATCH_2" > "$DATADIR/git-branch.wanted" + diff -u "$DATADIR/git-branch.wanted" "$DATADIR/git-branch.actual" + +To produce buildable morphologies, we need them to be of the same +architecture as the machine doing the testing. This uses `morph +print-architecture` to get a value appropriate for morph. + + IMPLEMENTS WHEN the user creates an uncommitted system morphology called (\S+) for our architecture in system branch (\S+) + arch=$(run_morph print-architecture) + name="$(basename "${MATCH_1%.*}")" + install -m644 -D /dev/stdin << EOF "$DATADIR/workspace/$MATCH_2/test/morphs/$MATCH_1" + arch: $arch + configuration-extensions: [] + description: A system called $name for architectures $arch + kind: system + name: $name + strata: + - name: build-essential + morph: strata/build-essential.morph + - name: core + morph: strata/core.morph + EOF + +Reporting status of checked out repositories: + + IMPLEMENTS THEN morph reports no outstanding changes in (\S+) + cd "$DATADIR/workspace/$MATCH_1" + run_morph status > "$DATADIR/morph.stdout" + grep '^No repos have outstanding changes.' "$DATADIR/morph.stdout" + + IMPLEMENTS THEN morph reports changes in (\S+) in (\S+) only + cd "$DATADIR/workspace/$MATCH_1" + run_morph status > "$DATADIR/morph.stdout" + + # morph status is expected to produce records like this: + # On branch GITBRANCH, root baserock:baserock/morphs + # GITREPO: uncommitted changes + # We check thet GITREPO matches $MATCH_2. + + awk '/: uncommitted changes$/ { print substr($1,1,length($1)-1) }' \ + "$DATADIR/morph.stdout" > "$DATADIR/changed.actual" + echo "$MATCH_2" > "$DATADIR/changed.wanted" + diff -u "$DATADIR/changed.wanted" "$DATADIR/changed.actual" + + IMPLEMENTS THEN morph reports changes in (\S+) in (\S+) and (\S+) + cd "$DATADIR/workspace/$MATCH_1" + run_morph status > "$DATADIR/morph.stdout" + echo "status morph.stdout:" + cat "$DATADIR/morph.stdout" + awk '/: uncommitted changes$/ { print substr($1,1,length($1)-1) }' \ + "$DATADIR/morph.stdout" | sort > "$DATADIR/changed.actual" + (echo "$MATCH_2"; echo "$MATCH_3") | sort > "$DATADIR/changed.wanted" + diff -u "$DATADIR/changed.wanted" "$DATADIR/changed.actual" + + IMPLEMENTS WHEN creating file (\S+) in (\S+) in branch (\S+) + touch "$DATADIR/workspace/$MATCH_3/$MATCH_2/$MATCH_1" + + IMPLEMENTS WHEN adding file (\S+) in (\S+) in branch (\S+) to git + chunkdir="$(slashify_colons "$MATCH_2")" + cd "$DATADIR/workspace/$MATCH_3/$chunkdir" + git add "$MATCH_1" + + IMPLEMENTS WHEN committing changes in (\S+) in branch (\S+) + cd "$DATADIR/workspace/$MATCH_2/$(slashify_colons "$MATCH_1")" + git commit -a -m test-commit + +Running shell command in each checked out repository: + + IMPLEMENTS WHEN running shell command in each repo in (\S+) + cd "$DATADIR/workspace/$MATCH_1" + run_morph foreach -- pwd > "$DATADIR/morph.stdout" + + IMPLEMENTS THEN morph ran command in (\S+) in (\S+) + grep -Fx "$MATCH_1" "$DATADIR/morph.stdout" + grep -Fx "$DATADIR/workspace/$MATCH_2/$MATCH_1" "$DATADIR/morph.stdout" + +Generating a manifest. + + IMPLEMENTS GIVEN a system artifact + mkdir "$DATADIR/hello_world" + + git init "$DATADIR/hello_world" + touch "$DATADIR/hello_world/configure.ac" + run_in "$DATADIR/hello_world" git add configure.ac + run_in "$DATADIR/hello_world" git commit -m 'Add configure.ac' + + mkdir "$DATADIR/baserock" + run_in "$DATADIR/hello_world" cat << EOF \ + > "$DATADIR/baserock/hello_world.meta" + { + "artifact-name": "hello_world", + "cache-key": + "ab8d00a80298a842446ce23507cea6b4d0e34c7ddfa05c67f460318b04d21308", + "kind": "chunk", + "morphology": "hello_world.morph", + "original_ref": "$(run_in "$DATADIR/hello_world" git rev-parse HEAD)", + "repo": "file://$DATADIR/hello_world", + "repo-alias": "upstream:hello_world", + "sha1": "$(run_in "$DATADIR/hello_world" git rev-parse HEAD)", + "source-name": "hello_world" + } + EOF + run_in "$DATADIR" tar -c baserock > "$DATADIR/artifact.tar" + + IMPLEMENTS WHEN morph generates a manifest + run_morph generate-manifest "$DATADIR/artifact.tar" > "$DATADIR/manifest" + + IMPLEMENTS THEN the manifest is generated + + # Generated manifest should contain the name of the repository + if ! grep -q hello_world "$DATADIR/manifest"; then + die "Output isn't what we expect" + fi + +Implementations for temporary build branch handling +--------------------------------------------------- + + IMPLEMENTS GIVEN the workspace contains no temporary build branches + build_ref_prefix=baserock/builds/ + cd "$DATADIR/workspace" + # Want to use -execdir here, but busybox find doesn't support it + find . -name .git -print | while read gitdir; do ( + cd "$(dirname "$gitdir")" + eval "$(git for-each-ref --shell \ + --format='git update-ref -d %(refname) %(objectname)' \ + "refs/heads/$build_ref_prefix")" + ); done + + IMPLEMENTS GIVEN the git server contains no temporary build branches + build_ref_prefix=refs/heads/baserock/builds/ + cd "$DATADIR/gits" + # Want to use -execdir here, but busybox find doesn't support it + find . -name .git -print | while read gitdir; do ( + cd "$(dirname "$gitdir")" + eval "$(git for-each-ref --shell \ + --format='git update-ref -d %(refname) %(objectname)' \ + "$build_ref_prefix")" + git config receive.denyCurrentBranch ignore + rm -f .git/morph-pushed-branches + mkdir -p .git/hooks + cat >.git/hooks/post-receive <<'EOF' + #!/bin/sh + touch "$GIT_DIR/hook-ever-run" + exec cat >>"$GIT_DIR/morph-pushed-branches" + EOF + chmod +x .git/hooks/post-receive + ); done + + IMPLEMENTS GIVEN we can build with local branches + sed -i -e '/push-build-branches/d' "$DATADIR/morph.conf" + + IMPLEMENTS GIVEN we must build from pushed branches + cat >>"$DATADIR/morph.conf" <<'EOF' + push-build-branches = True + EOF + + IMPLEMENTS THEN the (\S+) repository in the workspace for (\S+) has temporary build branches + build_ref_prefix=refs/heads/baserock/builds/ + cd "$DATADIR/workspace/$MATCH_2/$(slashify_colons "test:$MATCH_1")" + git for-each-ref | grep -F "$build_ref_prefix" + + IMPLEMENTS THEN the (\S+) repository in the workspace for (\S+) has no temporary build branches + build_ref_prefix=refs/heads/baserock/builds/ + cd "$DATADIR/workspace/$MATCH_2/$(slashify_colons "test:$MATCH_1")" + if git for-each-ref | grep -F "$build_ref_prefix"; then + die Did not expect repo to contain build branches + fi + + IMPLEMENTS THEN no temporary build branches were pushed to the (\S+) repository + build_ref_prefix=refs/heads/baserock/builds/ + cd "$DATADIR/gits/$MATCH_1/.git" + if test -e morph-pushed-branches && grep -F "$build_ref_prefix" morph-pushed-branches; then + die Did not expect any pushed build branches + fi + + IMPLEMENTS THEN temporary build branches were pushed to the (\S+) repository + build_ref_prefix=refs/heads/baserock/builds/ + cd "$DATADIR/gits/$MATCH_1/.git" + test -e morph-pushed-branches && grep -F "$build_ref_prefix" morph-pushed-branches + +Implementation sections for building +==================================== + + IMPLEMENTS WHEN the user (attempts to build|builds) the system (\S+) in branch (\S+) + cd "$DATADIR/workspace/$MATCH_3" + set build "$MATCH_2" + if [ $MATCH_1 == "builds" ]; then run_morph "$@" + else attempt_morph "$@"; fi + +Implementation sections for cross-bootstraping +============================================== + + IMPLEMENTS THEN the user cross-bootstraps the system (\S+) in branch (\S+) of repo (\S+) to the arch (\S+) + cd "$DATADIR/workspace/$MATCH_2" + set -- cross-bootstrap "$MATCH_4" "$MATCH_3" "$MATCH_2" "$MATCH_1" + run_morph "$@" + +Implementation sections for deployment +====================================== + +Defaults are set in the cluster morphology, so we can deploy without +setting any extra parameters, but we also need to be able to override +them, so they can be added to the end of the implements section. + + IMPLEMENTS WHEN the user (attempts to deploy|deploys) the (system|cluster) (\S+) in branch (\S+)( with options (.*))? + cd "$DATADIR/workspace/$MATCH_4" + set -- deploy "$MATCH_3" + if [ "$MATCH_5" != '' ]; then + # eval used so word splitting in the text is preserved + eval set -- '"$@"' $MATCH_6 + fi + if [ $MATCH_1 == "deploys" ]; then run_morph "$@" + else attempt_morph "$@"; fi + + IMPLEMENTS WHEN the user (attempts to deploy|deploys) (.*) from cluster (\S+) in branch (\S+) + cd "$DATADIR/workspace/$MATCH_4" + set -- deploy "$MATCH_3" + systems=$(echo "$MATCH_2" | sed -e 's/, /\n/g' -e 's/ and /\n/g') + set -- "$@" $systems + if [ $MATCH_1 == "deploys" ]; then run_morph "$@" + else attempt_morph "$@"; fi + + IMPLEMENTS WHEN the user (attempts to upgrade|upgrades) the (system|cluster) (\S+) in branch (\S+)( with options (.*))? + cd "$DATADIR/workspace/$MATCH_4" + set -- upgrade "$MATCH_3" + if [ "$MATCH_5" != '' ]; then + # eval used so word splitting in the text is preserved + eval set -- '"$@"' $MATCH_6 + fi + if [ $MATCH_1 == "upgrades" ]; then run_morph "$@" + else attempt_morph "$@"; fi + +Implementations sections for reading error messages +=================================================== + + IMPLEMENTS THEN the (branch|build|checkout|deploy|edit|init) error message includes the string "(.*)" + grep "$MATCH_2" "$DATADIR/result-$MATCH_1" + +IMPLEMENTS for test file and directory handling +=============================================== + +The IMPLEMENTS sections in this chapter create files and directories +for use as test data, and set and test their contents and permissions +and ownerships. + +Create a directory +------------------ + + IMPLEMENTS GIVEN a directory called (\S+) + mkdir -p "$DATADIR/$MATCH_1" + +Create a file +------------- + +The file contents is used as a `printf`(1) format string. + + IMPLEMENTS GIVEN a file called (\S+) containing "(.*)" + printf "$MATCH_2" > "$DATADIR/$MATCH_1" + +Remove a file +------------- + + IMPLEMENTS GIVEN the file(s)? (.*) (is|are) removed + cd "$DATADIR" + files=$(echo "$MATCH_2" | sed -e 's/, /\n/g' -e 's/ and /\n/g') + rm $files + +Set attributes on a file or directory +------------------------------------- + + IMPLEMENTS GIVEN (\S+) is owned by uid (\S+) + chown "$MATCH_2" "$DATADIR/$MATCH_1" + + IMPLEMENTS GIVEN (\S+) is owned by gid (\S+) + chgrp "$MATCH_2" "$DATADIR/$MATCH_1" + + IMPLEMENTS GIVEN (\S+) has permissions (\S+) + chmod "$MATCH_2" "$DATADIR/$MATCH_1" + +Check attributes of a file on the filesystem +-------------------------------------------- + + IMPLEMENTS THEN file (\S+) exists + test -e "$DATADIR/$MATCH_1" + + IMPLEMENTS THEN file (\S+) does not exist + test ! -e "$DATADIR/$MATCH_1" + + IMPLEMENTS THEN file (\S+) has permissions (\S+) + stat -c %A "$DATADIR/$MATCH_1" | grep -Fx -e "$MATCH_2" + + IMPLEMENTS THEN file (\S+) is owned by uid (\d+) + stat -c %u "$DATADIR/$MATCH_1" | grep -Fx -e "$MATCH_2" + + IMPLEMENTS THEN file (\S+) is owned by gid (\d+) + stat -c %g "$DATADIR/$MATCH_1" | grep -Fx -e "$MATCH_2" + + IMPLEMENTS THEN file (\S+) is empty + stat -c %s "$DATADIR/$MATCH_1" | grep -Fx 0 + + IMPLEMENTS THEN file (\S+) matches (.*) + grep -q "$MATCH_2" "$DATADIR/$MATCH_1" + +Disk image manipulation +----------------------- + +We need to test disk images we create. In the absence of tools for +inspecting disks without mounting them, we need commands to handle this. + + IMPLEMENTS WHEN disk image (\S+) is mounted at (.*) + mkdir -p "$DATADIR/$MATCH_2" + mount -o loop "$DATADIR/$MATCH_1" "$DATADIR/$MATCH_2" + + IMPLEMENTS FINALLY (\S+) is unmounted + umount -d "$DATADIR/$MATCH_1" + +We may not have enough space to run some tests that have disk images. + + IMPLEMENTS ASSUMING there is space for (\d+) (\d+)(\S*) disk images? + # Count is included as an argument, so that if we change the disk + # image sizes then it's more obvious when we need to change the + # assumption, since it's the same value. + count="$MATCH_1" + case "$MATCH_3" in + '') + size="$MATCH_2" + ;; + M) + size=$(expr "$MATCH_2" '*' 1024 '*' 1024 ) + ;; + G) + size=$(expr "$MATCH_2" '*' 1024 '*' 1024 '*' 1024 ) + ;; + *) + echo Unrecognized size suffix: "$MATCH_3" >&2 + exit 1 + esac + total_image_size="$(expr "$size" '*' "$count" )" + blocks="$(stat -f -c %a "$DATADIR")" + block_size="$(stat -f -c %S "$DATADIR")" + disk_free=$(expr "$blocks" '*' "$block_size" ) + test "$disk_free" -gt "$total_image_size" + +Check contents of a file +------------------------ + +We treat the contents of the file in the step as a `printf`(1) format +string, to allow newlines and other such stuff to be expressed. + + IMPLEMENTS THEN file (\S+) contains "(.*)" + printf "$MATCH_2" | diff - "$DATADIR/$MATCH_1" + + +IMPLEMENTS for running programs +=============================== + +This chapter contains IMPLEMENTS sections for running programs. It is +currently a bit of a placeholder. + +Remember environment variables to set when running +-------------------------------------------------- + +We need to manage the environment. We store the extra environment +variables in `$DATADIR/env`. We treat the value as a format string for +`printf`(1) so that newlines etc can be used. + + IMPLEMENTS GIVEN an environment variable (\S+) containing "(.*)" + printf "export $MATCH_1=$MATCH_2" >> "$DATADIR/env" + +Implementations for building systems +------------------------------------ + + IMPLEMENTS THEN morph build the system (\S+) of the (branch|tag) (\S+) + cd "$DATADIR/workspace/$MATCH_3" + run_morph build "$MATCH_1" + + IMPLEMENTS WHEN the user builds (\S+) of the (\S+) (branch|tag) + cd "$DATADIR/workspace/$MATCH_2" + run_morph build "$MATCH_1" + +Implementations for tarball inspection +-------------------------------------- + + IMPLEMENTS THEN tarball (\S+) contains (.*) + tar -tf "$DATADIR/$MATCH_1" | grep -Fe "$MATCH_2" + + IMPLEMENTS THEN tarball (\S+) doesn't contain (.*) + ! tar -tf "$DATADIR/$MATCH_1" | grep -Fe "$MATCH_2" + +Implementations for morphology manipulation +========================================== + +Altering morphologies in their source repositories +-------------------------------------------------- + + IMPLEMENTS GIVEN system (\S+) uses (.+) from (\S+) + "$SRCDIR/scripts/edit-morph" set-system-artifact-depends \ + "$DATADIR/gits/morphs/$MATCH_1" "$MATCH_3" "$MATCH_2" + run_in "$DATADIR/gits/morphs" git add "$MATCH_1" + run_in "$DATADIR/gits/morphs" git commit -m "Make $MATCH_1 only use $MATCH_2" + + IMPLEMENTS GIVEN stratum (\S+) has match rules: (.*) + cd "$DATADIR/gits/morphs" + "$SRCDIR/scripts/edit-morph" set-stratum-match-rules \ + "$MATCH_1" "$MATCH_2" + git add "$MATCH_1" + git commit -m "Make $MATCH_1 match $MATCH_2" + +Altering morphologies in the workspace +-------------------------------------- + +### Altering strata ### + + IMPLEMENTS GIVEN stratum (\S+) in system branch (\S+) has match rules: (.*) + cd "$DATADIR/workspace/$MATCH_2/test/morphs" + "$SRCDIR/scripts/edit-morph" set-stratum-match-rules \ + "$MATCH_1" "$MATCH_3" + +### Altering clusters ### + + IMPLEMENTS GIVEN a cluster called (\S+) in system branch (\S+) + name="$MATCH_1" + branch="$MATCH_2" + "$SRCDIR/scripts/edit-morph" cluster-init \ + "$DATADIR/workspace/$branch/test/morphs/$name" + + IMPLEMENTS GIVEN a (sub)?system in cluster (\S+) in branch (\S+) called (\S+) + cluster="$MATCH_2" + branch="$MATCH_3" + name="$MATCH_4" + "$SRCDIR/scripts/edit-morph" cluster-system-init \ + "$DATADIR/workspace/$branch/test/morphs/$cluster" "$name" + + IMPLEMENTS GIVEN (sub)?system (\S+) in cluster (\S+) in branch (\S+) builds (\S+) + name="$MATCH_2" + cluster="$MATCH_3" + branch="$MATCH_4" + morphology="$MATCH_5" + "$SRCDIR/scripts/edit-morph" cluster-system-set-morphology \ + "$DATADIR/workspace/$branch/test/morphs/$cluster" "$name" \ + "$morphology" + + IMPLEMENTS GIVEN (sub)?system (\S+) in cluster (\S+) in branch (\S+) has deployment type: (\S+) + name="$MATCH_2" + cluster="$MATCH_3" + branch="$MATCH_4" + type="$MATCH_5" + "$SRCDIR/scripts/edit-morph" cluster-system-set-deploy-type \ + "$DATADIR/workspace/$branch/test/morphs/$cluster" "$name" \ + "$type" + + IMPLEMENTS GIVEN (sub)?system (\S+) in cluster (\S+) in branch (\S+) has deployment location: (\S+) + name="$MATCH_2" + cluster="$MATCH_3" + branch="$MATCH_4" + location="$MATCH_5" + "$SRCDIR/scripts/edit-morph" cluster-system-set-deploy-location \ + "$DATADIR/workspace/$branch/test/morphs/$cluster" "$name" \ + "$location" + + IMPLEMENTS GIVEN (sub)?system (\S+) in cluster (\S+) in branch (\S+) has deployment variable: ([^=]+)=(.*) + name="$MATCH_2" + cluster="$MATCH_3" + branch="$MATCH_4" + key="$MATCH_5" + val="$MATCH_6" + "$SRCDIR/scripts/edit-morph" cluster-system-set-deploy-variable \ + "$DATADIR/workspace/$branch/test/morphs/$cluster" "$name" \ + "$key" "$val" diff --git a/yarns/morph.shell-lib b/yarns/morph.shell-lib new file mode 100644 index 00000000..9d67f2ab --- /dev/null +++ b/yarns/morph.shell-lib @@ -0,0 +1,186 @@ +# Shell library for Morph yarns. +# +# The shell functions in this library are meant to make writing IMPLEMENTS +# sections for yarn scenario tests easier. + +# Copyright (C) 2013-2014 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +# Add $SRCDIR to PYTHONPATH. + +case "$PYTHONPATH" in + '') PYTHONPATH="$SRCDIR)" ;; + *) PYTHONPATH="$SRCDIR:$PYTHONPATH" ;; +esac +export PYTHONPATH + + +# Run Morph from the source tree, ignoring any configuration files. +# This way the test suite is not affected by any configuration the user +# or system may have. Instead, we'll use the `$DATADIR/morph.conf` file, +# which tests can create, if they want to. + +run_morph() +{ + { + set +e + "$SRCDIR"/morph --verbose \ + --cachedir-min-space=0 --tempdir-min-space=0 \ + --no-default-config --config "$DATADIR/morph.conf" "$@" \ + 2> "$DATADIR/result-$1" > "$DATADIR/out-$1" + local exit_code="$?" + cat "$DATADIR/out-$1" + cat "$DATADIR/result-$1" >&2 + return "$exit_code" + } +} + + +# Sometimes we want to try running morph, but are OK if it fails, we just +# need to remember that it did. + +attempt_morph() +{ + if run_morph "$@" + then + echo 0 > "$DATADIR/morph-exit" + else + echo "$?" > "$DATADIR/morph-exit" + fi +} + + +# Perl's die() function is often very useful: it prints an error message +# and terminates the process with a non-zero exit code. Let's have a +# shell function to do that. + +die() +{ + echo "ERROR: $@" 1>&2 + exit 1 +} + + +# Tests often need to check that specific files or directories exist +# and have the right ownerships etc. Here's some shell functions to +# test that kind of thing. + +is_dir() +{ + if [ ! -d "$1" ] + then + die "Expected $1 to be a directory" + fi +} + +is_file() +{ + if [ ! -f "$1" ] + then + die "Expected $1 to be a regular file" + fi +} + + +# General assertions. + +assert_equal() +{ + if [ "$1" != "$2" ] + then + die "Expected '$1' and '$2' to be equal" + fi +} + + +# Sometimes it's nice to run a command in a different directory, without +# having to bother changing the directory before and after the command, +# or spawning subshells. This function helps with that. + +run_in() +{ + (cd "$1" && shift && exec "$@") +} + + +# Extract all refs in all given morphologies. Each ref is reported +# as filename:ref. The referred-to repository is not listed. + +list_refs() +{ + awk '/ ref: / { printf "%s %s\n", FILENAME, $NF }' "$@" +} + + +# Is a ref petrified? Or a specific branch? + +is_petrified_or_branch() +{ + if echo "$1" | + awk -v "branch=$2" '$NF ~ /[0-9a-fA-F]{40}/ || $NF == branch' | + grep . + then + return 0 + else + return 1 + fi +} + + +# Are named morphologies petrified? Die if not. First arg is the +# branch that is allowed in addition to SHA1s. + +assert_morphologies_are_petrified() +{ + local branch="$1" + shift + list_refs "$@" | + while read filename ref + do + if ! is_petrified_or_branch "$ref" "$branch" + then + die "Found non-SHA1 ref in $filename: $ref" + fi + done +} + + +# Added until it's fixed in upstream. +# It's a solution to create an empty home directory each execution +export HOME="$DATADIR/home" +if [ ! -d "$HOME" ] +then + mkdir "$HOME" +fi + +# Generating a default git user to run the tests +if ! test -r "$HOME/.gitconfig" +then + cat > "$HOME/.gitconfig" <<EOF +[user] + name = Tomjon Codethinker + email = tomjon@codethink.co.uk +EOF +fi + + +# Change colons to slashes. This is used when converting an aliases +# repository URL (e.g., test:morphs) into a directory path. + +slashify_colons() +{ + echo "$1" | sed s,:,/,g +} diff --git a/yarns/print-architecture.yarn b/yarns/print-architecture.yarn new file mode 100644 index 00000000..c2496147 --- /dev/null +++ b/yarns/print-architecture.yarn @@ -0,0 +1,43 @@ +"morph print-architecture" tests +================================ + +This is short and simple. Morph can print the name for the current +architecture, and we verify not that it is correct, but that exactly +one line is printed to the standard output. The reason we're not +checking it's correct is because that would require the test code +to duplicate the architecture name list that is in the code already, +and that wouldn't help with tests. However, verifying there's exactly +one line in stdout (and nothing in stderr) means the plugin does at +least something sensible. + +Oh, and the one line should contain no spaces, either. + + SCENARIO morph print-architecture prints out a single word + WHEN morph print-architecture is run + THEN stdout contains a single line + AND stdout contains no spaces + AND stderr is empty + + IMPLEMENTS WHEN morph print-architecture is run + set +x + run_morph print-architecture > "$DATADIR/stdout" 2> "$DATADIR/stderr" + + IMPLEMENTS THEN stdout contains a single line + n=$(wc -l < "$DATADIR/stdout") + if [ "$n" != 1 ] + then + die "stdout contains $n lines, not 1" + fi + + IMPLEMENTS THEN stdout contains no spaces + n=$(tr < "$DATADIR/stdout" -cd ' ' | wc -c) + if [ "$n" != 0 ] + then + die "stdout contains spaces" + fi + + IMPLEMENTS THEN stderr is empty + if [ -s "$DATADIR/stderr" ] + then + die "stderr is not empty" + fi diff --git a/yarns/regression.yarn b/yarns/regression.yarn new file mode 100644 index 00000000..c424f437 --- /dev/null +++ b/yarns/regression.yarn @@ -0,0 +1,107 @@ +"regression" tests +================== + +Tests for check we don't introduce some bugs again. + + +Testing if we can build after checking out from a tag. + + SCENARIO morph build works after checkout from a tag + GIVEN a workspace + AND a git server + WHEN the user checks out the system tag called test-tag + THEN morph build the system systems/test-system.morph of the tag test-tag + FINALLY the git server is shut down + + +Running `morph branch` when the branch directory exists doesn't +remove the existing directory. + + SCENARIO re-running 'morph branch' fails, original branch untouched + GIVEN a workspace + AND a git server + WHEN the user creates a system branch called foo + THEN the system branch foo is checked out + +The branch is checked out correctly, now it should fail if the user executes +`morph branch` with the same branch name. + + WHEN the user attempts to create a system branch called foo + THEN morph failed + AND the branch error message includes the string "File exists" + +The branch still checked out. + + AND the system branch foo is checked out + FINALLY the git server is shut down + + +It doesn't make much sense to be able to build a system with only +bootstrap chunks, since they will have been constructed without a staging +area, hence their results cannot be trusted. + + SCENARIO building a system with only bootstrap chunks fails + GIVEN a workspace + AND a git server + AND a system containing only bootstrap chunks called bootstrap-system.morph + WHEN the user checks out the system branch called master + AND the user attempts to build the system bootstrap-system.morph in branch master + THEN the build error message includes the string "No non-bootstrap chunks found" + FINALLY the git server is shut down + +When we started allowing multiple artifacts, a long-standing bug in +cache-key computation was discovered, it didn't include artifact names, +which would cause a collision if a morphology changed which artifacts +from a source it depended on, but not the number of artifacts from that +source it depended on. + + SCENARIO changing the artifacts a system uses + GIVEN a workspace + AND a git server + AND system systems/test-system.morph uses core-runtime from core + AND stratum strata/core.morph has match rules: [{artifact: core-runtime, include: [.*-(bins|libs|locale)]}, {artifact: core-devel, include: [.*-(devel|doc|misc)]}] + WHEN the user checks out the system branch called master + GIVEN a cluster called test-cluster.morph in system branch master + AND a system in cluster test-cluster.morph in branch master called test-system + AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch master has deployment type: tar + WHEN the user builds the system systems/test-system.morph in branch master + GIVEN stratum strata/core.morph in system branch master has match rules: [{artifact: core-runtime, include: [.*-(bins|libs|misc)]}, {artifact: core-devel, include: [.*-(devel|doc|locale)]}] + WHEN the user builds the system systems/test-system.morph in branch master + AND the user deploys the cluster test-cluster.morph in branch master with options test-system.location="$DATADIR/test.tar" + THEN tarball test.tar contains baserock/test-chunk-misc.meta + FINALLY the git server is shut down + + +Implementations +--------------- + + IMPLEMENTS GIVEN a system containing only bootstrap chunks called (\S+) + arch=$(run_morph print-architecture) + name="$(basename "${MATCH_1%.*}")" + install -m644 -D /dev/stdin <<EOF "$DATADIR/gits/morphs/$MATCH_1" + name: $name + kind: system + arch: $arch + strata: + - morph: strata/bootstrap-stratum.morph + EOF + + install -m644 -D /dev/stdin << EOF "$DATADIR/gits/morphs/strata/bootstrap-stratum.morph" + name: bootstrap-stratum + kind: stratum + chunks: + - name: bootstrap-chunk + morph: bootstrap-chunk.morph + repo: test:test-chunk + unpetrify-ref: master + ref: $(run_in "$DATADIR/gits/test-chunk" git rev-parse master) + build-mode: bootstrap + build-depends: [] + EOF + sed -e 's/name: test-chunk/name: bootstrap-chunk/g' \ + "$DATADIR/gits/morphs/test-chunk.morph" \ + > "$DATADIR/gits/morphs/bootstrap-chunk.morph" + + run_in "$DATADIR/gits/morphs" git add . + run_in "$DATADIR/gits/morphs" git commit -m "Add bootstrap-system" diff --git a/yarns/splitting.yarn b/yarns/splitting.yarn new file mode 100644 index 00000000..2726d294 --- /dev/null +++ b/yarns/splitting.yarn @@ -0,0 +1,211 @@ +Artifact splitting tests +======================== + +Parsing and validation +---------------------- + +To verify that the products fields are parsed correctly, we have a +scenario that uses all of them, not relying on the default rules. + + SCENARIO building a system with morphologies that have splitting rules + GIVEN a workspace + AND a git server + +To test that all the fields are recognised, we set the new fields to +their default values. + + AND chunk test-chunk includes the default splitting rules + AND stratum strata/core.morph includes the default splitting rules + AND system systems/test-system.morph includes the default splitting rules + +The default rules produce a system that is identical to not providing +them, and since this test is about validation, we don't care about the +result, so much as it succeeding to build something. + + WHEN the user checks out the system branch called master + THEN morph build the system systems/test-system.morph of the branch master + FINALLY the git server is shut down + +Smaller systems +--------------- + +An example use-case for splitting is to only include the runtime +strata for a target system, rather than including all the development +information, such as the documentation, C library headers and C static +libraries. + + SCENARIO building a system only using runtime strata + GIVEN a workspace + AND a git server + +The only change we need to make is to add a field to the system morphology +to select which artifact to use in the system. + + AND system systems/test-system.morph uses core-runtime from core + WHEN the user checks out the system branch called master + +The best way to test that only using some stratum artifacts works is +to check which files the output has, so we deploy a tarball and inspect +its contents. + + GIVEN a cluster called test-cluster.morph in system branch master + AND a system in cluster test-cluster.morph in branch master called test-system + AND system test-system in cluster test-cluster.morph in branch master builds systems/test-system.morph + AND system test-system in cluster test-cluster.morph in branch master has deployment type: tar + WHEN the user builds the system systems/test-system.morph in branch master + AND the user attempts to deploy the cluster test-cluster.morph in branch master with options test-system.location="$DATADIR/test.tar" + +The -runtime artifacts include executables and shared libraries. + + THEN morph succeeded + AND tarball test.tar contains bin/test + AND tarball test.tar contains lib/libtest.so + +The -devel artifacts include static libraries and documentation, so if +we've successfully excluded it, we won't have those files. + + AND tarball test.tar doesn't contain lib/libtest.a + AND tarball test.tar doesn't contain man/man3/test.3.gz + +As a consequence of how dependencies are generated, if we select strata +to go into our system, such that there are chunk artifacts that are not +needed, then they don't get built. + + SCENARIO building a system that has unused chunks + GIVEN a workspace + AND a git server + +This GIVEN has a chunk in the stratum that never successfully builds, +so we know that if the system successfully builds, then we only built +chunks that were needed. + + AND stratum strata/core.morph has chunks that aren't used in core-minimal + AND system systems/test-system.morph uses core-minimal from core + WHEN the user checks out the system branch called master + THEN morph build the system systems/test-system.morph of the branch master + FINALLY the git server is shut down + + +Implementations +--------------- + + IMPLEMENTS GIVEN chunk (\S+) includes the default splitting rules + # Append default products rules + name="$(basename "${MATCH_1%.*}")" + cat <<EOF >>"$DATADIR/gits/morphs/$MATCH_1.morph" + products: + - artifact: $name-bins + include: [ "(usr/)?s?bin/.*" ] + - artifact: $name-libs + include: + - (usr/)?lib(32|64)?/lib[^/]*\.so(\.\d+)* + - (usr/)?libexec/.* + - artifact: $name-devel + include: + - (usr/)?include/.* + - (usr/)?lib(32|64)?/lib.*\.a + - (usr/)?lib(32|64)?/lib.*\.la + - (usr/)?(lib(32|64)?|share)/pkgconfig/.*\.pc + - artifact: $name-doc + include: + - (usr/)?share/doc/.* + - (usr/)?share/man/.* + - (usr/)?share/info/.* + - artifact: $name-locale + include: + - (usr/)?share/locale/.* + - (usr/)?share/i18n/.* + - (usr/)?share/zoneinfo/.* + - artifact: $name-misc + include: [ .* ] + EOF + run_in "$DATADIR/gits/morphs" git add "$MATCH_1.morph" + run_in "$DATADIR/gits/morphs" git commit -m 'Add default splitting rules' + + IMPLEMENTS GIVEN stratum (\S+) includes the default splitting rules + name=$(basename "${MATCH_1%.*}") + cat <<EOF >"$DATADIR/gits/morphs/$MATCH_1" + name: $name + kind: stratum + build-depends: + - morph: strata/build-essential.morph + products: + - artifact: $name-devel + include: + - .*-devel + - .*-debug + - .*-doc + - artifact: $name-runtime + include: + - .*-bins + - .*-libs + - .*-locale + - .*-misc + - .* + chunks: + - name: test-chunk + repo: test:test-chunk + unpetrify-ref: master + ref: $(run_in "$DATADIR/gits/test-chunk" git rev-parse master) + morph: test-chunk.morph + build-depends: [] + artifacts: + test-chunk-bins: $name-runtime + test-chunk-libs: $name-runtime + test-chunk-locale: $name-runtime + test-chunk-misc: $name-runtime + test-chunk-devel: $name-devel + test-chunk-doc: $name-devel + EOF + run_in "$DATADIR/gits/morphs" git add "$MATCH_1" + run_in "$DATADIR/gits/morphs" git commit -m 'Add default splitting rules' + + IMPLEMENTS GIVEN system (\S+) includes the default splitting rules + cat << EOF >> "$DATADIR/gits/morphs/$MATCH_1" + strata: + - name: build-essential + morph: strata/build-essential.morph + - name: core + morph: strata/core.morph + artifacts: + - core-runtime + - core-devel + EOF + run_in "$DATADIR/gits/morphs" git add "$MATCH_1" + run_in "$DATADIR/gits/morphs" git commit -m 'Add default splitting rules' + + IMPLEMENTS GIVEN stratum (\S+) has chunks that aren't used in (\S+) + # Create an extra chunk that will never successfully build + cat >"$DATADIR/gits/morphs/unbuildable-chunk.morph" <<EOF + name: unbuildable-chunk + kind: chunk + install-commands: + - "false" + EOF + run_in "$DATADIR/gits/morphs" git add unbuildable-chunk.morph + run_in "$DATADIR/gits/morphs" git commit -m 'Add unbuildable chunk' + + # Create a stratum that has an artifact that doesn't include any + # artifacts from unbuildable-chunk + cat >>"$DATADIR/gits/morphs/$MATCH_1" <<EOF + products: + - artifact: $MATCH_2 + include: + - test-chunk-.* + chunks: + - name: test-chunk + repo: test:test-chunk + morph: test-chunk.morph + unpetrify-ref: master + ref: $(run_in "$DATADIR/gits/test-chunk" git rev-parse master) + build-depends: [] + - name: unbuildable-chunk + repo: test:test-chunk + unpetrify-ref: refs/heads/master + ref: $(run_in "$DATADIR/gits/test-chunk" git rev-parse master) + morph: unbuildable-chunk.morph + build-depends: + - test-chunk + EOF + run_in "$DATADIR/gits/morphs" git add "$MATCH_1" + run_in "$DATADIR/gits/morphs" git commit -m "add $MATCH_2 to stratum" |