Add tool to run cross-project tests

Provide a script for the devstack-gate hook to run tests to
ensure that changes in this project do not break unit tests
in consuming projects.

See https://review.openstack.org/#/c/76381 for the script
addition in the gate configuration.

Change-Id: I08d8380183c2128c62fa0ca52306950d8076f3c2

2 files changed, 109 insertions, 4 deletions

diff --git a/README.rst b/README.rst
index 4474a1a..9c61274 100644
--- a/README.rst
+++ b/README.rst
@@ -2,12 +2,49 @@
  oslotest
 ==========
 
-OpenStack test framework
+OpenStack test framework and test fixtures
 
 * Free software: Apache license
 * Documentation: http://docs.openstack.org/developer/oslotest
 
-Features
---------
+Cross-testing With Other Projects
+=================================
 
-* TODO
+The oslotest package is cross-tested against its consuming projects to
+ensure that no changes to the library break the tests in those other
+projects.
+
+In the Gate
+-----------
+
+To add your project to the list for cross-testing, update
+``modules/openstack_project/files/jenkins_job_builder/config/projects.yaml``
+in the openstack-infra/config git repository and add sections like:
+
+::
+
+   - '{pipeline}-oslo.test-dsvm-{name}{branch-designator}':
+       pipeline: check
+       node: 'devstack-precise || devstack-precise-check'
+       branch-designator: ''
+       branch-override: default
+   - '{pipeline}-oslo.test-dsvm-{name}{branch-designator}':
+       pipeline: gate
+       node: devstack-precise
+       branch-designator: ''
+       branch-override: default
+
+To the ``jobs`` list for your project. Refer to
+https://review.openstack.org/#/c/76381 for an example.
+
+Locally
+-------
+
+To run the cross-tests locally, invoke the script directly, passing
+the path to the other source repository and the tox environment name
+to use:
+
+::
+
+  $ cd oslo.test
+  $ ./tools/run_cross_tests.sh ~/repos/openstack/oslo.config py27
diff --git a/tools/run_cross_tests.sh b/tools/run_cross_tests.sh
new file mode 100755
index 0000000..0a20839
--- /dev/null
+++ b/tools/run_cross_tests.sh
@@ -0,0 +1,68 @@
+#!/bin/bash
+#
+# Run cross-project tests
+
+# Fail the build if any command fails
+set -e
+
+project_dir="$1"
+venv="$2"
+
+# Set up the virtualenv without running the tests
+(cd $project_dir && tox --notest -e $venv)
+
+tox_envbin=$project_dir/.tox/$venv/bin
+
+# Replace the pip-installed package with the version in our source
+# tree. Look to see if oslotest is already installed before trying to
+# uninstall it, to avoid failures from packages that do not use it
+# yet.
+if $tox_envbin/pip freeze | grep -q oslotest
+then
+    $tox_envbin/pip uninstall -y oslotest
+fi
+$tox_envbin/pip install -U .
+
+# Run the tests
+(cd $project_dir && tox -e $venv)
+result=$?
+
+
+# The below checks are modified from
+# openstack-infra/config/modules/jenkins/files/slave_scripts/run-unittests.sh.
+
+# They expect to be run in the project being tested.
+cd $project_dir
+
+echo "Begin pip freeze output from test virtualenv:"
+echo "======================================================================"
+.tox/$venv/bin/pip freeze
+echo "======================================================================"
+
+# We only want to run the next check if the tool is installed, so look
+# for it before continuing.
+if [ -f /usr/local/jenkins/slave_scripts/subunit2html.py -a -d ".testrepository" ] ; then
+    if [ -f ".testrepository/0.2" ] ; then
+        cp .testrepository/0.2 ./subunit_log.txt
+    elif [ -f ".testrepository/0" ] ; then
+        .tox/$venv/bin/subunit-1to2 < .testrepository/0 > ./subunit_log.txt
+    fi
+    .tox/$venv/bin/python /usr/local/jenkins/slave_scripts/subunit2html.py ./subunit_log.txt testr_results.html
+    gzip -9 ./subunit_log.txt
+    gzip -9 ./testr_results.html
+
+    export PYTHON=.tox/$venv/bin/python
+    set -e
+    rancount=$(.tox/$venv/bin/testr last | sed -ne 's/Ran \([0-9]\+\).*tests in.*/\1/p')
+    if [ "$rancount" -eq "0" ] ; then
+        echo
+        echo "Zero tests were run. At least one test should have been run."
+        echo "Failing this test as a result"
+        echo
+        exit 1
+    fi
+fi
+
+# If we make it this far, report status based on the tests that were
+# run.
+exit $result