Use virtualenv and Pip to find runtime deps; remove searching for upstream

Co-authored-by: Sam Thursfield <sam.thursfield@codethink.co.uk>

Since upstream pip do not want to merge https://github.com/pypa/pip/pull/2371
we should avoid depending on this pull request.

To find runtime dependencies we now run pip install inside a virtual env
then run pip freeze to obtain the dependency set, this has the advantage
that nearly all the work is being done by pip.

Originally the python extensions were designed to look for upstream git
repos, in practice this is unreliable and won't be compatible with obtaining
dependencies using pip install, so the downside of this approach is that
all lorries will be tarballs, the upshot is that we can now automatically
import many packages that we couldn't import before.

Another upshot of this approach is that we may be able to consider the
removal of a lot of the spec processing and validation code if we're willing
to worry less about build dependencies, we're not sure whether we should
be willing to worry less about build dependencies though.

We've had encouraging results using this patch so far, we are now able
to import, without user intervention, packages that failed previously,
such as boto, persistent-pineapple, jmespath, coverage, requests also
almost imported successfully but appears to require a release of pytest
that is uploaded as a zip.

Change-Id: I705c6f6bd722df041d17630287382f851008e97a

1 files changed, 0 insertions, 65 deletions

diff --git a/TODO.python b/TODO.python
index 16b7889..6c276f7 100644
--- a/TODO.python
+++ b/TODO.python
@@ -25,68 +25,3 @@ this will be confusing, we should emit nice version numbers.
 i.e. nixtla
 
 * add a test runner
-
-* Importing python packages that use pbr fails, see
-https://bitbucket.org/pypa/setuptools/issue/73/typeerror-dist-must-be-a-distribution#comment-7267980
-The most sensible option would seem to be to make use of the sane environment
-that pbr provides: just read the dependency information from the text files
-that pbr projects provide, see, http://docs.openstack.org/developer/pbr/
-
-Results from running the import tool on various python packages follow:
-
-* Imports tested so far (stratum is generated)
-    * SUCCEEDS
-        * nixtla: fine but requires compilation
-        * ryser
-        * Twisted
-        * Django
-        * textdata
-        * whale-agent
-        * virtualenv
-        * lxml
-        * nose
-        * six
-        * simplejson
-        * pika
-        * MarkupSafe
-        * zc.buildout
-        * Paste
-        * pycrypto
-        * Jinja2
-        * Flask
-        * bcdoc
-        * pymongo
-
-    * FAILS
-        * python-keystoneclient
-            * All openstack stuff requires pbr, pbr does not play nicely with
-                current setuptools see: [Issue 73](https://bitbucket.org/pypa/setuptoolsissue/73/typeerror-dist-must-be-a-distribution#comment-7267980)
-            we can either fix setuptools/pbr or make use of the sane environment
-            pbr provides.
-        * persistent-pineapple
-            * Git repo[1] has different layout to tarball[2] downloadeable from pypi,
-            git repo's layout isn't 'installable' by pip, so dependencies can
-            not be determined.
-            [1]: https://github.com/JasonAUnrein/Persistent-Pineapple
-            [2]: https://pypi.python.org/packages/source/p/persistent_pineapple/persistent_pineapple-1.0.0.tar.gz
-        * ftw.blog
-            * cannot satisfy dependencies
-        * boto
-            * cannot satisfy dependencies
-        * jmespath
-            * cannot satisfy dependencies
-        * rejester
-            * its setup.py subclasses distutils.core
-        * requests
-            * cannot satisfy dependencies
-        * MySQL-python
-            * egg_info blows up,
-            * python setup.py install doesn't even work
-            * maybe the user's expected to do some manual stuff first, who knows
-        * rejester (its setup.py subclasses distutils.core)
-        * redis-jobs (succeeded at first, no longer exists on pypi)
-        * coverage (stratum couldn't be generated because some tags are missing)
-
-* Imports completely tested, built, deployed and executed successfully:
-
-    * Flask