PEP-517 source distribution support (#954)

create a ``.package`` virtual environment to perform build operations inside
Resolves #573 and #820.

29 files changed, 656 insertions, 163 deletions

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 28c10ffd..e4054d81 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -15,7 +15,7 @@ repos:
     rev: v1.0.1
     hooks:
     -   id: seed-isort-config
-        args: [--application-directories, src]
+        args: [--application-directories, "src:."]
 -   repo: https://github.com/pre-commit/mirrors-isort
     rev: v4.3.4
     hooks:
diff --git a/.vsts-ci.yml b/.vsts-ci.yml
index 865a092e..b4134316 100644
--- a/.vsts-ci.yml
+++ b/.vsts-ci.yml
@@ -1,5 +1,8 @@
 name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr)
 
+variables:
+  "System.PreferGit": true
+
 trigger:
   branches:
     include:
diff --git a/changelog/573.feature.rst b/changelog/573.feature.rst
new file mode 100644
index 00000000..a982c1ed
--- /dev/null
+++ b/changelog/573.feature.rst
@@ -0,0 +1,2 @@
+`PEP-517 <https://www.python.org/dev/peps/pep-0517/>`_ source distribution support (create a
+``.package`` virtual environment to perform build operations inside) by :user:`gaborbernat`
diff --git a/changelog/820.feature.rst b/changelog/820.feature.rst
new file mode 100644
index 00000000..1a8be1bd
--- /dev/null
+++ b/changelog/820.feature.rst
@@ -0,0 +1 @@
+`flit <https://flit.readthedocs.io>`_ support via implementing ``PEP-517`` by :user:`gaborbernat`
diff --git a/doc/config.rst b/doc/config.rst
index ed4b9d42..4b3679e2 100644
--- a/doc/config.rst
+++ b/doc/config.rst
@@ -79,6 +79,8 @@ and will first lookup global tox settings in this section:
 
 .. confval:: requires=LIST
 
+    .. versionadded:: 3.2.0
+
     Specify python packages that need to exist alongside the tox installation for the tox build
     to be able to start. Use this to specify plugin requirements and build dependencies.
 
@@ -88,6 +90,25 @@ and will first lookup global tox settings in this section:
         requires = setuptools >= 30.0.0
                    py
 
+.. confval:: isolated_build=True|False(default)
+
+    .. versionadded:: 3.3.0
+
+    Activate isolated build environment. tox will use a virtual environment to build
+    a source distribution from the source tree. For build tools and arguments use
+    the ``pyproject.toml`` file as specified in
+    `PEP-517 <https://www.python.org/dev/peps/pep-0517/>`_ and
+    `PEP-518 <https://www.python.org/dev/peps/pep-0518/>`_. To specify the virtual
+    environment Python version define use the :confval:`isolated_build_env` config
+    section.
+
+.. confval:: isolated_build_env=str
+
+    .. versionadded:: 3.3.0
+
+    Name of the virtual environment used to create a source distribution from the
+    source tree. By **default ``.package``** is used.
+
 
 Virtualenv test environment settings
 ------------------------------------
diff --git a/doc/example/package.rst b/doc/example/package.rst
new file mode 100644
index 00000000..a5d5e741
--- /dev/null
+++ b/doc/example/package.rst
@@ -0,0 +1,60 @@
+packaging
+=========
+
+Although one can use tox to develop and test applications one of its most popular
+usage is to help library creators. Libraries need first to be packaged, so then
+they can be installed inside a virtual environment for testing. To help with this
+tox implements `PEP-517 <https://www.python.org/dev/peps/pep-0517/>`_ and
+`PEP-518 <https://www.python.org/dev/peps/pep-0518/>`_. This means that by default
+tox will build source distribution out of source trees. Before running test commands
+``pip`` is used to install the source distribution inside the build environment.
+
+To create a source distribution there are multiple tools out there and with ``PEP-517``
+and ``PEP-518`` you can easily use your favorite one with tox. Historically tox
+only supported ``setuptools``, and always used the tox host environment to build
+a source distribution from the source tree. This is still the default behavior.
+To opt out of this behaviour you need to set isolated builds to true.
+
+setuptools
+----------
+Using the ``pyproject.toml`` file at the root folder (alongside ``setup.py``) one can specify
+build requirements.
+
+.. code-block:: toml
+
+    [build-system]
+    requires = [
+        "setuptools >= 35.0.2",
+        "setuptools_scm >= 2.0.0, <3"
+    ]
+    build-backend = "setuptools.build_meta"
+
+.. code-block:: ini
+
+   # tox.ini
+   [tox]
+   build_isolated = True
+
+flit
+----
+`flit <https://flit.readthedocs.io/en/latest/>`_ requires ``Python 3``, however the generated source
+distribution can be installed under ``python 2``. Furthermore it does not require a ``setup.py``
+file as that information is also added to the ``pyproject.toml`` file.
+
+.. code-block:: toml
+
+    [build-system]
+    requires = ["flit >= 1.1"]
+    build-backend = "flit.buildapi"
+
+    [tool.flit.metadata]
+    module = "package_toml_flit"
+    author = "Happy Harry"
+    author-email = "happy@harry.com"
+    home-page = "https://github.com/happy-harry/is"
+
+.. code-block:: ini
+
+   # tox.ini
+   [tox]
+   build_isolated = True
diff --git a/doc/examples.rst b/doc/examples.rst
index 6ae0f588..7975343d 100644
--- a/doc/examples.rst
+++ b/doc/examples.rst
@@ -6,6 +6,7 @@ tox configuration and usage examples
    :maxdepth: 2
 
    example/basic.rst
+   example/package.rst
    example/pytest.rst
    example/unittest
    example/nose.rst
diff --git a/setup.py b/setup.py
index 100cc001..471ce265 100644
--- a/setup.py
+++ b/setup.py
@@ -66,6 +66,7 @@ def main():
             "py >= 1.4.17, <2",
             "six >= 1.0.0, <2",
             "virtualenv >= 1.11.2",
+            "toml >=0.9.4",
         ],
         extras_require={
             "testing": [
@@ -76,7 +77,11 @@ def main():
                 "pytest-xdist >= 1.22.2, <2",
                 "pytest-randomly >= 1.2.3, <2",
             ],
-            "docs": ["sphinx >= 1.7.5, < 2", "towncrier >= 18.5.0"],
+            "docs": [
+                "sphinx >= 1.7.5, < 2",
+                "towncrier >= 18.5.0",
+                "pygments-github-lexers >= 0.0.5",
+            ],
         },
         classifiers=[
             "Development Status :: 5 - Production/Stable",
diff --git a/src/tox/_pytestplugin.py b/src/tox/_pytestplugin.py
index 321d511b..9e218cde 100644
--- a/src/tox/_pytestplugin.py
+++ b/src/tox/_pytestplugin.py
@@ -277,7 +277,7 @@ def initproj(tmpdir):
             setup.py
     """
 
-    def initproj_(nameversion, filedefs=None, src_root="."):
+    def initproj_(nameversion, filedefs=None, src_root=".", add_missing_setup_py=True):
         if filedefs is None:
             filedefs = {}
         if not src_root:
@@ -297,7 +297,7 @@ def initproj(tmpdir):
 
         base.ensure(dir=1)
         create_files(base, filedefs)
-        if not _filedefs_contains(base, filedefs, "setup.py"):
+        if not _filedefs_contains(base, filedefs, "setup.py") and add_missing_setup_py:
             create_files(
                 base,
                 {
@@ -319,7 +319,18 @@ def initproj(tmpdir):
             )
         if not _filedefs_contains(base, filedefs, src_root_path.join(name)):
             create_files(
-                src_root_path, {name: {"__init__.py": "__version__ = {!r}".format(version)}}
+                src_root_path,
+                {
+                    name: {
+                        "__init__.py": textwrap.dedent(
+                            '''
+                """ module {} """
+                __version__ = {!r}'''
+                        )
+                        .strip()
+                        .format(name, version)
+                    }
+                },
             )
         manifestlines = [
             "include {}".format(p.relto(base)) for p in base.visit(lambda x: x.check(file=1))
diff --git a/src/tox/config.py b/src/tox/config.py
index 891eb0a9..70dcb425 100755
--- a/src/tox/config.py
+++ b/src/tox/config.py
@@ -974,7 +974,8 @@ class parseini:
         config.logdir = config.toxworkdir.join("log")
         self._make_thread_safe_path(config, "logdir", unique_id)
 
-        config.envlist, all_envs = self._getenvdata(reader)
+        self.parse_build_isolation(config, reader)
+        config.envlist, all_envs = self._getenvdata(reader, config)
 
         # factors used in config or predefined
         known_factors = self._list_section_factors("testenv")
@@ -1006,6 +1007,16 @@ class parseini:
 
         config.skipsdist = reader.getbool("skipsdist", all_develop)
 
+    def parse_build_isolation(self, config, reader):
+        config.isolated_build = reader.getbool("isolated_build", False)
+        config.isolated_build_env = reader.getstring("isolated_build_env", ".package")
+        if config.isolated_build is True:
+            name = config.isolated_build_env
+            if name not in config.envconfigs:
+                config.envconfigs[name] = self.make_envconfig(
+                    name, testenvprefix + name, reader._subs, config
+                )
+
     def _make_thread_safe_path(self, config, attr, unique_id):
         if config.option.parallel_safe_build:
             path = getattr(config, attr)
@@ -1069,7 +1080,7 @@ class parseini:
                 reader.addsubstitutions(**{env_attr.name: res})
         return tc
 
-    def _getenvdata(self, reader):
+    def _getenvdata(self, reader, config):
         candidates = (
             self.config.option.env,
             os.environ.get("TOXENV"),
@@ -1086,9 +1097,17 @@ class parseini:
         if not all_envs:
             all_envs.add("python")
 
+        package_env = config.isolated_build_env
+        if config.isolated_build is True and package_env in all_envs:
+            all_envs.remove(package_env)
+
         if not env_list or "ALL" in env_list:
             env_list = sorted(all_envs)
 
+        if config.isolated_build is True and package_env in env_list:
+            msg = "isolated_build_env {} cannot be part of envlist".format(package_env)
+            raise tox.exception.ConfigError(msg)
+
         return env_list, all_envs
 
 
diff --git a/src/tox/package.py b/src/tox/package.py
index b23cb1a3..54e6558b 100644
--- a/src/tox/package.py
+++ b/src/tox/package.py
@@ -1,8 +1,17 @@
+import json
 import sys
+import textwrap
+from collections import namedtuple
 
+import pkg_resources
 import py
+import six
+import toml
 
 import tox
+from tox.config import DepConfig
+
+BuildInfo = namedtuple("BuildInfo", ["requires", "backend_module", "backend_object"])
 
 
 @tox.hookimpl
@@ -27,10 +36,9 @@ def get_package(session):
         report.info("using package {!r}, skipping 'sdist' activity ".format(str(path)))
     else:
         try:
-            path = make_sdist(report, config, session)
-        except tox.exception.InvocationError:
-            v = sys.exc_info()[1]
-            report.error("FAIL could not package project - v = {!r}".format(v))
+            path = build_package(config, report, session)
+        except tox.exception.InvocationError as exception:
+            report.error("FAIL could not package project - v = {!r}".format(exception))
             return None
         sdist_file = config.distshare.join(path.basename)
         if sdist_file != path:
@@ -44,7 +52,14 @@ def get_package(session):
     return path
 
 
-def make_sdist(report, config, session):
+def build_package(config, report, session):
+    if not config.isolated_build:
+        return make_sdist_legacy(report, config, session)
+    else:
+        return build_isolated(config, report, session)
+
+
+def make_sdist_legacy(report, config, session):
     setup = config.setupdir.join("setup.py")
     if not setup.check():
         report.error(
@@ -83,3 +98,113 @@ def make_sdist(report, config, session):
                 "     python setup.py sdist"
             )
             raise SystemExit(1)
+
+
+def build_isolated(config, report, session):
+    build_info = get_build_info(config.setupdir, report)
+    package_venv = session.getvenv(config.isolated_build_env)
+    package_venv.envconfig.deps_matches_subset = True
+
+    # we allow user specified dependencies so the users can write extensions to
+    # install additional type of dependencies (e.g. binary)
+    user_specified_deps = package_venv.envconfig.deps
+    package_venv.envconfig.deps = [DepConfig(r, None) for r in build_info.requires]
+    package_venv.envconfig.deps.extend(user_specified_deps)
+
+    if not session.setupenv(package_venv):
+        session.finishvenv(package_venv)
+
+    build_requires = get_build_requires(build_info, package_venv, session)
+    # we need to filter out requirements already specified in pyproject.toml or user deps
+    base_build_deps = {pkg_resources.Requirement(r.name).key for r in package_venv.envconfig.deps}
+    build_requires_dep = [
+        DepConfig(r, None)
+        for r in build_requires
+        if pkg_resources.Requirement(r).key not in base_build_deps
+    ]
+    if build_requires_dep:
+        with session.newaction(
+            package_venv, "build_requires", package_venv.envconfig.envdir
+        ) as action:
+            package_venv.run_install_command(packages=build_requires_dep, action=action)
+        session.finishvenv(package_venv)
+    return perform_isolated_build(build_info, package_venv, session, config)
+
+
+def get_build_info(folder, report):
+    toml_file = folder.join("pyproject.toml")
+
+    # as per https://www.python.org/dev/peps/pep-0517/
+
+    def abort(message):
+        report.error("{} inside {}".format(message, toml_file))
+        raise SystemExit(1)
+
+    if not toml_file.exists():
+        report.error("missing {}".format(toml_file))
+        raise SystemExit(1)
+
+    with open(str(toml_file)) as file_handler:
+        config_data = toml.load(file_handler)
+
+    if "build-system" not in config_data:
+        abort("build-system section missing")
+
+    build_system = config_data["build-system"]
+
+    if "requires" not in build_system:
+        abort("missing requires key at build-system section")
+    if "build-backend" not in build_system:
+        abort("missing build-backend key at build-system section")
+
+    requires = build_system["requires"]
+    if not isinstance(requires, list) or not all(isinstance(i, six.text_type) for i in requires):
+        abort("requires key at build-system section must be a list of string")
+
+    backend = build_system["build-backend"]
+    if not isinstance(backend, six.text_type):
+        abort("build-backend key at build-system section must be a string")
+
+    args = backend.split(":")
+    module = args[0]
+    obj = "" if len(args) == 1 else ".{}".format(args[1])
+
+    return BuildInfo(requires, module, "{}{}".format(module, obj))
+
+
+def perform_isolated_build(build_info, package_venv, session, config):
+    with session.newaction(
+        package_venv, "perform isolated build", package_venv.envconfig.envdir
+    ) as action:
+        script = textwrap.dedent(
+            """
+            import sys
+            import {}
+            basename = {}.build_{}({!r}, {{ "--global-option": ["--formats=gztar"]}})
+            print(basename)""".format(
+                build_info.backend_module, build_info.backend_object, "sdist", str(config.distdir)
+            )
+        )
+        config.distdir.ensure_dir()
+        result = action.popen([package_venv.envconfig.envpython, "-c", script], returnout=True)
+        return config.distdir.join(result.split("\n")[-2])
+
+
+def get_build_requires(build_info, package_venv, session):
+    with session.newaction(
+        package_venv, "get build requires", package_venv.envconfig.envdir
+    ) as action:
+        script = textwrap.dedent(
+            """
+                import {}
+                import json
+
+                backend = {}
+                for_build_requires = backend.get_requires_for_build_{}(None)
+                print(json.dumps(for_build_requires))
+                        """.format(
+                build_info.backend_module, build_info.backend_object, "sdist"
+            )
+        ).strip()
+        result = action.popen([package_venv.envconfig.envpython, "-c", script], returnout=True)
+        return json.loads(result.split("\n")[-2])
diff --git a/src/tox/session.py b/src/tox/session.py
index 12db45cf..ae6eb283 100644
--- a/src/tox/session.py
+++ b/src/tox/session.py
@@ -263,7 +263,7 @@ class Reporter(object):
     def __init__(self, session):
         self.tw = py.io.TerminalWriter()
         self.session = session
-        self._reportedlines = []
+        self.reported_lines = []
 
     @property
     def verbosity(self):
@@ -338,7 +338,7 @@ class Reporter(object):
             self.logline("SKIPPED: {}".format(msg), yellow=True)
 
     def logline(self, msg, **opts):
-        self._reportedlines.append(msg)
+        self.reported_lines.append(msg)
         self.tw.line("{}".format(msg), **opts)
 
     def verbosity0(self, msg, **opts):
@@ -619,7 +619,9 @@ class Session:
     def showenvs(self, all_envs=False, description=False):
         env_conf = self.config.envconfigs  # this contains all environments
         default = self.config.envlist  # this only the defaults
-        extra = sorted(e for e in env_conf if e not in default) if all_envs else []
+        ignore = {self.config.isolated_build_env}.union(default)
+        extra = sorted(e for e in env_conf if e not in ignore) if all_envs else []
+
         if description:
             self.report.line("default environments:")
             max_length = max(len(env) for env in (default + extra))
diff --git a/src/tox/venv.py b/src/tox/venv.py
index f93b9445..978ff5df 100755
--- a/src/tox/venv.py
+++ b/src/tox/venv.py
@@ -53,7 +53,7 @@ class CreationConfig:
         except Exception:
             return None
 
-    def matches(self, other):
+    def matches(self, other, deps_matches_subset=False):
         return (
             other
             and self.md5 == other.md5
@@ -62,7 +62,11 @@ class CreationConfig:
             and self.sitepackages == other.sitepackages
             and self.usedevelop == other.usedevelop
             and self.alwayscopy == other.alwayscopy
-            and self.deps == other.deps
+            and (
+                all(d in self.deps for d in other.deps)
+                if deps_matches_subset is True
+                else self.deps == other.deps
+            )
         )
 
 
@@ -159,7 +163,13 @@ class VirtualEnv(object):
             if status string is empty, all is ok.
         """
         rconfig = CreationConfig.readconfig(self.path_config)
-        if not self.envconfig.recreate and rconfig and rconfig.matches(self._getliveconfig()):
+        if (
+            not self.envconfig.recreate
+            and rconfig
+            and rconfig.matches(
+                self._getliveconfig(), getattr(self.envconfig, "deps_matches_subset", False)
+            )
+        ):
             action.info("reusing", self.envconfig.envdir)
             return
         if rconfig is None:
@@ -173,9 +183,8 @@ class VirtualEnv(object):
             return sys.exc_info()[1]
         try:
             self.hook.tox_testenv_install_deps(action=action, venv=self)
-        except tox.exception.InvocationError:
-            v = sys.exc_info()[1]
-            return "could not install deps {}; v = {!r}".format(self.envconfig.deps, v)
+        except tox.exception.InvocationError as exception:
+            return "could not install deps {}; v = {!r}".format(self.envconfig.deps, exception)
 
     def _getliveconfig(self):
         python = self.envconfig.python_info.executable
diff --git a/tests/__init__.py b/tests/__init__.py
new file mode 100644
index 00000000..e69de29b
--- /dev/null
+++ b/tests/__init__.py
diff --git a/tests/integration/test_package_int.py b/tests/integration/test_package_int.py
new file mode 100644
index 00000000..0558d0ff
--- /dev/null
+++ b/tests/integration/test_package_int.py
@@ -0,0 +1,78 @@
+"""Tests that require external access (e.g. pip install, virtualenv creation)"""
+import os
+import subprocess
+import sys
+
+import pytest
+
+from tests.lib import need_git
+
+
+@pytest.mark.network
+def test_package_isolated_build_setuptools(initproj, cmd):
+    initproj(
+        "package_toml_setuptools-0.1",
+        filedefs={
+            "tox.ini": """
+                    [tox]
+                    isolated_build = true
+                    [testenv:.package]
+                    basepython = python
+                """,
+            "pyproject.toml": """
+                    [build-system]
+                    requires = ["setuptools >= 35.0.2", "setuptools_scm >= 2.0.0, <3"]
+                    build-backend = 'setuptools.build_meta'
+                    """,
+        },
+    )
+    result = cmd("--sdistonly")
+    assert result.ret == 0, result.out
+
+    result2 = cmd("--sdistonly")
+    assert result2.ret == 0, result.out
+    assert ".package recreate" not in result2.out
+
+
+@pytest.mark.network
+@need_git
+@pytest.mark.skipif(sys.version_info < (3, 0), reason="flit is Python 3 only")
+def test_package_isolated_build_flit(initproj, cmd):
+    initproj(
+        "package_toml_flit-0.1",
+        filedefs={
+            "tox.ini": """
+                    [tox]
+                    isolated_build = true
+                    [testenv:.package]
+                    basepython = python
+                """,
+            "pyproject.toml": """
+                    [build-system]
+                    requires = ["flit"]
+                    build-backend = "flit.buildapi"
+
+                    [tool.flit.metadata]
+                    module = "package_toml_flit"
+                    author = "Happy Harry"
+                    author-email = "happy@harry.com"
+                    home-page = "https://github.com/happy-harry/is"
+                    """,
+            ".gitignore": ".tox",
+        },
+        add_missing_setup_py=False,
+    )
+    env = os.environ.copy()
+    env["GIT_COMMITTER_NAME"] = "committer joe"
+    env["GIT_AUTHOR_NAME"] = "author joe"
+    env["EMAIL"] = "joe@example.com"
+    subprocess.check_call(["git", "init"], env=env)
+    subprocess.check_call(["git", "add", "-A", "."], env=env)
+    subprocess.check_call(["git", "commit", "-m", "first commit", "--no-gpg-sign"], env=env)
+    result = cmd("--sdistonly")
+    assert result.ret == 0, result.out
+
+    result2 = cmd("--sdistonly")
+
+    assert result2.ret == 0, result.out
+    assert ".package recreate" not in result2.out
diff --git a/tests/lib/__init__.py b/tests/lib/__init__.py
new file mode 100644
index 00000000..0f711b20
--- /dev/null
+++ b/tests/lib/__init__.py
@@ -0,0 +1,18 @@
+import subprocess
+
+import pytest
+
+
+def need_executable(name, check_cmd):
+    def wrapper(fn):
+        try:
+            subprocess.check_output(check_cmd)
+        except OSError:
+            return pytest.mark.skip(reason="{} is not available".format(name))(fn)
+        return fn
+
+    return wrapper
+
+
+def need_git(fn):
+    return pytest.mark.git(need_executable("git", ("git", "--version"))(fn))
diff --git a/tests/unit/session/__init__.py b/tests/unit/session/__init__.py
new file mode 100644
index 00000000..e69de29b
--- /dev/null
+++ b/tests/unit/session/__init__.py
diff --git a/tests/unit/session/test_list_env.py b/tests/unit/session/test_list_env.py
new file mode 100644
index 00000000..8ccee897
--- /dev/null
+++ b/tests/unit/session/test_list_env.py
@@ -0,0 +1,152 @@
+def test_listenvs(cmd, initproj):
+    initproj(
+        "listenvs",
+        filedefs={
+            "tox.ini": """
+        [tox]
+        envlist=py36,py27,py34,pypi,docs
+        description= py27: run pytest on Python 2.7
+                     py34: run pytest on Python 3.6
+                     pypi: publish to PyPI
+                     docs: document stuff
+                     notincluded: random extra
+
+        [testenv:notincluded]
+        changedir = whatever
+
+        [testenv:docs]
+        changedir = docs
+        """
+        },
+    )
+    result = cmd("-l")
+    assert result.outlines == ["py36", "py27", "py34", "pypi", "docs"]
+
+
+def test_listenvs_verbose_description(cmd, initproj):
+    initproj(
+        "listenvs_verbose_description",
+        filedefs={
+            "tox.ini": """
+        [tox]
+        envlist=py36,py27,py34,pypi,docs
+        [testenv]
+        description= py36: run pytest on Python 3.6
+                     py27: run pytest on Python 2.7
+                     py34: run pytest on Python 3.4
+                     pypi: publish to PyPI
+                     docs: document stuff
+                     notincluded: random extra
+
+        [testenv:notincluded]
+        changedir = whatever
+
+        [testenv:docs]
+        changedir = docs
+        description = let me overwrite that
+        """
+        },
+    )
+    result = cmd("-lv")
+    expected = [
+        "default environments:",
+        "py36 -> run pytest on Python 3.6",
+        "py27 -> run pytest on Python 2.7",
+        "py34 -> run pytest on Python 3.4",
+        "pypi -> publish to PyPI",
+        "docs -> let me overwrite that",
+    ]
+    assert result.outlines[2:] == expected
+
+
+def test_listenvs_all(cmd, initproj):
+    initproj(
+        "listenvs_all",
+        filedefs={
+            "tox.ini": """
+        [tox]
+        envlist=py36,py27,py34,pypi,docs
+
+        [testenv:notincluded]
+        changedir = whatever
+
+        [testenv:docs]
+        changedir = docs
+        """
+        },
+    )
+    result = cmd("-a")
+    expected = ["py36", "py27", "py34", "pypi", "docs", "notincluded"]
+    assert result.outlines == expected
+
+
+def test_listenvs_all_verbose_description(cmd, initproj):
+    initproj(
+        "listenvs_all_verbose_description",
+        filedefs={
+            "tox.ini": """
+        [tox]
+        envlist={py27,py36}-{windows,linux} # py35
+        [testenv]
+        description= py27: run pytest on Python 2.7
+                     py36: run pytest on Python 3.6
+                     windows: on Windows platform
+                     linux: on Linux platform
+                     docs: generate documentation
+        commands=pytest {posargs}
+
+        [testenv:docs]
+        changedir = docs
+        """
+        },
+    )
+    result = cmd("-av")
+    expected = [
+        "default environments:",
+        "py27-windows -> run pytest on Python 2.7 on Windows platform",
+        "py27-linux   -> run pytest on Python 2.7 on Linux platform",
+        "py36-windows -> run pytest on Python 3.6 on Windows platform",
+        "py36-linux   -> run pytest on Python 3.6 on Linux platform",
+        "",
+        "additional environments:",
+        "docs         -> generate documentation",
+    ]
+    assert result.outlines[-len(expected) :] == expected
+
+
+def test_listenvs_all_verbose_description_no_additional_environments(cmd, initproj):
+    initproj(
+        "listenvs_all_verbose_description",
+        filedefs={
+            "tox.ini": """
+        [tox]
+        envlist=py27,py36
+        """
+        },
+    )
+    result = cmd("-av")
+    expected = ["default environments:", "py27 -> [no description]", "py36 -> [no description]"]
+    assert result.out.splitlines()[-3:] == expected
+    assert "additional environments" not in result.out
+
+
+def test_listenvs_packaging_excluded(cmd, initproj):
+    initproj(
+        "listenvs",
+        filedefs={
+            "tox.ini": """
+        [tox]
+        envlist = py36,py27,py34,pypi,docs
+        isolated_build = True
+
+        [testenv:notincluded]
+        changedir = whatever
+
+        [testenv:docs]
+        changedir = docs
+        """
+        },
+    )
+    result = cmd("-a")
+    expected = ["py36", "py27", "py34", "pypi", "docs", "notincluded"]
+    assert result.outlines == expected, result.outlines
diff --git a/tests/test_session.py b/tests/unit/session/test_session.py
index 61125fac..61125fac 100644
--- a/tests/test_session.py
+++ b/tests/unit/session/test_session.py
diff --git a/tests/test_config.py b/tests/unit/test_config.py
index a21abd85..cf458e99 100644
--- a/tests/test_config.py
+++ b/tests/unit/test_config.py
@@ -2376,137 +2376,6 @@ class TestCmdInvocation:
         assert "some-repr" in version_info
         assert "1.0" in version_info
 
-    def test_listenvs(self, cmd, initproj):
-        initproj(
-            "listenvs",
-            filedefs={
-                "tox.ini": """
-            [tox]
-            envlist=py36,py27,py34,pypi,docs
-            description= py27: run pytest on Python 2.7
-                         py34: run pytest on Python 3.6
-                         pypi: publish to PyPI
-                         docs: document stuff
-                         notincluded: random extra
-
-            [testenv:notincluded]
-            changedir = whatever
-
-            [testenv:docs]
-            changedir = docs
-            """
-            },
-        )
-        result = cmd("-l")
-        assert result.outlines == ["py36", "py27", "py34", "pypi", "docs"]
-
-    def test_listenvs_verbose_description(self, cmd, initproj):
-        initproj(
-            "listenvs_verbose_description",
-            filedefs={
-                "tox.ini": """
-            [tox]
-            envlist=py36,py27,py34,pypi,docs
-            [testenv]
-            description= py36: run pytest on Python 3.6
-                         py27: run pytest on Python 2.7
-                         py34: run pytest on Python 3.4
-                         pypi: publish to PyPI
-                         docs: document stuff
-                         notincluded: random extra
-
-            [testenv:notincluded]
-            changedir = whatever
-
-            [testenv:docs]
-            changedir = docs
-            description = let me overwrite that
-            """
-            },
-        )
-        result = cmd("-lv")
-        expected = [
-            "default environments:",
-            "py36 -> run pytest on Python 3.6",
-            "py27 -> run pytest on Python 2.7",
-            "py34 -> run pytest on Python 3.4",
-            "pypi -> publish to PyPI",
-            "docs -> let me overwrite that",
-        ]
-        assert result.outlines[2:] == expected
-
-    def test_listenvs_all(self, cmd, initproj):
-        initproj(
-            "listenvs_all",
-            filedefs={
-                "tox.ini": """
-            [tox]
-            envlist=py36,py27,py34,pypi,docs
-
-            [testenv:notincluded]
-            changedir = whatever
-
-            [testenv:docs]
-            changedir = docs
-            """
-            },
-        )
-        result = cmd("-a")
-        expected = ["py36", "py27", "py34", "pypi", "docs", "notincluded"]
-        assert result.outlines == expected
-
-    def test_listenvs_all_verbose_description(self, cmd, initproj):
-        initproj(
-            "listenvs_all_verbose_description",
-            filedefs={
-                "tox.ini": """
-            [tox]
-            envlist={py27,py36}-{windows,linux} # py35
-            [testenv]
-            description= py27: run pytest on Python 2.7
-                         py36: run pytest on Python 3.6
-                         windows: on Windows platform
-                         linux: on Linux platform
-                         docs: generate documentation
-            commands=pytest {posargs}
-
-            [testenv:docs]
-            changedir = docs
-            """
-            },
-        )
-        result = cmd("-av")
-        expected = [
-            "default environments:",
-            "py27-windows -> run pytest on Python 2.7 on Windows platform",
-            "py27-linux   -> run pytest on Python 2.7 on Linux platform",
-            "py36-windows -> run pytest on Python 3.6 on Windows platform",
-            "py36-linux   -> run pytest on Python 3.6 on Linux platform",
-            "",
-            "additional environments:",
-            "docs         -> generate documentation",
-        ]
-        assert result.outlines[-len(expected) :] == expected
-
-    def test_listenvs_all_verbose_description_no_additional_environments(self, cmd, initproj):
-        initproj(
-            "listenvs_all_verbose_description",
-            filedefs={
-                "tox.ini": """
-            [tox]
-            envlist=py27,py36
-            """
-            },
-        )
-        result = cmd("-av")
-        expected = [
-            "default environments:",
-            "py27 -> [no description]",
-            "py36 -> [no description]",
-        ]
-        assert result.out.splitlines()[-3:] == expected
-        assert "additional environments" not in result.out
-
     def test_config_specific_ini(self, tmpdir, cmd):
         ini = tmpdir.ensure("hello.ini")
         result = cmd("-c", ini, "--showconfig")
@@ -2738,3 +2607,20 @@ def test_plugin_require(newconfig, capsys):
         ]
     )
     assert not out
+
+
+def test_isolated_build_env_cannot_be_in_envlist(newconfig, capsys):
+    inisource = """
+            [tox]
+            envlist = py36,package
+            isolated_build = True
+            isolated_build_env = package
+        """
+    with pytest.raises(
+        tox.exception.ConfigError, match="isolated_build_env package cannot be part of envlist"
+    ):
+        newconfig([], inisource)
+
+    out, err = capsys.readouterr()
+    assert not err
+    assert not out
diff --git a/tests/test_docs.py b/tests/unit/test_docs.py
index 85d7049b..c3e9b7b7 100644
--- a/tests/test_docs.py
+++ b/tests/unit/test_docs.py
@@ -18,7 +18,7 @@ INI_BLOCK_RE = re.compile(
 
 
 RST_FILES = []
-TOX_ROOT = os.path.join(os.path.dirname(os.path.dirname(__file__)))
+TOX_ROOT = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
 for root, _, filenames in os.walk(os.path.join(TOX_ROOT, "doc")):
     for f in filenames:
         if f.endswith(".rst"):
diff --git a/tests/test_interpreters.py b/tests/unit/test_interpreters.py
index bcc27450..bcc27450 100644
--- a/tests/test_interpreters.py
+++ b/tests/unit/test_interpreters.py
diff --git a/tests/test_package.py b/tests/unit/test_package.py
index e8bd0961..ea885544 100644
--- a/tests/test_package.py
+++ b/tests/unit/test_package.py
@@ -1,8 +1,11 @@
 import re
 
+import py
+import pytest
+
 from tox.config import parseconfig
-from tox.package import get_package
-from tox.session import Session
+from tox.package import get_build_info, get_package
+from tox.session import Reporter, Session
 
 
 def test_make_sdist(initproj):
@@ -138,3 +141,91 @@ def test_installpkg(tmpdir, newconfig):
     session = Session(config)
     sdist_path = get_package(session)
     assert sdist_path == p
+
+
+def test_package_isolated_no_pyproject_toml(initproj, cmd):
+    initproj(
+        "package_no_toml-0.1",
+        filedefs={
+            "tox.ini": """
+                [tox]
+                isolated_build = true
+            """
+        },
+    )
+    result = cmd("--sdistonly")
+    assert result.ret == 1
+    assert result.outlines == ["ERROR: missing {}".format(py.path.local().join("pyproject.toml"))]
+
+
+def toml_file_check(initproj, version, message, toml):
+    initproj(
+        "package_toml-{}".format(version),
+        filedefs={
+            "tox.ini": """
+                        [tox]
+                        isolated_build = true
+                    """,
+            "pyproject.toml": toml,
+        },
+    )
+    reporter = Reporter(None)
+
+    with pytest.raises(SystemExit, message=1):
+        get_build_info(py.path.local(), reporter)
+    toml_file = py.path.local().join("pyproject.toml")
+    msg = "ERROR: {} inside {}".format(message, toml_file)
+    assert reporter.reported_lines == [msg]
+
+
+def test_package_isolated_toml_no_build_system(initproj, cmd):
+    toml_file_check(initproj, 1, "build-system section missing", "")
+
+
+def test_package_isolated_toml_no_requires(initproj, cmd):
+    toml_file_check(
+        initproj,
+        2,
+        "missing requires key at build-system section",
+        """
+    [build-system]
+    """,
+    )
+
+
+def test_package_isolated_toml_no_backend(initproj, cmd):
+    toml_file_check(
+        initproj,
+        3,
+        "missing build-backend key at build-system section",
+        """
+    [build-system]
+    requires = []
+    """,
+    )
+
+
+def test_package_isolated_toml_bad_requires(initproj, cmd):
+    toml_file_check(
+        initproj,
+        4,
+        "requires key at build-system section must be a list of string",
+        """
+    [build-system]
+    requires = ""
+    build-backend = ""
+    """,
+    )
+
+
+def test_package_isolated_toml_bad_backend(initproj, cmd):
+    toml_file_check(
+        initproj,
+        5,
+        "build-backend key at build-system section must be a string",
+        """
+    [build-system]
+    requires = []
+    build-backend = []
+    """,
+    )
diff --git a/tests/test_pytest_plugins.py b/tests/unit/test_pytest_plugins.py
index 61280000..873731b0 100644
--- a/tests/test_pytest_plugins.py
+++ b/tests/unit/test_pytest_plugins.py
@@ -3,6 +3,8 @@ Test utility tests, intended to cover use-cases not used in the current
 project test suite, e.g. as shown by the code coverage report.
 
 """
+import os
+
 import py.path
 import pytest
 
@@ -16,13 +18,15 @@ class TestInitProj:
     def test_no_src_root(self, kwargs, tmpdir, initproj):
         initproj("black_knight-42", **kwargs)
         init_file = tmpdir.join("black_knight", "black_knight", "__init__.py")
-        assert init_file.read_binary() == b"__version__ = '42'"
+        expected = b'""" module black_knight """' + linesep_bytes() + b"__version__ = '42'"
+        assert init_file.read_binary() == expected
 
     def test_existing_src_root(self, tmpdir, initproj):
         initproj("spam-666", src_root="ham")
         assert not tmpdir.join("spam", "spam").check(exists=1)
         init_file = tmpdir.join("spam", "ham", "spam", "__init__.py")
-        assert init_file.read_binary() == b"__version__ = '666'"
+        expected = b'""" module spam """' + linesep_bytes() + b"__version__ = '666'"
+        assert init_file.read_binary() == expected
 
     def test_prebuilt_src_dir_with_no_src_root(self, tmpdir, initproj):
         initproj("spam-1.0", filedefs={"spam": {}})
@@ -56,7 +60,12 @@ class TestInitProj:
         initproj("spam-666", src_root=src_root)
 
         init_file = tmpdir.join("spam", "spam", "__init__.py")
-        assert init_file.read_binary() == b"__version__ = '666'"
+        expected = b'""" module spam """' + linesep_bytes() + b"__version__ = '666'"
+        assert init_file.read_binary() == expected
+
+
+def linesep_bytes():
+    return os.linesep.encode()
 
 
 class TestPathParts:
diff --git a/tests/test_quickstart.py b/tests/unit/test_quickstart.py
index 724790cc..724790cc 100644
--- a/tests/test_quickstart.py
+++ b/tests/unit/test_quickstart.py
diff --git a/tests/test_result.py b/tests/unit/test_result.py
index 38f02bc1..38f02bc1 100644
--- a/tests/test_result.py
+++ b/tests/unit/test_result.py
diff --git a/tests/test_venv.py b/tests/unit/test_venv.py
index 6e8af7cb..6e8af7cb 100644
--- a/tests/test_venv.py
+++ b/tests/unit/test_venv.py
diff --git a/tests/test_z_cmdline.py b/tests/unit/test_z_cmdline.py
index 20eed9f7..20eed9f7 100644
--- a/tests/test_z_cmdline.py
+++ b/tests/unit/test_z_cmdline.py
diff --git a/tox.ini b/tox.ini
index 2c4cde35..0af68841 100644
--- a/tox.ini
+++ b/tox.ini
@@ -20,7 +20,7 @@ passenv = http_proxy https_proxy no_proxy SSL_CERT_FILE TOXENV CI TRAVIS TRAVIS_
 deps =
 extras = testing
 changedir = {toxinidir}/tests
-commands = pytest {posargs:--cov="{envsitepackagesdir}/tox" --cov-config="{toxinidir}/tox.ini"  --timeout=180 . -n {env:PYTEST_XDIST_PROC_NR:auto} --junitxml={toxworkdir}/test-results.{envname}.xml }
+commands = pytest {posargs:--cov="{envsitepackagesdir}/tox" --cov-config="{toxinidir}/tox.ini"  --timeout=180 . -n {env:PYTEST_XDIST_PROC_NR:auto} --junitxml={toxworkdir}/test-results.{envname}.xml}
 
 [testenv:docs]
 description = invoke sphinx-build to build the HTML docs and check that all links are valid
@@ -28,7 +28,7 @@ basepython = python3.7
 extras = docs
 changedir = {toxinidir}
 commands = sphinx-build -d "{toxworkdir}/docs_doctree" doc "{toxworkdir}/docs_out" --color -W -bhtml {posargs}
-           python -c 'print("documentation available under file://{toxworkdir}/docs_out/index.html")'
+           python -c 'import pathlib; print("documentation available under file://\{0\}".format(pathlib.Path(r"{toxworkdir}") / "docs_out" / "index.html"))'
 
 [testenv:package-description]
 description = check that the long description is valid
@@ -52,7 +52,7 @@ deps = pre-commit == 1.10.3
 skip_install = True
 changedir = {toxinidir}
 commands = pre-commit run --all-files --show-diff-on-failure
-           python -c 'print("hint: run {envdir}/bin/pre-commit install to add checks as pre-commit hook")'
+           python -c 'import pathlib; print("hint: run \{\} install to add checks as pre-commit hook".format(pathlib.Path(r"{envdir}") / "bin" / "pre-commit"))'
 
 
 [testenv:coverage]
@@ -126,8 +126,8 @@ multi_line_output = 3
 include_trailing_comma = True
 force_grid_wrap = 0
 line_length = 99
-known_first_party = tox
-known_third_party = apiclient,git,httplib2,oauth2client,packaging,pkg_resources,pluggy,py,pytest,setuptools,six
+known_first_party = tox,tests
+known_third_party = apiclient,git,httplib2,oauth2client,packaging,pkg_resources,pluggy,py,pytest,setuptools,six,toml
 
 [testenv:release]
 description = do a release, required posarg of the version number