#18894: merge with 3.3.

1 files changed, 121 insertions, 4 deletions

diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
index bdecb3ada5..6401d305ea 100644
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -563,6 +563,68 @@ Skipped tests will not have :meth:`setUp` or :meth:`tearDown` run around them.
 Skipped classes will not have :meth:`setUpClass` or :meth:`tearDownClass` run.
 
 
+.. _subtests:
+
+Distinguishing test iterations using subtests
+---------------------------------------------
+
+.. versionadded:: 3.4
+
+When some of your tests differ only by a some very small differences, for
+instance some parameters, unittest allows you to distinguish them inside
+the body of a test method using the :meth:`~TestCase.subTest` context manager.
+
+For example, the following test::
+
+   class NumbersTest(unittest.TestCase):
+
+       def test_even(self):
+           """
+           Test that numbers between 0 and 5 are all even.
+           """
+           for i in range(0, 6):
+               with self.subTest(i=i):
+                   self.assertEqual(i % 2, 0)
+
+will produce the following output::
+
+   ======================================================================
+   FAIL: test_even (__main__.NumbersTest) (i=1)
+   ----------------------------------------------------------------------
+   Traceback (most recent call last):
+     File "subtests.py", line 32, in test_even
+       self.assertEqual(i % 2, 0)
+   AssertionError: 1 != 0
+
+   ======================================================================
+   FAIL: test_even (__main__.NumbersTest) (i=3)
+   ----------------------------------------------------------------------
+   Traceback (most recent call last):
+     File "subtests.py", line 32, in test_even
+       self.assertEqual(i % 2, 0)
+   AssertionError: 1 != 0
+
+   ======================================================================
+   FAIL: test_even (__main__.NumbersTest) (i=5)
+   ----------------------------------------------------------------------
+   Traceback (most recent call last):
+     File "subtests.py", line 32, in test_even
+       self.assertEqual(i % 2, 0)
+   AssertionError: 1 != 0
+
+Without using a subtest, execution would stop after the first failure,
+and the error would be less easy to diagnose because the value of ``i``
+wouldn't be displayed::
+
+   ======================================================================
+   FAIL: test_even (__main__.NumbersTest)
+   ----------------------------------------------------------------------
+   Traceback (most recent call last):
+     File "subtests.py", line 32, in test_even
+       self.assertEqual(i % 2, 0)
+   AssertionError: 1 != 0
+
+
 .. _unittest-contents:
 
 Classes and functions
@@ -676,6 +738,21 @@ Test cases
       .. versionadded:: 3.1
 
 
+   .. method:: subTest(msg=None, **params)
+
+      Return a context manager which executes the enclosed code block as a
+      subtest.  *msg* and *params* are optional, arbitrary values which are
+      displayed whenever a subtest fails, allowing you to identify them
+      clearly.
+
+      A test case can contain any number of subtest declarations, and
+      they can be arbitrarily nested.
+
+      See :ref:`subtests` for more information.
+
+      .. versionadded:: 3.4
+
+
    .. method:: debug()
 
       Run the test without collecting the result.  This allows exceptions raised
@@ -1393,15 +1470,24 @@ Grouping tests
 
       Tests grouped by a :class:`TestSuite` are always accessed by iteration.
       Subclasses can lazily provide tests by overriding :meth:`__iter__`. Note
-      that this method maybe called several times on a single suite
-      (for example when counting tests or comparing for equality)
-      so the tests returned must be the same for repeated iterations.
+      that this method may be called several times on a single suite (for
+      example when counting tests or comparing for equality) so the tests
+      returned by repeated iterations before :meth:`TestSuite.run` must be the
+      same for each call iteration. After :meth:`TestSuite.run`, callers should
+      not rely on the tests returned by this method unless the caller uses a
+      subclass that overrides :meth:`TestSuite._removeTestAtIndex` to preserve
+      test references.
 
       .. versionchanged:: 3.2
          In earlier versions the :class:`TestSuite` accessed tests directly rather
          than through iteration, so overriding :meth:`__iter__` wasn't sufficient
          for providing tests.
 
+      .. versionchanged:: 3.4
+         In earlier versions the :class:`TestSuite` held references to each
+         :class:`TestCase` after :meth:`TestSuite.run`. Subclasses can restore
+         that behavior by overriding :meth:`TestSuite._removeTestAtIndex`.
+
    In the typical usage of a :class:`TestSuite` object, the :meth:`run` method
    is invoked by a :class:`TestRunner` rather than by the end-user test harness.
 
@@ -1500,7 +1586,9 @@ Loading and running tests
       directory must be specified separately.
 
       If importing a module fails, for example due to a syntax error, then this
-      will be recorded as a single error and discovery will continue.
+      will be recorded as a single error and discovery will continue.  If the
+      import failure is due to :exc:`SkipTest` being raised, it will be recorded
+      as a skip instead of an error.
 
       If a test package name (directory with :file:`__init__.py`) matches the
       pattern then the package will be checked for a ``load_tests``
@@ -1519,6 +1607,15 @@ Loading and running tests
 
       .. versionadded:: 3.2
 
+      .. versionchanged:: 3.4
+         Modules that raise :exc:`SkipTest` on import are recorded as skips,
+         not errors.
+
+      .. versionchanged:: 3.4
+         Paths are sorted before being imported to ensure execution order for a
+         given test suite is the same even if the underlying file system's ordering
+         is not dependent on file name like in ext3/4.
+
 
    The following attributes of a :class:`TestLoader` can be configured either by
    subclassing or assignment on an instance:
@@ -1728,6 +1825,22 @@ Loading and running tests
       :attr:`unexpectedSuccesses` attribute.
 
 
+   .. method:: addSubTest(test, subtest, outcome)
+
+      Called when a subtest finishes.  *test* is the test case
+      corresponding to the test method.  *subtest* is a custom
+      :class:`TestCase` instance describing the subtest.
+
+      If *outcome* is :const:`None`, the subtest succeeded.  Otherwise,
+      it failed with an exception where *outcome* is a tuple of the form
+      returned by :func:`sys.exc_info`: ``(type, value, traceback)``.
+
+      The default implementation does nothing when the outcome is a
+      success, and records subtest failures as normal failures.
+
+      .. versionadded:: 3.4
+
+
 .. class:: TextTestResult(stream, descriptions, verbosity)
 
    A concrete implementation of :class:`TestResult` used by the
@@ -1837,6 +1950,10 @@ Loading and running tests
       The *verbosity*, *failfast*, *catchbreak*, *buffer*
       and *warnings* parameters were added.
 
+   .. versionchanged:: 3.4
+      The *defaultTest* parameter was changed to also accept an iterable of
+      test names.
+
 
 load_tests Protocol
 ###################