bpo-39429: Add a new "Python Development Mode" doc page (GH-18132)

13 files changed, 268 insertions, 49 deletions

diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index 79a8815ed4..108bd2c024 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -466,7 +466,7 @@ PyConfig
 
    .. c:member:: int dev_mode
 
-      Development mode: see :option:`-X dev <-X>`.
+      If non-zero, enable the :ref:`Python Development Mode <devmode>`.
 
    .. c:member:: int dump_refs
 
diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst
index 101e7817a9..ff51c4fa3b 100644
--- a/Doc/library/asyncio-dev.rst
+++ b/Doc/library/asyncio-dev.rst
@@ -25,7 +25,7 @@ There are several ways to enable asyncio debug mode:
 
 * Setting the :envvar:`PYTHONASYNCIODEBUG` environment variable to ``1``.
 
-* Using the :option:`-X` ``dev`` Python command line option.
+* Using the :ref:`Python Development Mode <devmode>`.
 
 * Passing ``debug=True`` to :func:`asyncio.run`.
 
diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index 25a3692695..0029d94f0b 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -1200,7 +1200,7 @@ Enabling debug mode
 
    .. versionchanged:: 3.7
 
-      The new ``-X dev`` command line option can now also be used
+      The new :ref:`Python Development Mode <devmode>` can now also be used
       to enable the debug mode.
 
 .. seealso::
diff --git a/Doc/library/development.rst b/Doc/library/development.rst
index ab34e1f7ce..9edce75868 100644
--- a/Doc/library/development.rst
+++ b/Doc/library/development.rst
@@ -18,12 +18,10 @@ The list of modules described in this chapter is:
 
    typing.rst
    pydoc.rst
+   devmode.rst
    doctest.rst
    unittest.rst
    unittest.mock.rst
    unittest.mock-examples.rst
    2to3.rst
    test.rst
-
-See also the Python development mode: the :option:`-X` ``dev`` option and
-:envvar:`PYTHONDEVMODE` environment variable.
diff --git a/Doc/library/devmode.rst b/Doc/library/devmode.rst
new file mode 100644
index 0000000000..d5a40cdeea
--- /dev/null
+++ b/Doc/library/devmode.rst
@@ -0,0 +1,214 @@
+.. _devmode:
+
+Python Development Mode
+=======================
+
+.. versionadded:: 3.7
+
+The Python Development Mode introduces additional runtime checks that are too
+expensive to be enabled by default. It should not be more verbose than the
+default if the code is correct; new warnings are only emitted when an issue is
+detected.
+
+It can be enabled using the :option:`-X dev <-X>` command line option or by
+setting the :envvar:`PYTHONDEVMODE` environment variable to ``1``.
+
+Effects of the Python Development Mode
+======================================
+
+Enabling the Python Development Mode is similar to the following command, but
+with additional effects described below::
+
+    PYTHONMALLOC=debug PYTHONASYNCIODEBUG=1 python3 -W default -X faulthandler
+
+Effects of the Python Development Mode:
+
+* Add ``default`` :ref:`warning filter <describing-warning-filters>`. The
+  following warnings are shown:
+
+  * :exc:`DeprecationWarning`
+  * :exc:`ImportWarning`
+  * :exc:`PendingDeprecationWarning`
+  * :exc:`ResourceWarning`
+
+  Normally, the above warnings are filtered by the default :ref:`warning
+  filters <describing-warning-filters>`.
+
+  It behaves as if the :option:`-W default <-W>` command line option is used.
+
+  Use the :option:`-W error <-W>` command line option or set the
+  :envvar:`PYTHONWARNINGS` environment variable to ``error`` to treat warnings
+  as errors.
+
+* Install debug hooks on memory allocators to check for:
+
+  * Buffer underflow
+  * Buffer overflow
+  * Memory allocator API violation
+  * Unsafe usage of the GIL
+
+  See the :c:func:`PyMem_SetupDebugHooks` C function.
+
+  It behaves as if the :envvar:`PYTHONMALLOC` environment variable is set to
+  ``debug``.
+
+  To enable the Python Development Mode without installing debug hooks on
+  memory allocators, set the :envvar:`PYTHONMALLOC` environment variable to
+  ``default``.
+
+* Call :func:`faulthandler.enable` at Python startup to install handlers for
+  the :const:`SIGSEGV`, :const:`SIGFPE`, :const:`SIGABRT`, :const:`SIGBUS` and
+  :const:`SIGILL` signals to dump the Python traceback on a crash.
+
+  It behaves as if the :option:`-X faulthandler <-X>` command line option is
+  used or if the :envvar:`PYTHONFAULTHANDLER` environment variable is set to
+  ``1``.
+
+* Enable :ref:`asyncio debug mode <asyncio-debug-mode>`. For example,
+  :mod:`asyncio` checks for coroutines that were not awaited and logs them.
+
+  It behaves as if the :envvar:`PYTHONASYNCIODEBUG` environment variable is set
+  to ``1``.
+
+* Check the *encoding* and *errors* arguments for string encoding and decoding
+  operations. Examples: :func:`open`, :meth:`str.encode` and
+  :meth:`bytes.decode`.
+
+  By default, for best performance, the *errors* argument is only checked at
+  the first encoding/decoding error and the *encoding* argument is sometimes
+  ignored for empty strings.
+
+* The :class:`io.IOBase` destructor logs ``close()`` exceptions.
+* Set the :attr:`~sys.flags.dev_mode` attribute of :attr:`sys.flags` to
+  ``True``.
+
+The Python Development Mode does not enable the :mod:`tracemalloc` module by
+default, because the overhead cost (to performance and memory) would be too
+large. Enabling the :mod:`tracemalloc` module provides additional information
+on the origin of some errors. For example, :exc:`ResourceWarning` logs the
+traceback where the resource was allocated, and a buffer overflow error logs
+the traceback where the memory block was allocated.
+
+The Python Development Mode does not prevent the :option:`-O` command line
+option from removing :keyword:`assert` statements nor from setting
+:const:`__debug__` to ``False``.
+
+.. versionchanged:: 3.8
+   The :class:`io.IOBase` destructor now logs ``close()`` exceptions.
+
+.. versionchanged:: 3.9
+   The *encoding* and *errors* arguments are now checked for string encoding
+   and decoding operations.
+
+
+ResourceWarning Example
+=======================
+
+Example of a script counting the number of lines of the text file specified in
+the command line::
+
+    import sys
+
+    def main():
+        fp = open(sys.argv[1])
+        nlines = len(fp.readlines())
+        print(nlines)
+        # The file is closed implicitly
+
+    if __name__ == "__main__":
+        main()
+
+The script does not close the file explicitly. By default, Python does not emit
+any warning. Example using README.txt, which has 269 lines:
+
+.. code-block:: shell-session
+
+    $ python3 script.py README.txt
+    269
+
+Enabling the Python Development Mode displays a :exc:`ResourceWarning` warning:
+
+.. code-block:: shell-session
+
+    $ python3 -X dev script.py README.txt
+    269
+    script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
+      main()
+    ResourceWarning: Enable tracemalloc to get the object allocation traceback
+
+In addition, enabling :mod:`tracemalloc` shows the line where the file was
+opened:
+
+.. code-block:: shell-session
+
+    $ python3 -X dev -X tracemalloc=5 script.py README.rst
+    269
+    script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
+      main()
+    Object allocated at (most recent call last):
+      File "script.py", lineno 10
+        main()
+      File "script.py", lineno 4
+        fp = open(sys.argv[1])
+
+The fix is to close explicitly the file. Example using a context manager::
+
+    def main():
+        # Close the file explicitly when exiting the with block
+        with open(sys.argv[1]) as fp:
+            nlines = len(fp.readlines())
+        print(nlines)
+
+Not closing a resource explicitly can leave a resource open for way longer than
+expected; it can cause severe issues upon exiting Python. It is bad in
+CPython, but it is even worse in PyPy. Closing resources explicitly makes an
+application more deterministic and more reliable.
+
+
+Bad file descriptor error example
+=================================
+
+Script displaying the first line of itself::
+
+    import os
+
+    def main():
+        fp = open(__file__)
+        firstline = fp.readline()
+        print(firstline.rstrip())
+        os.close(fp.fileno())
+        # The file is closed implicitly
+
+    main()
+
+By default, Python does not emit any warning:
+
+.. code-block:: shell-session
+
+    $ python3 script.py
+    import os
+
+The Python Development Mode shows a :exc:`ResourceWarning` and logs a "Bad file
+descriptor" error when finalizing the file object:
+
+.. code-block:: shell-session
+
+    $ python3 script.py
+    import os
+    script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
+      main()
+    ResourceWarning: Enable tracemalloc to get the object allocation traceback
+    Exception ignored in: <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
+    Traceback (most recent call last):
+      File "script.py", line 10, in <module>
+        main()
+    OSError: [Errno 9] Bad file descriptor
+
+``os.close(fp.fileno())`` closes the file descriptor. When the file object
+finalizer tries to close the file descriptor again, it fails with the ``Bad
+file descriptor`` error. A file descriptor must be closed only once. In the
+worst case scenario, closing it twice can lead to a crash (see :issue:`18748`
+for an example).
+
+The fix is to remove the ``os.close(fp.fileno())`` line, or open the file with
+``closefd=False``.
diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
index 52a505e0a0..df2cda9d67 100644
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -688,6 +688,10 @@ The following exceptions are used as warning categories; see the
    Base class for warnings about deprecated features when those warnings are
    intended for other Python developers.
 
+   Ignored by the default warning filters, except in the ``__main__`` module
+   (:pep:`565`). Enabling the :ref:`Python Development Mode <devmode>` shows
+   this warning.
+
 
 .. exception:: PendingDeprecationWarning
 
@@ -699,6 +703,9 @@ The following exceptions are used as warning categories; see the
    upcoming deprecation is unusual, and :exc:`DeprecationWarning`
    is preferred for already active deprecations.
 
+   Ignored by the default warning filters. Enabling the :ref:`Python
+   Development Mode <devmode>` shows this warning.
+
 
 .. exception:: SyntaxWarning
 
@@ -720,6 +727,9 @@ The following exceptions are used as warning categories; see the
 
    Base class for warnings about probable mistakes in module imports.
 
+   Ignored by the default warning filters. Enabling the :ref:`Python
+   Development Mode <devmode>` shows this warning.
+
 
 .. exception:: UnicodeWarning
 
@@ -733,8 +743,10 @@ The following exceptions are used as warning categories; see the
 
 .. exception:: ResourceWarning
 
-   Base class for warnings related to resource usage. Ignored by the default
-   warning filters.
+   Base class for warnings related to resource usage.
+
+   Ignored by the default warning filters. Enabling the :ref:`Python
+   Development Mode <devmode>` shows this warning.
 
    .. versionadded:: 3.2
 
diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst
index b588dfa18d..59274c1dd7 100644
--- a/Doc/library/faulthandler.rst
+++ b/Doc/library/faulthandler.rst
@@ -40,6 +40,9 @@ alternatively be passed to :func:`faulthandler.enable`.
 The module is implemented in C, so tracebacks can be dumped on a crash or when
 Python is deadlocked.
 
+The :ref:`Python Development Mode <devmode>` calls :func:`faulthandler.enable`
+at Python startup.
+
 
 Dumping the traceback
 ---------------------
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
index 3e25faaa42..fd3401fd18 100644
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -1559,8 +1559,8 @@ expression support in the :mod:`re` module).
    list of possible encodings, see section :ref:`standard-encodings`.
 
    By default, the *errors* argument is not checked for best performances, but
-   only used at the first encoding error. Enable the development mode
-   (:option:`-X` ``dev`` option), or use a debug build, to check *errors*.
+   only used at the first encoding error. Enable the :ref:`Python Development
+   Mode <devmode>`, or use a debug build to check *errors*.
 
    .. versionchanged:: 3.1
       Support for keyword arguments added.
@@ -2596,8 +2596,8 @@ arbitrary binary data.
    list of possible encodings, see section :ref:`standard-encodings`.
 
    By default, the *errors* argument is not checked for best performances, but
-   only used at the first decoding error. Enable the development mode
-   (:option:`-X` ``dev`` option), or use a debug build, to check *errors*.
+   only used at the first decoding error. Enable the :ref:`Python Development
+   Mode <devmode>`, or use a debug build to check *errors*.
 
    .. note::
 
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index 351a8e4c9e..d28b3565c1 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -428,9 +428,9 @@ always available.
    The :term:`named tuple` *flags* exposes the status of command line
    flags. The attributes are read only.
 
-   ============================= =============================
+   ============================= ================================================================
    attribute                     flag
-   ============================= =============================
+   ============================= ================================================================
    :const:`debug`                :option:`-d`
    :const:`inspect`              :option:`-i`
    :const:`interactive`          :option:`-i`
@@ -444,9 +444,9 @@ always available.
    :const:`bytes_warning`        :option:`-b`
    :const:`quiet`                :option:`-q`
    :const:`hash_randomization`   :option:`-R`
-   :const:`dev_mode`             :option:`-X` ``dev``
-   :const:`utf8_mode`            :option:`-X` ``utf8``
-   ============================= =============================
+   :const:`dev_mode`             :option:`-X dev <-X>` (:ref:`Python Development Mode <devmode>`)
+   :const:`utf8_mode`            :option:`-X utf8 <-X>`
+   ============================= ================================================================
 
    .. versionchanged:: 3.2
       Added ``quiet`` attribute for the new :option:`-q` flag.
@@ -461,8 +461,9 @@ always available.
       Added ``isolated`` attribute for :option:`-I` ``isolated`` flag.
 
    .. versionchanged:: 3.7
-      Added ``dev_mode`` attribute for the new :option:`-X` ``dev`` flag
-      and ``utf8_mode`` attribute for the new  :option:`-X` ``utf8`` flag.
+      Added the ``dev_mode`` attribute for the new :ref:`Python Development
+      Mode <devmode>` and the ``utf8_mode`` attribute for the new  :option:`-X`
+      ``utf8`` flag.
 
 
 .. data:: float_info
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
index 9e149806c3..146003b147 100644
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -442,24 +442,9 @@ Miscellaneous options
      nested imports).  Note that its output may be broken in multi-threaded
      application.  Typical usage is ``python3 -X importtime -c 'import
      asyncio'``.  See also :envvar:`PYTHONPROFILEIMPORTTIME`.
-   * ``-X dev``: enable CPython's "development mode", introducing additional
-     runtime checks which are too expensive to be enabled by default. It should
-     not be more verbose than the default if the code is correct: new warnings
-     are only emitted when an issue is detected. Effect of the developer mode:
-
-     * Check *encoding* and *errors* arguments on string encoding and decoding
-       operations. Examples: :func:`open`, :meth:`str.encode` and
-       :meth:`bytes.decode`.
-     * Add ``default`` warning filter, as :option:`-W` ``default``.
-     * Install debug hooks on memory allocators: see the
-       :c:func:`PyMem_SetupDebugHooks` C function.
-     * Enable the :mod:`faulthandler` module to dump the Python traceback
-       on a crash.
-     * Enable :ref:`asyncio debug mode <asyncio-debug-mode>`.
-     * Set the :attr:`~sys.flags.dev_mode` attribute of :attr:`sys.flags` to
-       ``True``.
-     * :class:`io.IOBase` destructor logs ``close()`` exceptions.
-
+   * ``-X dev``: enable :ref:`Python Development Mode <devmode>`, introducing
+     additional runtime checks that are too expensive to be enabled by
+     default.
    * ``-X utf8`` enables UTF-8 mode for operating system interfaces, overriding
      the default locale-aware mode. ``-X utf8=0`` explicitly disables UTF-8
      mode (even when it would otherwise activate automatically).
@@ -890,8 +875,9 @@ conflict.
 
 .. envvar:: PYTHONDEVMODE
 
-   If this environment variable is set to a non-empty string, enable the
-   CPython "development mode". See the :option:`-X` ``dev`` option.
+   If this environment variable is set to a non-empty string, enable
+   :ref:`Python Development Mode <devmode>`, introducing additional runtime
+   checks that are too expensive to be enabled by default.
 
    .. versionadded:: 3.7
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 8a70fe22d5..04cfa57e14 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -102,7 +102,7 @@ CPython implementation improvements:
   * :ref:`PEP 538 <whatsnew37-pep538>`, legacy C locale coercion
   * :ref:`PEP 540 <whatsnew37-pep540>`, forced UTF-8 runtime mode
 * :ref:`PEP 552 <whatsnew37-pep552>`, deterministic .pycs
-* :ref:`the new development runtime mode <whatsnew37-devmode>`
+* :ref:`New Python Development Mode <whatsnew37-devmode>`
 * :ref:`PEP 565 <whatsnew37-pep565>`, improved :exc:`DeprecationWarning`
   handling
 
@@ -479,15 +479,15 @@ Three new translations have been added:
 
 .. _whatsnew37-devmode:
 
-Development Runtime Mode: -X dev
+Python Development Mode (-X dev)
 --------------------------------
 
 The new :option:`-X` ``dev`` command line option or the new
 :envvar:`PYTHONDEVMODE` environment variable can be used to enable
-CPython's *development mode*.  When in development mode, CPython performs
+:ref:`Python Development Mode <devmode>`.  When in development mode, Python performs
 additional runtime checks that are too expensive to be enabled by default.
-See :option:`-X` ``dev`` documentation for the full description of the effects
-of this mode.
+See :ref:`Python Development Mode <devmode>` documentation for the full
+description.
 
 
 Other Language Changes
diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index a6e938faa9..ff5cb1486f 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -90,9 +90,10 @@ Other Language Changes
   in this case.
   (Contributed by Victor Stinner in :issue:`20443`.)
 
-* In development mode and in debug build, *encoding* and *errors* arguments are
-  now checked on string encoding and decoding operations. Examples:
-  :func:`open`, :meth:`str.encode` and :meth:`bytes.decode`.
+* In the :ref:`Python Development Mode <devmode>` and in debug build, the
+  *encoding* and *errors* arguments are now checked for string encoding and
+  decoding operations. Examples: :func:`open`, :meth:`str.encode` and
+  :meth:`bytes.decode`.
 
   By default, for best performance, the *errors* argument is only checked at
   the first encoding/decoding error and the *encoding* argument is sometimes
diff --git a/Include/cpython/initconfig.h b/Include/cpython/initconfig.h
index 4b5ceafe02..54e662347e 100644
--- a/Include/cpython/initconfig.h
+++ b/Include/cpython/initconfig.h
@@ -113,7 +113,11 @@ typedef struct {
        "POSIX", otherwise it is set to 0. Inherit Py_UTF8Mode value value. */
     int utf8_mode;
 
-    int dev_mode;           /* Development mode. PYTHONDEVMODE, -X dev */
+    /* If non-zero, enable the Python Development Mode.
+
+       Set to 1 by the -X dev command line option. Set by the PYTHONDEVMODE
+       environment variable. */
+    int dev_mode;
 
     /* Memory allocator: PYTHONMALLOC env var.
        See PyMemAllocatorName for valid values. */
@@ -131,7 +135,7 @@ typedef struct {
 
     int isolated;         /* Isolated mode? see PyPreConfig.isolated */
     int use_environment;  /* Use environment variables? see PyPreConfig.use_environment */
-    int dev_mode;         /* Development mode? See PyPreConfig.dev_mode */
+    int dev_mode;         /* Python Development Mode? See PyPreConfig.dev_mode */
 
     /* Install signal handlers? Yes by default. */
     int install_signal_handlers;