bpo-43774: Remove unused PYMALLOC_DEBUG macro (GH-25711)

Enhance also the documentation of debug hooks on memory allocators.

8 files changed, 135 insertions, 143 deletions

diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index 325e607061..de85029481 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -229,17 +229,20 @@ PyPreConfig
       Name of the Python memory allocators:
 
       * ``PYMEM_ALLOCATOR_NOT_SET`` (``0``): don't change memory allocators
-        (use defaults)
-      * ``PYMEM_ALLOCATOR_DEFAULT`` (``1``): default memory allocators
-      * ``PYMEM_ALLOCATOR_DEBUG`` (``2``): default memory allocators with
-        debug hooks
-      * ``PYMEM_ALLOCATOR_MALLOC`` (``3``): force usage of ``malloc()``
+        (use defaults).
+      * ``PYMEM_ALLOCATOR_DEFAULT`` (``1``): :ref:`default memory allocators
+        <default-memory-allocators>`.
+      * ``PYMEM_ALLOCATOR_DEBUG`` (``2``): :ref:`default memory allocators
+        <default-memory-allocators>` with :ref:`debug hooks
+        <pymem-debug-hooks>`.
+      * ``PYMEM_ALLOCATOR_MALLOC`` (``3``): use ``malloc()`` of the C library.
       * ``PYMEM_ALLOCATOR_MALLOC_DEBUG`` (``4``): force usage of
-        ``malloc()`` with debug hooks
+        ``malloc()`` with :ref:`debug hooks <pymem-debug-hooks>`.
       * ``PYMEM_ALLOCATOR_PYMALLOC`` (``5``): :ref:`Python pymalloc memory
-        allocator <pymalloc>`
+        allocator <pymalloc>`.
       * ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` (``6``): :ref:`Python pymalloc
-        memory allocator <pymalloc>` with debug hooks
+        memory allocator <pymalloc>` with :ref:`debug hooks
+        <pymem-debug-hooks>`.
 
       ``PYMEM_ALLOCATOR_PYMALLOC`` and ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` are
       not supported if Python is :option:`configured using --without-pymalloc
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
index b945429d87..1cb0990f5d 100644
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -389,7 +389,8 @@ Legend:
 * ``malloc``: system allocators from the standard C library, C functions:
   :c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`.
 * ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`.
-* "+ debug": with debug hooks installed by :c:func:`PyMem_SetupDebugHooks`.
+* "+ debug": with :ref:`debug hooks on the Python memory allocators
+  <pymem-debug-hooks>`.
 * "Debug build": :ref:`Python build in debug mode <debug-build>`.
 
 .. _customize-memory-allocators:
@@ -478,45 +479,113 @@ Customize Memory Allocators
 
 .. c:function:: void PyMem_SetupDebugHooks(void)
 
-   Setup hooks to detect bugs in the Python memory allocator functions.
+   Setup :ref:`debug hooks in the Python memory allocators <pymem-debug-hooks>`
+   to detect memory errors.
+
+
+.. _pymem-debug-hooks:
+
+Debug hooks on the Python memory allocators
+===========================================
+
+When :ref:`Python is built is debug mode <debug-build>`, the
+:c:func:`PyMem_SetupDebugHooks` function is called at the :ref:`Python
+preinitialization <c-preinit>` to setup debug hooks on Python memory allocators
+to detect memory errors.
+
+The :envvar:`PYTHONMALLOC` environment variable can be used to install debug
+hooks on a Python compiled in release mode (ex: ``PYTHONMALLOC=debug``).
+
+The :c:func:`PyMem_SetupDebugHooks` function can be used to set debug hooks
+after calling :c:func:`PyMem_SetAllocator`.
+
+These debug hooks fill dynamically allocated memory blocks with special,
+recognizable bit patterns. Newly allocated memory is filled with the byte
+``0xCD`` (``PYMEM_CLEANBYTE``), freed memory is filled with the byte ``0xDD``
+(``PYMEM_DEADBYTE``). Memory blocks are surrounded by "forbidden bytes"
+filled with the byte ``0xFD`` (``PYMEM_FORBIDDENBYTE``). Strings of these bytes
+are unlikely to be valid addresses, floats, or ASCII strings.
+
+Runtime checks:
+
+- Detect API violations. For example, detect if :c:func:`PyObject_Free` is
+  called on a memory block allocated by :c:func:`PyMem_Malloc`.
+- Detect write before the start of the buffer (buffer underflow).
+- Detect write after the end of the buffer (buffer overflow).
+- Check that the :term:`GIL <global interpreter lock>` is held when
+  allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex:
+  :c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex:
+  :c:func:`PyMem_Malloc`) domains are called.
+
+On error, the debug hooks use the :mod:`tracemalloc` module to get the
+traceback where a memory block was allocated. The traceback is only displayed
+if :mod:`tracemalloc` is tracing Python memory allocations and the memory block
+was traced.
+
+Let *S* = ``sizeof(size_t)``. ``2*S`` bytes are added at each end of each block
+of *N* bytes requested.  The memory layout is like so, where p represents the
+address returned by a malloc-like or realloc-like function (``p[i:j]`` means
+the slice of bytes from ``*(p+i)`` inclusive up to ``*(p+j)`` exclusive; note
+that the treatment of negative indices differs from a Python slice):
+
+``p[-2*S:-S]``
+    Number of bytes originally asked for.  This is a size_t, big-endian (easier
+    to read in a memory dump).
+``p[-S]``
+    API identifier (ASCII character):
+
+    * ``'r'`` for :c:data:`PYMEM_DOMAIN_RAW`.
+    * ``'m'`` for :c:data:`PYMEM_DOMAIN_MEM`.
+    * ``'o'`` for :c:data:`PYMEM_DOMAIN_OBJ`.
+
+``p[-S+1:0]``
+    Copies of PYMEM_FORBIDDENBYTE.  Used to catch under- writes and reads.
+
+``p[0:N]``
+    The requested memory, filled with copies of PYMEM_CLEANBYTE, used to catch
+    reference to uninitialized memory.  When a realloc-like function is called
+    requesting a larger memory block, the new excess bytes are also filled with
+    PYMEM_CLEANBYTE.  When a free-like function is called, these are
+    overwritten with PYMEM_DEADBYTE, to catch reference to freed memory.  When
+    a realloc- like function is called requesting a smaller memory block, the
+    excess old bytes are also filled with PYMEM_DEADBYTE.
+
+``p[N:N+S]``
+    Copies of PYMEM_FORBIDDENBYTE.  Used to catch over- writes and reads.
+
+``p[N+S:N+2*S]``
+    Only used if the ``PYMEM_DEBUG_SERIALNO`` macro is defined (not defined by
+    default).
+
+    A serial number, incremented by 1 on each call to a malloc-like or
+    realloc-like function.  Big-endian ``size_t``.  If "bad memory" is detected
+    later, the serial number gives an excellent way to set a breakpoint on the
+    next run, to capture the instant at which this block was passed out.  The
+    static function bumpserialno() in obmalloc.c is the only place the serial
+    number is incremented, and exists so you can set such a breakpoint easily.
+
+A realloc-like or free-like function first checks that the PYMEM_FORBIDDENBYTE
+bytes at each end are intact.  If they've been altered, diagnostic output is
+written to stderr, and the program is aborted via Py_FatalError().  The other
+main failure mode is provoking a memory error when a program reads up one of
+the special bit patterns and tries to use it as an address.  If you get in a
+debugger then and look at the object, you're likely to see that it's entirely
+filled with PYMEM_DEADBYTE (meaning freed memory is getting used) or
+PYMEM_CLEANBYTE (meaning uninitialized memory is getting used).
 
-   Newly allocated memory is filled with the byte ``0xCD`` (``CLEANBYTE``),
-   freed memory is filled with the byte ``0xDD`` (``DEADBYTE``). Memory blocks
-   are surrounded by "forbidden bytes" (``FORBIDDENBYTE``: byte ``0xFD``).
-
-   Runtime checks:
-
-   - Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
-     allocated by :c:func:`PyMem_Malloc`
-   - Detect write before the start of the buffer (buffer underflow)
-   - Detect write after the end of the buffer (buffer overflow)
-   - Check that the :term:`GIL <global interpreter lock>` is held when
-     allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex:
-     :c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex:
-     :c:func:`PyMem_Malloc`) domains are called
-
-   On error, the debug hooks use the :mod:`tracemalloc` module to get the
-   traceback where a memory block was allocated. The traceback is only
-   displayed if :mod:`tracemalloc` is tracing Python memory allocations and the
-   memory block was traced.
-
-   These hooks are :ref:`installed by default <default-memory-allocators>` if
-   :ref:`Python is built in debug mode <debug-build>`.
-   The :envvar:`PYTHONMALLOC` environment variable can be used to install
-   debug hooks on a Python compiled in release mode.
-
-   .. versionchanged:: 3.6
-      This function now also works on Python compiled in release mode.
-      On error, the debug hooks now use :mod:`tracemalloc` to get the traceback
-      where a memory block was allocated. The debug hooks now also check
-      if the GIL is held when functions of :c:data:`PYMEM_DOMAIN_OBJ` and
-      :c:data:`PYMEM_DOMAIN_MEM` domains are called.
+.. versionchanged:: 3.6
+   The :c:func:`PyMem_SetupDebugHooks` function now also works on Python
+   compiled in release mode.  On error, the debug hooks now use
+   :mod:`tracemalloc` to get the traceback where a memory block was allocated.
+   The debug hooks now also check if the GIL is held when functions of
+   :c:data:`PYMEM_DOMAIN_OBJ` and :c:data:`PYMEM_DOMAIN_MEM` domains are
+   called.
 
-   .. versionchanged:: 3.8
-      Byte patterns ``0xCB`` (``CLEANBYTE``), ``0xDB`` (``DEADBYTE``) and
-      ``0xFB`` (``FORBIDDENBYTE``) have been replaced with ``0xCD``, ``0xDD``
-      and ``0xFD`` to use the same values than Windows CRT debug ``malloc()``
-      and ``free()``.
+.. versionchanged:: 3.8
+   Byte patterns ``0xCB`` (``PYMEM_CLEANBYTE``), ``0xDB`` (``PYMEM_DEADBYTE``)
+   and ``0xFB`` (``PYMEM_FORBIDDENBYTE``) have been replaced with ``0xCD``,
+   ``0xDD`` and ``0xFD`` to use the same values than Windows CRT debug
+   ``malloc()`` and ``free()``.
 
 
 .. _pymalloc:
@@ -539,6 +608,10 @@ The arena allocator uses the following functions:
 * :c:func:`mmap` and :c:func:`munmap` if available,
 * :c:func:`malloc` and :c:func:`free` otherwise.
 
+This allocator is disabled if Python is configured with the
+:option:`--without-pymalloc` option. It can also be disabled at runtime using
+the :envvar:`PYTHONMALLOC` environment variable (ex: ``PYTHONMALLOC=malloc``).
+
 Customize pymalloc Arena Allocator
 ----------------------------------
 
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
index 09d8fa97f0..25e05d413d 100644
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -799,17 +799,13 @@ conflict.
      :c:data:`PYMEM_DOMAIN_MEM` and :c:data:`PYMEM_DOMAIN_OBJ` domains and use
      the :c:func:`malloc` function for the :c:data:`PYMEM_DOMAIN_RAW` domain.
 
-   Install debug hooks:
+   Install :ref:`debug hooks <pymem-debug-hooks>`:
 
    * ``debug``: install debug hooks on top of the :ref:`default memory
      allocators <default-memory-allocators>`.
    * ``malloc_debug``: same as ``malloc`` but also install debug hooks.
    * ``pymalloc_debug``: same as ``pymalloc`` but also install debug hooks.
 
-   See the :ref:`default memory allocators <default-memory-allocators>` and the
-   :c:func:`PyMem_SetupDebugHooks` function (install debug hooks on Python
-   memory allocators).
-
    .. versionchanged:: 3.7
       Added the ``"default"`` allocator.
 
diff --git a/Include/Python.h b/Include/Python.h
index 1df2fd5343..4d0335d3c5 100644
--- a/Include/Python.h
+++ b/Include/Python.h
@@ -72,15 +72,6 @@
 #  endif
 #endif
 
-/* Debug-mode build with pymalloc implies PYMALLOC_DEBUG.
- *  PYMALLOC_DEBUG is in error if pymalloc is not in use.
- */
-#if defined(Py_DEBUG) && defined(WITH_PYMALLOC) && !defined(PYMALLOC_DEBUG)
-#define PYMALLOC_DEBUG
-#endif
-#if defined(PYMALLOC_DEBUG) && !defined(WITH_PYMALLOC)
-#error "PYMALLOC_DEBUG requires WITH_PYMALLOC"
-#endif
 #include "pymath.h"
 #include "pymem.h"
 
diff --git a/Include/objimpl.h b/Include/objimpl.h
index 689c42b747..450befad67 100644
--- a/Include/objimpl.h
+++ b/Include/objimpl.h
@@ -48,8 +48,8 @@ Functions and macros for modules that implement new object types.
 
 Note that objects created with PyObject_{New, NewVar} are allocated using the
 specialized Python allocator (implemented in obmalloc.c), if WITH_PYMALLOC is
-enabled.  In addition, a special debugging allocator is used if PYMALLOC_DEBUG
-is also #defined.
+enabled.  In addition, a special debugging allocator is used if Py_DEBUG
+macro is also defined.
 
 In case a specific form of memory management is needed (for example, if you
 must use the platform malloc heap(s), or shared memory, or C++ local storage or
diff --git a/Include/pymem.h b/Include/pymem.h
index 2f770b1fbf..66cdb0d297 100644
--- a/Include/pymem.h
+++ b/Include/pymem.h
@@ -25,8 +25,8 @@ extern "C" {
    heap used by the Python DLL; it could be a disaster if you free()'ed that
    directly in your own extension.  Using PyMem_Free instead ensures Python
    can return the memory to the proper heap.  As another example, in
-   PYMALLOC_DEBUG mode, Python wraps all calls to all PyMem_ and PyObject_
-   memory functions in special debugging wrappers that add additional
+   a debug build (Py_DEBUG macro), Python wraps all calls to all PyMem_ and
+   PyObject_ memory functions in special debugging wrappers that add additional
    debugging info to dynamic memory blocks.  The system routines have no idea
    what to do with that stuff, and the Python wrappers have no idea what to do
    with raw blocks obtained directly by the system routines then.
diff --git a/Misc/NEWS.d/next/C API/2021-04-29-10-17-21.bpo-43774.5MGfgN.rst b/Misc/NEWS.d/next/C API/2021-04-29-10-17-21.bpo-43774.5MGfgN.rst
new file mode 100644
index 0000000000..9664b55271
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2021-04-29-10-17-21.bpo-43774.5MGfgN.rst
@@ -0,0 +1,5 @@
+Remove the now unused ``PYMALLOC_DEBUG`` macro. Debug hooks on memory
+allocators are now installed by default if Python is built in debug mode (if
+``Py_DEBUG`` macro is defined). Moreover, they can now be used on Python
+build in release mode (ex: using ``PYTHONMALLOC=debug`` environment
+variable).
diff --git a/Misc/SpecialBuilds.txt b/Misc/SpecialBuilds.txt
index 27369abfb3..a7cee8098e 100644
--- a/Misc/SpecialBuilds.txt
+++ b/Misc/SpecialBuilds.txt
@@ -77,90 +77,14 @@ envvar PYTHONDUMPREFS
     combinerefs.py, were new in Python 2.3b1.
 
 
-PYMALLOC_DEBUG
---------------
-
-When pymalloc is enabled (WITH_PYMALLOC is defined), calls to the PyObject_
-memory routines are handled by Python's own small-object allocator, while calls
-to the PyMem_ memory routines are directed to the system malloc/ realloc/free.
-If PYMALLOC_DEBUG is also defined, calls to both PyObject_ and PyMem_ memory
-routines are directed to a special debugging mode of Python's small-object
-allocator.
-
-This mode fills dynamically allocated memory blocks with special, recognizable
-bit patterns, and adds debugging info on each end of dynamically allocated
-memory blocks.  The special bit patterns are:
-
-#define CLEANBYTE     0xCB   /* clean (newly allocated) memory */
-#define DEADBYTE      0xDB   /* dead (newly freed) memory */
-#define FORBIDDENBYTE 0xFB   /* forbidden -- untouchable bytes */
-
-Strings of these bytes are unlikely to be valid addresses, floats, or 7-bit
-ASCII strings.
-
-Let S = sizeof(size_t). 2*S bytes are added at each end of each block of N bytes
-requested.  The memory layout is like so, where p represents the address
-returned by a malloc-like or realloc-like function (p[i:j] means the slice of
-bytes from *(p+i) inclusive up to *(p+j) exclusive; note that the treatment of
-negative indices differs from a Python slice):
-
-p[-2*S:-S]
-    Number of bytes originally asked for.  This is a size_t, big-endian (easier
-    to read in a memory dump).
-p[-S]
-    API ID.  See PEP 445.  This is a character, but seems undocumented.
-p[-S+1:0]
-    Copies of FORBIDDENBYTE.  Used to catch under- writes and reads.
-p[0:N]
-    The requested memory, filled with copies of CLEANBYTE, used to catch
-    reference to uninitialized memory.  When a realloc-like function is called
-    requesting a larger memory block, the new excess bytes are also filled with
-    CLEANBYTE.  When a free-like function is called, these are overwritten with
-    DEADBYTE, to catch reference to freed memory.  When a realloc- like function
-    is called requesting a smaller memory block, the excess old bytes are also
-    filled with DEADBYTE.
-p[N:N+S]
-    Copies of FORBIDDENBYTE.  Used to catch over- writes and reads.
-p[N+S:N+2*S]
-    A serial number, incremented by 1 on each call to a malloc-like or
-    realloc-like function.  Big-endian size_t.  If "bad memory" is detected
-    later, the serial number gives an excellent way to set a breakpoint on the
-    next run, to capture the instant at which this block was passed out.  The
-    static function bumpserialno() in obmalloc.c is the only place the serial
-    number is incremented, and exists so you can set such a breakpoint easily.
-
-A realloc-like or free-like function first checks that the FORBIDDENBYTEs at
-each end are intact.  If they've been altered, diagnostic output is written to
-stderr, and the program is aborted via Py_FatalError().  The other main failure
-mode is provoking a memory error when a program reads up one of the special bit
-patterns and tries to use it as an address.  If you get in a debugger then and
-look at the object, you're likely to see that it's entirely filled with 0xDB
-(meaning freed memory is getting used) or 0xCB (meaning uninitialized memory is
-getting used).
-
-Note that PYMALLOC_DEBUG requires WITH_PYMALLOC. Py_DEBUG implies
-PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled).
-
-Special gimmicks:
-
-envvar PYTHONMALLOCSTATS
-    If this envvar exists, a report of pymalloc summary statistics is printed to
-    stderr whenever a new arena is allocated, and also by Py_FinalizeEx().
-
-Changed in 2.5:  The number of extra bytes allocated is 4*sizeof(size_t).
-Before it was 16 on all boxes, reflecting that Python couldn't make use of
-allocations >= 2**32 bytes even on 64-bit boxes before 2.5.
-
-
 Py_DEBUG
 --------
 
 This is what is generally meant by "a debug build" of Python.
 
-Py_DEBUG implies LLTRACE, Py_REF_DEBUG, and PYMALLOC_DEBUG (if
-WITH_PYMALLOC is enabled).  In addition, C assert()s are enabled (via the C way:
-by not defining NDEBUG), and some routines do additional sanity checks inside
-"#ifdef Py_DEBUG" blocks.
+Py_DEBUG implies LLTRACE and Py_REF_DEBUG. In addition, C assert()s are enabled
+(via the C way: by not defining NDEBUG), and some routines do additional sanity
+checks inside "#ifdef Py_DEBUG" blocks.
 
 
 LLTRACE