ENH,API: Store exported buffer info on the array (#16938)

* ENH,API: Store exported buffer info on the array

This speeds up array deallocation and buffer exports, since it
removes the need to global dictionary lookups.  It also somewhat
simplifies the logic.  The main advantage is prossibly less the
speedup itself (which is not large compared to most things that
happen in the livetime of an array), but rather that no unnecessary
work is done for shortlived arrays, which never export a buffer.

The downside of this approach is that the ABI changes for anyone
who would be subclassing ndarray in C.

* MAINT: Do not tag the NULL (no buffers exported)

The allocation is not the right place to initialize to anything but
NULL, so take the easy path and do not tag the NULL default.

* TST: Add test for best try RuntimeError on corrupt buffer-info

* Remove NPY_SIZEOF_PYARRAYOBJECT and add some documentation

* Use 3 to tag the pointer and object for a "bad" one in the test

* DEP: deprecate the NPY_SIZEOF_PYARRAYOBJECT macro

* Tune down matti's deprecation to write the error instead.

* Tweak macro, so that clang hopefully doesn't complain.

* Use None instead of NULL in PyErr_WriteUnraisable, pypy seems to have a bug with it

* Just comment it out...

* Apply suggestions from code review

Co-authored-by: Matti Picus <matti.picus@gmail.com>

Co-authored-by: mattip <matti.picus@gmail.com>

1 files changed, 24 insertions, 0 deletions

diff --git a/doc/source/reference/c-api/types-and-structures.rst b/doc/source/reference/c-api/types-and-structures.rst
index 6a9c4a9cf..763f985a6 100644
--- a/doc/source/reference/c-api/types-and-structures.rst
+++ b/doc/source/reference/c-api/types-and-structures.rst
@@ -79,6 +79,8 @@ PyArray_Type and PyArrayObject
    of :c:type:`NPY_AO` (deprecated) which is defined to be equivalent to
    :c:type:`PyArrayObject`. Direct access to the struct fields are
    deprecated. Use the ``PyArray_*(arr)`` form instead.
+   As of NumPy 1.20, the size of this struct is not considered part of
+   the NumPy ABI (see note at the end of the member list).
 
    .. code-block:: c
 
@@ -92,6 +94,7 @@ PyArray_Type and PyArrayObject
           PyArray_Descr *descr;
           int flags;
           PyObject *weakreflist;
+          /* version dependend private members */
       } PyArrayObject;
 
    .. c:macro:: PyObject_HEAD
@@ -173,6 +176,27 @@ PyArray_Type and PyArrayObject
        This member allows array objects to have weak references (using the
        weakref module).
 
+   .. note::
+
+      Further members are considered private and version dependend. If the size
+      of the struct is important for your code, special care must be taken.
+      A possible use-case when this is relevant is subclassing in C.
+      If your code relies on ``sizeof(PyArrayObject)`` to be constant,
+      you must add the following check at import time:
+
+      .. code-block:: c
+
+         if (sizeof(PyArrayObject) < PyArray_Type.tp_basicsize) {
+             PyErr_SetString(PyExc_ImportError,
+                "Binary incompatibility with NumPy, must recompile/update X.");
+             return NULL;
+         }
+
+      To ensure that your code does not have to be compiled for a specific
+      NumPy version, you may add a constant, leaving room for changes in NumPy.
+      A solution guaranteed to be compatible with any future NumPy version
+      requires the use of a runtime calculate offset and allocation size.
+
 
 PyArrayDescr_Type and PyArray_Descr
 -----------------------------------