Fixes gh-17467. Adds a public struct to hold memory manipulation routines PyDataMem_Handler and two new API functions PyDataMem_SetHandler to replace the current routines with the new ones, and PyDataMem_GetHandlerName to get the string name of the current routines (either globally or for a specific ndarray object). This also changes the size of the ndarray object to hold the PyDataMem_Handler active when it was created so subsequent actions on its data memory will remain consistent.

Tests and documentation are included. Along the way, I found some places in the code where the current policy is inconsistent (all data memory handling should have gone through npy_*_cache not PyDataMem_*) so even if this is rejected it might improve the cache handling.

The PyDataMem_Handler has fields to override memcpy, these are currently not implemented: memcpy in the code base is untouched. I think this PR is invasive enough as-is, if desired memcpy can be handled in a follow-up PR.


* ENH: add and use global configurable memory routines

* ENH: add tests and a way to compile c-extensions from tests

* fix allocation/free exposed by tests

* DOC: document the new APIs (and some old ones too)

* BUG: return void from FREE, also some cleanup

* MAINT: changes from review

* fixes from linter

* setting ndarray->descr on 0d or scalars mess with FREE

* make scalar allocation more consistent wrt np_alloc_cache

* change formatting for sphinx

* remove memcpy variants

* update to match NEP 49

* ENH: add a python-level get_handler_name

* ENH: add core.multiarray.get_handler_name

* Allow closure-like definition of the data mem routines

* Fix incompatible pointer warnings

* Note PyDataMemAllocator and PyMemAllocatorEx differentiation

Co-authored-by: Matti Picus <matti.picus@gmail.com>

* Redefine default allocator handling

* Always allocate new arrays using the current_handler

* Search for the mem_handler name of the data owner

* Sub-comparisons don't need a local mem_handler

* Make the default_handler a valid PyDataMem_Handler

* Fix PyDataMem_SetHandler description (NEP discussion)

* Pass the allocators by reference

* Implement allocator context-locality

* Fix documentation, make PyDataMem_GetHandler return const

* remove import of setuptools==49.1.3, doesn't work on python3.10

* Fix refcount leaks

* fix function signatures in test

* Return early on PyDataMem_GetHandler error (VOID_compare)

* Add context/thread-locality tests, allow testing custom policies

* ENH: add and use global configurable memory routines

* ENH: add tests and a way to compile c-extensions from tests

* fix allocation/free exposed by tests

* DOC: document the new APIs (and some old ones too)

* BUG: return void from FREE, also some cleanup

* MAINT: changes from review

* fixes from linter

* setting ndarray->descr on 0d or scalars mess with FREE

* make scalar allocation more consistent wrt np_alloc_cache

* change formatting for sphinx

* remove memcpy variants

* update to match NEP 49

* ENH: add a python-level get_handler_name

* ENH: add core.multiarray.get_handler_name

* Allow closure-like definition of the data mem routines

* Fix incompatible pointer warnings

* Note PyDataMemAllocator and PyMemAllocatorEx differentiation

Co-authored-by: Matti Picus <matti.picus@gmail.com>

* Redefine default allocator handling

* Always allocate new arrays using the current_handler

* Search for the mem_handler name of the data owner

* Sub-comparisons don't need a local mem_handler

* Make the default_handler a valid PyDataMem_Handler

* Fix PyDataMem_SetHandler description (NEP discussion)

* Pass the allocators by reference

* remove import of setuptools==49.1.3, doesn't work on python3.10

* fix function signatures in test

* try to fix cygwin extension building

* YAPF mem_policy test

* Less empty lines, more comments (tests)

* Apply suggestions from code review (set an exception and)

Co-authored-by: Matti Picus <matti.picus@gmail.com>

* skip test on cygwin

* update API hash for changed signature

* TST: add gc.collect to make sure cycles are broken

* Implement thread-locality for PyPy

Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>

* Update numpy/core/tests/test_mem_policy.py

Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>

* fixes from review

* update circleci config

* fix test

* make the connection between OWNDATA and having a allocator handle more explicit

* improve docstring, fix flake8 for tests

* update PyDataMem_GetHandler() from review

* Implement allocator lifetime management

* update NEP and add best-effort handling of error in PyDataMem_UserFREE

* ENH: fix and test for blindly taking ownership of data

* Update doc/neps/nep-0049.rst

Co-authored-by: Elias Koromilas <elias.koromilas@gmail.com>

As numpy is Python 3 only, these import statements are now unnecessary
and don't alter runtime behavior.

pytesttester is used by every single subpackage, so making it depend on np.testing just creates cyclic dependencies that can lead to circular imports

Relates to #11457

Numpy can now be tested using the standard

    `python -c"import numpy; numpy.test()"`

construct.

This is to prepare for the switch to pytest.

* Rename `numpy/testing/nose_tools` to `numpy/testing/_private`.
* Redirect imports as needed.
* Copy `_testutils.py` from scipy to `numpy/testing/_private`.
* Rename `_testutils.py` to `_pytester.py` and remove unneeded bits.