Fixes gh-17467. Adds a public struct to hold memory manipulation routines PyDataMem_Handler and two new API functions PyDataMem_SetHandler to replace the current routines with the new ones, and PyDataMem_GetHandlerName to get the string name of the current routines (either globally or for a specific ndarray object). This also changes the size of the ndarray object to hold the PyDataMem_Handler active when it was created so subsequent actions on its data memory will remain consistent.
Tests and documentation are included. Along the way, I found some places in the code where the current policy is inconsistent (all data memory handling should have gone through npy_*_cache not PyDataMem_*) so even if this is rejected it might improve the cache handling.
The PyDataMem_Handler has fields to override memcpy, these are currently not implemented: memcpy in the code base is untouched. I think this PR is invasive enough as-is, if desired memcpy can be handled in a follow-up PR.
* ENH: add and use global configurable memory routines
* ENH: add tests and a way to compile c-extensions from tests
* fix allocation/free exposed by tests
* DOC: document the new APIs (and some old ones too)
* BUG: return void from FREE, also some cleanup
* MAINT: changes from review
* fixes from linter
* setting ndarray->descr on 0d or scalars mess with FREE
* make scalar allocation more consistent wrt np_alloc_cache
* change formatting for sphinx
* remove memcpy variants
* update to match NEP 49
* ENH: add a python-level get_handler_name
* ENH: add core.multiarray.get_handler_name
* Allow closure-like definition of the data mem routines
* Fix incompatible pointer warnings
* Note PyDataMemAllocator and PyMemAllocatorEx differentiation
Co-authored-by: Matti Picus <matti.picus@gmail.com>
* Redefine default allocator handling
* Always allocate new arrays using the current_handler
* Search for the mem_handler name of the data owner
* Sub-comparisons don't need a local mem_handler
* Make the default_handler a valid PyDataMem_Handler
* Fix PyDataMem_SetHandler description (NEP discussion)
* Pass the allocators by reference
* Implement allocator context-locality
* Fix documentation, make PyDataMem_GetHandler return const
* remove import of setuptools==49.1.3, doesn't work on python3.10
* Fix refcount leaks
* fix function signatures in test
* Return early on PyDataMem_GetHandler error (VOID_compare)
* Add context/thread-locality tests, allow testing custom policies
* ENH: add and use global configurable memory routines
* ENH: add tests and a way to compile c-extensions from tests
* fix allocation/free exposed by tests
* DOC: document the new APIs (and some old ones too)
* BUG: return void from FREE, also some cleanup
* MAINT: changes from review
* fixes from linter
* setting ndarray->descr on 0d or scalars mess with FREE
* make scalar allocation more consistent wrt np_alloc_cache
* change formatting for sphinx
* remove memcpy variants
* update to match NEP 49
* ENH: add a python-level get_handler_name
* ENH: add core.multiarray.get_handler_name
* Allow closure-like definition of the data mem routines
* Fix incompatible pointer warnings
* Note PyDataMemAllocator and PyMemAllocatorEx differentiation
Co-authored-by: Matti Picus <matti.picus@gmail.com>
* Redefine default allocator handling
* Always allocate new arrays using the current_handler
* Search for the mem_handler name of the data owner
* Sub-comparisons don't need a local mem_handler
* Make the default_handler a valid PyDataMem_Handler
* Fix PyDataMem_SetHandler description (NEP discussion)
* Pass the allocators by reference
* remove import of setuptools==49.1.3, doesn't work on python3.10
* fix function signatures in test
* try to fix cygwin extension building
* YAPF mem_policy test
* Less empty lines, more comments (tests)
* Apply suggestions from code review (set an exception and)
Co-authored-by: Matti Picus <matti.picus@gmail.com>
* skip test on cygwin
* update API hash for changed signature
* TST: add gc.collect to make sure cycles are broken
* Implement thread-locality for PyPy
Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>
* Update numpy/core/tests/test_mem_policy.py
Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>
* fixes from review
* update circleci config
* fix test
* make the connection between OWNDATA and having a allocator handle more explicit
* improve docstring, fix flake8 for tests
* update PyDataMem_GetHandler() from review
* Implement allocator lifetime management
* update NEP and add best-effort handling of error in PyDataMem_UserFREE
* ENH: fix and test for blindly taking ownership of data
* Update doc/neps/nep-0049.rst
Co-authored-by: Elias Koromilas <elias.koromilas@gmail.com>