From 3d54cbf6d95967875507fe3b70d01414dd6455d9 Mon Sep 17 00:00:00 2001 From: Stefano Miccoli Date: Thu, 6 Jan 2022 19:24:43 +0100 Subject: DOC: explicitly define numpy.datetime64 semantics Explicity state datetime64 semantics: - adoption of proleptic Gregorian Calendar - Astronomical year numbering - 86400 s constant length of day. Add some exmples of shortcomings of the above semnatics with ref. to UTC timescale (missing leap seconds) and UT timescale (variable legth of day). Closes GH-20675 --- doc/source/reference/arrays.datetime.rst | 129 ++++++++++++++++++++++++++----- 1 file changed, 111 insertions(+), 18 deletions(-) (limited to 'doc/source/reference') diff --git a/doc/source/reference/arrays.datetime.rst b/doc/source/reference/arrays.datetime.rst index 63c93821b..76539c24e 100644 --- a/doc/source/reference/arrays.datetime.rst +++ b/doc/source/reference/arrays.datetime.rst @@ -9,23 +9,51 @@ Datetimes and Timedeltas .. versionadded:: 1.7.0 Starting in NumPy 1.7, there are core array data types which natively -support datetime functionality. The data type is called "datetime64", -so named because "datetime" is already taken by the datetime library -included in Python. +support datetime functionality. The data type is called :class:`datetime64`, +so named because :class:`~datetime.datetime` is already taken by the Python standard library. + +Datetime64 Conventions and Assumptions +====================================== + +Similar to the Python `~datetime.date` class, dates are expressed in the current +Gregorian Calendar, indefinitely extended both in the future and in the past. +[#]_ Contrary to Python `~datetime.date`, which supports only years in the 1 AD — 9999 +AD range, `datetime64` allows also for dates BC; years BC follow the `Astronomical +year numbering `_ +convention, i.e. year 2 BC is numbered −1, year 1 BC is numbered 0, year 1 AD is +numbered 1. + +Time instants, say 16:23:32.234, are represented counting hours, minutes, +seconds and fractions from midnight: i.e. 00:00:00.000 is midnight, 12:00:00.000 +is noon, etc. Each calendar day has exactly 86400 seconds. This is a "naive" +time, with no explicit notion of timezones or specific time scales (UT1, UTC, TAI, +etc.). [#]_ + +.. [#] The calendar obtained by extending the Gregorian calendar before its + official adoption on Oct. 15, 1582 is called `Proleptic Gregorian Calendar + `_ + +.. [#] The assumption of 86400 seconds per calendar day is not valid for UTC, + the present day civil time scale. In fact due to the presence of + `leap seconds `_ on rare occasions + a day may be 86401 or 86399 seconds long. On the contrary the 86400s day + assumption holds for the TAI timescale. An explicit support for TAI and + TAI to UTC conversion, accounting for leap seconds, is proposed but not + yet implemented. See also the `shortcomings`_ section below. Basic Datetimes =============== -The most basic way to create datetimes is from strings in ISO 8601 date -or datetime format. It is also possible to create datetimes from an integer by +The most basic way to create datetimes is from strings in ISO 8601 date +or datetime format. It is also possible to create datetimes from an integer by offset relative to the Unix epoch (00:00:00 UTC on 1 January 1970). -The unit for internal storage is automatically selected from the +The unit for internal storage is automatically selected from the form of the string, and can be either a :ref:`date unit ` or a :ref:`time unit `. The date units are years ('Y'), months ('M'), weeks ('W'), and days ('D'), while the time units are hours ('h'), minutes ('m'), seconds ('s'), milliseconds ('ms'), and -some additional SI-prefix seconds-based units. The datetime64 data type +some additional SI-prefix seconds-based units. The `datetime64` data type also accepts the string "NAT", in any combination of lowercase/uppercase letters, for a "Not A Time" value. @@ -35,11 +63,11 @@ letters, for a "Not A Time" value. >>> np.datetime64('2005-02-25') numpy.datetime64('2005-02-25') - + From an integer and a date unit, 1 year since the UNIX epoch: >>> np.datetime64(1, 'Y') - numpy.datetime64('1971') + numpy.datetime64('1971') Using months for the unit: @@ -122,19 +150,19 @@ because the moment of time is still being represented exactly. NumPy does not store timezone information. For backwards compatibility, datetime64 still parses timezone offsets, which it handles by converting to - UTC. This behaviour is deprecated and will raise an error in the + UTC±00:00 (Zulu time). This behaviour is deprecated and will raise an error in the future. Datetime and Timedelta Arithmetic ================================= -NumPy allows the subtraction of two Datetime values, an operation which +NumPy allows the subtraction of two datetime values, an operation which produces a number with a time unit. Because NumPy doesn't have a physical -quantities system in its core, the timedelta64 data type was created -to complement datetime64. The arguments for timedelta64 are a number, +quantities system in its core, the `timedelta64` data type was created +to complement `datetime64`. The arguments for `timedelta64` are a number, to represent the number of units, and a date/time unit, such as -(D)ay, (M)onth, (Y)ear, (h)ours, (m)inutes, or (s)econds. The timedelta64 +(D)ay, (M)onth, (Y)ear, (h)ours, (m)inutes, or (s)econds. The `timedelta64` data type also accepts the string "NAT" in place of the number for a "Not A Time" value. .. admonition:: Example @@ -199,9 +227,8 @@ The Datetime and Timedelta data types support a large number of time units, as well as generic units which can be coerced into any of the other units based on input data. -Datetimes are always stored based on POSIX time (though having a TAI -mode which allows for accounting of leap-seconds is proposed), with -an epoch of 1970-01-01T00:00Z. This means the supported dates are +Datetimes are always stored with +an epoch of 1970-01-01T00:00. This means the supported dates are always a symmetric interval around the epoch, called "time span" in the table below. @@ -328,7 +355,7 @@ in an optimized form. np.is_busday(): ``````````````` -To test a datetime64 value to see if it is a valid day, use :func:`is_busday`. +To test a `datetime64` value to see if it is a valid day, use :func:`is_busday`. .. admonition:: Example @@ -384,3 +411,69 @@ Some examples:: weekmask = "Mon Tue Wed Thu Fri" # any amount of whitespace is allowed; abbreviations are case-sensitive. weekmask = "MonTue Wed Thu\tFri" + + +.. _shortcomings: + +Datetime64 shortcomings +======================= + +The assumption that all days are exactly 86400 seconds long makes `datetime64` +largely compatible with Python `datetime` and "POSIX time" semantics; therefore +they all share the same well known shortcomings with respect to the UTC +timescale and historical time determination. A brief non exhaustive summary is +given below. + +- It is impossible to parse valid UTC timestamps occurring during a positive + leap second. + + .. admonition:: Example + + "2016-12-31 23:59:60 UTC" was a leap second, therefore "2016-12-31 + 23:59:60.450 UTC" is a valid timestamp which is not parseable by + `datetime64`: + + >>> np.datetime64("2016-12-31 23:59:60.450") + Traceback (most recent call last): + File "", line 1, in + ValueError: Seconds out of range in datetime string "2016-12-31 23:59:60.450" + +- Timedelta64 computations between two UTC dates can be wrong by an integer + number of SI seconds. + + .. admonition:: Example + + Compute the number of SI seconds between "2021-01-01 12:56:23.423 UTC" and + "2001-01-01 00:00:00.000 UTC": + + >>> ( + ... np.datetime64("2021-01-01 12:56:23.423") + ... - np.datetime64("2001-01-01") + ... ) / np.timedelta64(1, "s") + 631198583.423 + + however correct answer is `631198588.423` SI seconds because there were 5 + leap seconds between 2001 and 2021. + +- Timedelta64 computations for dates in the past do not return SI seconds, as + one would expect. + + .. admonition:: Example + + Compute the number of seconds between "000-01-01 UT" and "1600-01-01 UT", + where UT is `universal time + `_: + + >>> a = np.datetime64("0000-01-01", "us") + >>> b = np.datetime64("1600-01-01", "us") + >>> b - a + numpy.timedelta64(50491123200000000,'us') + + The computed results, `50491123200` seconds, is obtained as the elapsed + number of days (`584388`) times `86400` seconds; this is the number of + seconds of a clock in sync with earth rotation. The exact value in SI + seconds can only be estimated, e.g using data published in `Measurement of + the Earth's rotation: 720 BC to AD 2015, 2016, Royal Society's Proceedings + A 472, by Stephenson et.al. `_. A + sensible estimate is `50491112870 ± 90` seconds, with a difference of 10330 + seconds. -- cgit v1.2.1 From 0457cc7da06d98cc818a0ae1d3cc98ea93a1893d Mon Sep 17 00:00:00 2001 From: Sebastian Berg Date: Fri, 11 Feb 2022 14:35:19 -0600 Subject: MAINT: Remove the RELAXED_STRIDES_CHECKING env variable An error is for now raised in `setup.py` if this is set, eventually we should just delete that. --- doc/source/reference/arrays.ndarray.rst | 17 +++-------------- doc/source/reference/global_state.rst | 18 +++++++----------- 2 files changed, 10 insertions(+), 25 deletions(-) (limited to 'doc/source/reference') diff --git a/doc/source/reference/arrays.ndarray.rst b/doc/source/reference/arrays.ndarray.rst index 66ebb66fb..985a11c88 100644 --- a/doc/source/reference/arrays.ndarray.rst +++ b/doc/source/reference/arrays.ndarray.rst @@ -161,26 +161,15 @@ An array is considered aligned if the memory offsets for all elements and the base offset itself is a multiple of `self.itemsize`. Understanding `memory-alignment` leads to better performance on most hardware. -.. note:: - - Points (1) and (2) can currently be disabled by the compile time - environmental variable ``NPY_RELAXED_STRIDES_CHECKING=0``, - which was the default before NumPy 1.10. - No users should have to do this. ``NPY_RELAXED_STRIDES_DEBUG=1`` - can be used to help find errors when incorrectly relying on the strides - in C-extension code (see below warning). - - You can check whether this option was enabled when your NumPy was - built by looking at the value of ``np.ones((10,1), - order='C').flags.f_contiguous``. If this is ``True``, then your - NumPy has relaxed strides checking enabled. - .. warning:: It does *not* generally hold that ``self.strides[-1] == self.itemsize`` for C-style contiguous arrays or ``self.strides[0] == self.itemsize`` for Fortran-style contiguous arrays is true. + ``NPY_RELAXED_STRIDES_DEBUG=1`` can be used to help find errors when + incorrectly relying on the strides in C-extension code (see below warning). + Data in new :class:`ndarrays ` is in the :term:`row-major` (C) order, unless otherwise specified, but, for example, :ref:`basic array slicing ` often produces :term:`views ` diff --git a/doc/source/reference/global_state.rst b/doc/source/reference/global_state.rst index 20874ceaa..81685ec7d 100644 --- a/doc/source/reference/global_state.rst +++ b/doc/source/reference/global_state.rst @@ -70,19 +70,15 @@ Debugging-Related Options Relaxed Strides Checking ------------------------ -The *compile-time* environment variables:: +The *compile-time* environment variable:: NPY_RELAXED_STRIDES_DEBUG=0 - NPY_RELAXED_STRIDES_CHECKING=1 - -control how NumPy reports contiguity for arrays. -The default that it is enabled and the debug mode is disabled. -This setting should always be enabled. Setting the -debug option can be interesting for testing code written -in C which iterates through arrays that may or may not be -contiguous in memory. -Most users will have no reason to change these; for details -see the :ref:`memory layout ` documentation. + +can be set to help debug code written in C which iteraters through arrays +manually. When an array is contiguous and iterated in a contiguous manner, +its ``strides`` should not be queried. This option can help find errors where +the ``strides`` are incorrectly used. +For details see the :ref:`memory layout ` documentation. Warn if no memory allocation policy when deallocating data -- cgit v1.2.1 From 0ae08d026eaf8a975156c9149e318812bb9586bb Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Sat, 22 Jan 2022 17:54:46 +0100 Subject: DEP: deprecate `numpy.distutils`, and add a migration guide --- doc/source/reference/distutils.rst | 5 ++ doc/source/reference/distutils_guide.rst | 6 ++ .../reference/distutils_status_migration.rst | 88 ++++++++++++++++++++++ doc/source/reference/index.rst | 1 + 4 files changed, 100 insertions(+) create mode 100644 doc/source/reference/distutils_status_migration.rst (limited to 'doc/source/reference') diff --git a/doc/source/reference/distutils.rst b/doc/source/reference/distutils.rst index 9db757c89..ff1ba3b0d 100644 --- a/doc/source/reference/distutils.rst +++ b/doc/source/reference/distutils.rst @@ -4,6 +4,11 @@ Packaging (:mod:`numpy.distutils`) .. module:: numpy.distutils +.. warning:: + + ``numpy.distutils`` is deprecated, and will be removed for + Python >= 3.12. For more details, see :ref:`distutils-status-migration` + NumPy provides enhanced distutils functionality to make it easier to build and install sub-packages, auto-generate code, and extension modules that use Fortran-compiled libraries. To use features of NumPy diff --git a/doc/source/reference/distutils_guide.rst b/doc/source/reference/distutils_guide.rst index 081719d16..5bb4c2878 100644 --- a/doc/source/reference/distutils_guide.rst +++ b/doc/source/reference/distutils_guide.rst @@ -3,5 +3,11 @@ NumPy Distutils - Users Guide ============================= +.. warning:: + + ``numpy.distutils`` is deprecated, and will be removed for + Python >= 3.12. For more details, see :ref:`distutils-status-migration` + + .. include:: ../../DISTUTILS.rst.txt :start-line: 6 diff --git a/doc/source/reference/distutils_status_migration.rst b/doc/source/reference/distutils_status_migration.rst new file mode 100644 index 000000000..b81e1341f --- /dev/null +++ b/doc/source/reference/distutils_status_migration.rst @@ -0,0 +1,88 @@ +.. _distutils-status-migration: + +Status of ``numpy.distutils`` and migration advice +================================================== + +`numpy.distutils` has been deprecated in NumPy ``1.23.0``. It will be removed +for Python 3.12; for Python <= 3.11 it will not be removed until 2 years after +the Python 3.12 release (Oct 2025). + + +Migration advice +---------------- + +It is **not necessary** to migrate immediately - the release date for Python 3.12 +is October 2023, so you have a good amount of time left. It may be beneficial +to wait with migrating until there are examples from other projects to follow +(see below). + +There are several build systems which are good options to migrate to. Assuming +you have compiled code in your package (if not, use Flit_) and you want to be +using a well-designed, modern and reliable build system, your two best options +are: + +1. Meson_ +2. CMake_ (or scikit-build_ as an interface to CMake) + +If you have modest needs and have been happy with ``numpy.distutils`` so far, +you can also consider switching to ``setuptools``. Note that most functionality +of ``numpy.disutils`` is unlikely to be ported to ``setuptools``. The likely +exception is nested ``setup.py`` files, but this is not yet done (help with +this is very welcome!). + + +Moving to Meson +``````````````` + +SciPy is moving to Meson for its 1.9.0 release, planned for July 2022. During +this process, any remaining issues with Meson's Python support and achieving +feature parity with ``numpy.distutils`` will be resolved. *Note: parity means a +large superset, but right now some BLAS/LAPACK support is missing and there are +a few open issues related to Cython.* SciPy uses almost all functionality that +``numpy.distutils`` offers, so if SciPy has successfully made a release with +Meson as the build system, there should be no blockers left to migrate, and +SciPy will be a good reference for other packages who are migrating. +For more details about the SciPy migration, see: + +- `RFC: switch to Meson as a build system `__ +- `Tracking issue for Meson support `__ + +NumPy itself will very likely migrate to Meson as well, once the SciPy +migration is done. + + +Moving to CMake / scikit-build +`````````````````````````````` + +See the `scikit-build documentation `__ +for how to use scikit-build. Please note that as of Jan 2022, scikit-build +still relies on setuptools, so it's probably not quite ready yet for a +post-distutils world. For more details on this, see +`this blog post by Henry Schreiner `__. + + +Interaction of ``numpy.disutils`` with ``setuptools`` +----------------------------------------------------- + +It is recommended to use ``setuptools < 60.0``. Newer versions may work, but +are not guaranteed to. The reason for this is that ``setuptools`` 60.0 enabled +a vendored copy of ``distutils``, including backwards incompatible changes that +affect some functionality in ``numpy.distutils``. + +If you are using only simple Cython or C extensions with minimal use of +``numpy.distutils`` functionality beyond nested ``setup.py`` files (its most +popular feature, see :class:`Configuration `), +then latest ``setuptools`` is likely to continue working. In case of problems, +you can also try ``SETUPTOOLS_USE_DISTUTILS=stdlib`` to avoid the backwards +incompatible changes in ``setuptools``. + +Whatever you do, it is recommended to put an upper bound on your ``setuptools`` +build requirement in ``pyproject.toml`` to avoid future breakage - see +:ref:`for-downstream-package-authors`. + + +.. _Flit: https://flit.readthedocs.io +.. _CMake: https://cmake.org/ +.. _Meson: https://mesonbuild.com/ +.. _scikit-build: https://scikit-build.readthedocs.io/ + diff --git a/doc/source/reference/index.rst b/doc/source/reference/index.rst index 00c929d66..66bb45d6a 100644 --- a/doc/source/reference/index.rst +++ b/doc/source/reference/index.rst @@ -25,6 +25,7 @@ For learning how to use NumPy, see the :ref:`complete documentation Date: Tue, 15 Feb 2022 15:58:28 +0100 Subject: DOC: add section on moving to Setuptools [skip azp] [skip actions] Also addresses review comments. --- .../reference/distutils_status_migration.rst | 69 +++++++++++++++++++--- 1 file changed, 61 insertions(+), 8 deletions(-) (limited to 'doc/source/reference') diff --git a/doc/source/reference/distutils_status_migration.rst b/doc/source/reference/distutils_status_migration.rst index b81e1341f..3bcf02d60 100644 --- a/doc/source/reference/distutils_status_migration.rst +++ b/doc/source/reference/distutils_status_migration.rst @@ -8,6 +8,12 @@ for Python 3.12; for Python <= 3.11 it will not be removed until 2 years after the Python 3.12 release (Oct 2025). +.. warning:: + + ``numpy.distutils`` is only tested with ``setuptools < 60.0``, newer + versions may break. See :ref:`numpy-setuptools-interaction` for details. + + Migration advice ---------------- @@ -17,16 +23,17 @@ to wait with migrating until there are examples from other projects to follow (see below). There are several build systems which are good options to migrate to. Assuming -you have compiled code in your package (if not, use Flit_) and you want to be -using a well-designed, modern and reliable build system, your two best options -are: +you have compiled code in your package (if not, we recommend using Flit_) and +you want to be using a well-designed, modern and reliable build system, we +recommend: 1. Meson_ 2. CMake_ (or scikit-build_ as an interface to CMake) -If you have modest needs and have been happy with ``numpy.distutils`` so far, -you can also consider switching to ``setuptools``. Note that most functionality -of ``numpy.disutils`` is unlikely to be ported to ``setuptools``. The likely +If you have modest needs (only simple Cython/C extensions, and perhaps nested +``setup.py`` files) and have been happy with ``numpy.distutils`` so far, you +can also consider switching to ``setuptools``. Note that most functionality of +``numpy.disutils`` is unlikely to be ported to ``setuptools``. The likely exception is nested ``setup.py`` files, but this is not yet done (help with this is very welcome!). @@ -55,12 +62,58 @@ Moving to CMake / scikit-build `````````````````````````````` See the `scikit-build documentation `__ -for how to use scikit-build. Please note that as of Jan 2022, scikit-build +for how to use scikit-build. Please note that as of Feb 2022, scikit-build still relies on setuptools, so it's probably not quite ready yet for a -post-distutils world. For more details on this, see +post-distutils world. How quickly this changes depends on funding, the current +(Feb 2022) estimate is that if funding arrives then a viable ``numpy.distutils`` +replacement will be ready at the end of 2022, and a very polished replacement +mid-2023. For more details on this, see `this blog post by Henry Schreiner `__. +Moving to ``setuptools`` +```````````````````````` + +For projects that only use ``numpy.distutils`` for historical reasons, and do +not actually use features beyond those that ``setuptools`` also supports, +moving to ``setuptools`` is likely the solution which costs the least effort. +To assess that, there are the ``numpy.distutils`` features that are *not* +present in ``setuptools``: + +- Nested ``setup.py`` files +- Fortran build support +- BLAS/LAPACK library support (OpenBLAS, MKL, ATLAS, Netlib LAPACK/BLAS, BLIS, 64-bit ILP interface, etc.) +- Support for a few other scientific libraries, like FFTW and UMFPACK +- Better MinGW support +- Per-compiler build flag customization (e.g. `-O3` and `SSE2` flags are default) +- a simple user build config system, see [site.cfg.example](https://github.com/numpy/numpy/blob/master/site.cfg.example) +- SIMD intrinsics support + +The most widely used feature is nested ``setup.py`` files. This feature will +likely be ported to ``setuptools`` (see +`gh-18588 `__ for status). +Projects only using that feature could move to ``setuptools`` after that is +done. In case a project uses only a couple of ``setup.py`` files, it also could +make sense to simply aggregate all the content of those files into a single +``setup.py`` file and then move to ``setuptools``. This involves dropping all +``Configuration`` instances, and using ``Extension`` instead. E.g.,:: + + from distutils.core import setup + from distutils.extension import Extension + setup(name='foobar', + version='1.0', + ext_modules=[ + Extension('foopkg.foo', ['foo.c']), + Extension('barpkg.bar', ['bar.c']), + ], + ) + +For more details, see the +`setuptools documentation `__ + + +.. _numpy-setuptools-interaction: + Interaction of ``numpy.disutils`` with ``setuptools`` ----------------------------------------------------- -- cgit v1.2.1 From ec8e9e78e9a47242f8fc739e615564aa65883fdd Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Wed, 16 Feb 2022 14:57:02 +0100 Subject: MAINT: fix failure due to importing warnings in `distutils/__init__.py` --- doc/source/reference/distutils_status_migration.rst | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) (limited to 'doc/source/reference') diff --git a/doc/source/reference/distutils_status_migration.rst b/doc/source/reference/distutils_status_migration.rst index 3bcf02d60..9ef5f7232 100644 --- a/doc/source/reference/distutils_status_migration.rst +++ b/doc/source/reference/distutils_status_migration.rst @@ -18,9 +18,8 @@ Migration advice ---------------- It is **not necessary** to migrate immediately - the release date for Python 3.12 -is October 2023, so you have a good amount of time left. It may be beneficial -to wait with migrating until there are examples from other projects to follow -(see below). +is October 2023. It may be beneficial to wait with migrating until there are +examples from other projects to follow (see below). There are several build systems which are good options to migrate to. Assuming you have compiled code in your package (if not, we recommend using Flit_) and @@ -33,9 +32,7 @@ recommend: If you have modest needs (only simple Cython/C extensions, and perhaps nested ``setup.py`` files) and have been happy with ``numpy.distutils`` so far, you can also consider switching to ``setuptools``. Note that most functionality of -``numpy.disutils`` is unlikely to be ported to ``setuptools``. The likely -exception is nested ``setup.py`` files, but this is not yet done (help with -this is very welcome!). +``numpy.disutils`` is unlikely to be ported to ``setuptools``. Moving to Meson -- cgit v1.2.1