diff options
Diffstat (limited to 'Doc')
127 files changed, 3410 insertions, 1415 deletions
diff --git a/Doc/Makefile b/Doc/Makefile index f653d70674..f113dd0653 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -202,6 +202,7 @@ dist: check: $(PYTHON) tools/rstlint.py -i tools -i $(VENVDIR) -i README.rst + $(PYTHON) tools/rstlint.py ../Misc/NEWS.d/next/ serve: $(PYTHON) ../Tools/scripts/serve.py build/html @@ -215,12 +216,11 @@ serve: # for development releases: always build autobuild-dev: - make dist SPHINXOPTS='$(SPHINXOPTS) -Ea -A daily=1 -A switchers=1' - -make suspicious + make dist SPHINXOPTS='$(SPHINXOPTS) -Ea -A daily=1' # for quick rebuilds (HTML only) autobuild-dev-html: - make html SPHINXOPTS='$(SPHINXOPTS) -Ea -A daily=1 -A switchers=1' + make html SPHINXOPTS='$(SPHINXOPTS) -Ea -A daily=1' # for stable releases: only build if not in pre-release stage (alpha, beta) # release candidate downloads are okay, since the stable tree can be in that stage diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index bdaae44e24..a91b3c7c95 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -482,7 +482,8 @@ API Functions *min* and no more than *max*; *min* and *max* may be equal. Additional arguments must be passed to the function, each of which should be a pointer to a :c:type:`PyObject*` variable; these will be filled in with the values from - *args*; they will contain borrowed references. The variables which correspond + *args*; they will contain :term:`borrowed references <borrowed reference>`. + The variables which correspond to optional parameters not given by *args* will not be filled in; these should be initialized by the caller. This function returns true on success and false if *args* is not a tuple or contains the wrong number of elements; an exception diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 3ce689203a..5736b83f21 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -376,6 +376,12 @@ Process-wide parameters The returned string points into static storage; the caller should not modify its value. + This function should not be called before :c:func:`Py_Initialize`, otherwise + it returns ``NULL``. + + .. versionchanged:: 3.10 + It now returns ``NULL`` if called before :c:func:`Py_Initialize`. + .. c:function:: wchar_t* Py_GetPrefix() @@ -389,6 +395,12 @@ Process-wide parameters script at build time. The value is available to Python code as ``sys.prefix``. It is only useful on Unix. See also the next function. + This function should not be called before :c:func:`Py_Initialize`, otherwise + it returns ``NULL``. + + .. versionchanged:: 3.10 + It now returns ``NULL`` if called before :c:func:`Py_Initialize`. + .. c:function:: wchar_t* Py_GetExecPrefix() @@ -424,6 +436,12 @@ Process-wide parameters while having :file:`/usr/local/plat` be a different filesystem for each platform. + This function should not be called before :c:func:`Py_Initialize`, otherwise + it returns ``NULL``. + + .. versionchanged:: 3.10 + It now returns ``NULL`` if called before :c:func:`Py_Initialize`. + .. c:function:: wchar_t* Py_GetProgramFullPath() @@ -437,6 +455,12 @@ Process-wide parameters static storage; the caller should not modify its value. The value is available to Python code as ``sys.executable``. + This function should not be called before :c:func:`Py_Initialize`, otherwise + it returns ``NULL``. + + .. versionchanged:: 3.10 + It now returns ``NULL`` if called before :c:func:`Py_Initialize`. + .. c:function:: wchar_t* Py_GetPath() @@ -455,8 +479,14 @@ Process-wide parameters can be (and usually is) modified later to change the search path for loading modules. + This function should not be called before :c:func:`Py_Initialize`, otherwise + it returns ``NULL``. + .. XXX should give the exact rules + .. versionchanged:: 3.10 + It now returns ``NULL`` if called before :c:func:`Py_Initialize`. + .. c:function:: void Py_SetPath(const wchar_t *) @@ -638,6 +668,12 @@ Process-wide parameters :c:func:`Py_SetPythonHome`, or the value of the :envvar:`PYTHONHOME` environment variable if it is set. + This function should not be called before :c:func:`Py_Initialize`, otherwise + it returns ``NULL``. + + .. versionchanged:: 3.10 + It now returns ``NULL`` if called before :c:func:`Py_Initialize`. + .. _threads: @@ -1077,7 +1113,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`. Get the current frame of the Python thread state *tstate*. - Return a strong reference. Return ``NULL`` if no frame is currently + Return a :term:`strong reference`. Return ``NULL`` if no frame is currently executing. See also :c:func:`PyEval_GetFrame`. diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index dad1f90bea..db7c1f4376 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -8,46 +8,24 @@ Python Initialization Configuration .. versionadded:: 3.8 -Structures: - -* :c:type:`PyConfig` -* :c:type:`PyPreConfig` -* :c:type:`PyStatus` -* :c:type:`PyWideStringList` - -Functions: - -* :c:func:`PyConfig_Clear` -* :c:func:`PyConfig_InitIsolatedConfig` -* :c:func:`PyConfig_InitPythonConfig` -* :c:func:`PyConfig_Read` -* :c:func:`PyConfig_SetArgv` -* :c:func:`PyConfig_SetBytesArgv` -* :c:func:`PyConfig_SetBytesString` -* :c:func:`PyConfig_SetString` -* :c:func:`PyConfig_SetWideStringList` -* :c:func:`PyPreConfig_InitIsolatedConfig` -* :c:func:`PyPreConfig_InitPythonConfig` -* :c:func:`PyStatus_Error` -* :c:func:`PyStatus_Exception` -* :c:func:`PyStatus_Exit` -* :c:func:`PyStatus_IsError` -* :c:func:`PyStatus_IsExit` -* :c:func:`PyStatus_NoMemory` -* :c:func:`PyStatus_Ok` -* :c:func:`PyWideStringList_Append` -* :c:func:`PyWideStringList_Insert` -* :c:func:`Py_ExitStatusException` -* :c:func:`Py_InitializeFromConfig` -* :c:func:`Py_PreInitialize` -* :c:func:`Py_PreInitializeFromArgs` -* :c:func:`Py_PreInitializeFromBytesArgs` -* :c:func:`Py_RunMain` -* :c:func:`Py_GetArgcArgv` - -The preconfiguration (``PyPreConfig`` type) is stored in -``_PyRuntime.preconfig`` and the configuration (``PyConfig`` type) is stored in -``PyInterpreterState.config``. +Python can be initialized with :c:func:`Py_InitializeFromConfig` and the +:c:type:`PyConfig` structure. It can be preinitialized with +:c:func:`Py_PreInitialize` and the :c:type:`PyPreConfig` structure. + +There are two kinds of configuration: + +* The :ref:`Python Configuration <init-python-config>` can be used to build a + customized Python which behaves as the regular Python. For example, + environments variables and command line arguments are used to configure + Python. + +* The :ref:`Isolated Configuration <init-isolated-conf>` can be used to embed + Python into an application. It isolates Python from the system. For example, + environments variables are ignored, the LC_CTYPE locale is left unchanged and + no signal handler is registred. + +The :c:func:`Py_RunMain` function can be used to write a customized Python +program. See also :ref:`Initialization, Finalization, and Threads <initialization>`. @@ -55,8 +33,47 @@ See also :ref:`Initialization, Finalization, and Threads <initialization>`. :pep:`587` "Python Initialization Configuration". +Example +======= + +Example of customized Python always running in isolated mode:: + + int main(int argc, char **argv) + { + PyStatus status; + + PyConfig config; + PyConfig_InitPythonConfig(&config); + config.isolated = 1; + + /* Decode command line arguments. + Implicitly preinitialize Python (in isolated mode). */ + status = PyConfig_SetBytesArgv(&config, argc, argv); + if (PyStatus_Exception(status)) { + goto exception; + } + + status = Py_InitializeFromConfig(&config); + if (PyStatus_Exception(status)) { + goto exception; + } + PyConfig_Clear(&config); + + return Py_RunMain(); + + exception: + PyConfig_Clear(&config); + if (PyStatus_IsExit(status)) { + return status.exitcode; + } + /* Display the error message and exit the process with + non-zero exit code */ + Py_ExitStatusException(status); + } + + PyWideStringList ----------------- +================ .. c:type:: PyWideStringList @@ -95,7 +112,7 @@ PyWideStringList List items. PyStatus --------- +======== .. c:type:: PyStatus @@ -128,6 +145,8 @@ PyStatus Initialization error with a message. + *err_msg* must not be ``NULL``. + .. c:function:: PyStatus PyStatus_NoMemory(void) Memory allocation failure (out of memory). @@ -185,7 +204,7 @@ Example:: PyPreConfig ------------ +=========== .. c:type:: PyPreConfig @@ -315,7 +334,7 @@ PyPreConfig .. _c-preinit: Preinitialize Python with PyPreConfig -------------------------------------- +===================================== The preinitialization of Python: @@ -324,12 +343,17 @@ The preinitialization of Python: * Set the :ref:`Python UTF-8 Mode <utf8-mode>` (:c:member:`PyPreConfig.utf8_mode`) +The current preconfiguration (``PyPreConfig`` type) is stored in +``_PyRuntime.preconfig``. + Functions to preinitialize Python: .. c:function:: PyStatus Py_PreInitialize(const PyPreConfig *preconfig) Preinitialize Python from *preconfig* preconfiguration. + *preconfig* must not be ``NULL``. + .. c:function:: PyStatus Py_PreInitializeFromBytesArgs(const PyPreConfig *preconfig, int argc, char * const *argv) Preinitialize Python from *preconfig* preconfiguration. @@ -337,6 +361,8 @@ Functions to preinitialize Python: Parse *argv* command line arguments (bytes strings) if :c:member:`~PyPreConfig.parse_argv` of *preconfig* is non-zero. + *preconfig* must not be ``NULL``. + .. c:function:: PyStatus Py_PreInitializeFromArgs(const PyPreConfig *preconfig, int argc, wchar_t * const * argv) Preinitialize Python from *preconfig* preconfiguration. @@ -344,6 +370,8 @@ Functions to preinitialize Python: Parse *argv* command line arguments (wide strings) if :c:member:`~PyPreConfig.parse_argv` of *preconfig* is non-zero. + *preconfig* must not be ``NULL``. + The caller is responsible to handle exceptions (error or exit) using :c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`. @@ -386,7 +414,7 @@ the :ref:`Python UTF-8 Mode <utf8-mode>`:: PyConfig --------- +======== .. c:type:: PyConfig @@ -447,8 +475,20 @@ PyConfig Fields which are already initialized are left unchanged. + The :c:func:`PyConfig_Read` function only parses + :c:member:`PyConfig.argv` arguments once: :c:member:`PyConfig.parse_argv` + is set to ``2`` after arguments are parsed. Since Python arguments are + strippped from :c:member:`PyConfig.argv`, parsing arguments twice would + parse the application options as Python options. + :ref:`Preinitialize Python <c-preinit>` if needed. + .. versionchanged:: 3.10 + The :c:member:`PyConfig.argv` arguments are now only parsed once, + :c:member:`PyConfig.parse_argv` is set to ``2`` after arguments are + parsed, and arguments are only parsed if + :c:member:`PyConfig.parse_argv` equals ``1``. + .. c:function:: void PyConfig_Clear(PyConfig *config) Release configuration memory. @@ -496,7 +536,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` output. + Part of the :ref:`Python Path Configuration <init-path-config>` output. .. c:member:: wchar_t* base_executable @@ -508,7 +548,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` output. + Part of the :ref:`Python Path Configuration <init-path-config>` output. .. c:member:: wchar_t* base_prefix @@ -516,7 +556,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` output. + Part of the :ref:`Python Path Configuration <init-path-config>` output. .. c:member:: int buffered_stdio @@ -598,7 +638,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` output. + Part of the :ref:`Python Path Configuration <init-path-config>` output. .. c:member:: wchar_t* executable @@ -607,7 +647,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` output. + Part of the :ref:`Python Path Configuration <init-path-config>` output. .. c:member:: int faulthandler @@ -690,7 +730,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` input. + Part of the :ref:`Python Path Configuration <init-path-config>` input. .. c:member:: int import_time @@ -781,7 +821,7 @@ PyConfig Default: value of the ``PLATLIBDIR`` macro which is set at configure time by ``--with-platlibdir`` (default: ``"lib"``). - Part of the :ref:`Path Configuration <init-path-config>` input. + Part of the :ref:`Python Path Configuration <init-path-config>` input. .. versionadded:: 3.9 @@ -794,7 +834,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` input. + Part of the :ref:`Python Path Configuration <init-path-config>` input. .. c:member:: PyWideStringList module_search_paths .. c:member:: int module_search_paths_set @@ -802,14 +842,14 @@ PyConfig Module search paths: :data:`sys.path`. If :c:member:`~PyConfig.module_search_paths_set` is equal to 0, the - function calculating the :ref:`Path Configuration <init-path-config>` + function calculating the :ref:`Python Path Configuration <init-path-config>` overrides the :c:member:`~PyConfig.module_search_paths` and sets :c:member:`~PyConfig.module_search_paths_set` to ``1``. Default: empty list (``module_search_paths``) and ``0`` (``module_search_paths_set``). - Part of the :ref:`Path Configuration <init-path-config>` output. + Part of the :ref:`Python Path Configuration <init-path-config>` output. .. c:member:: int optimization_level @@ -831,7 +871,7 @@ PyConfig If :c:member:`~PyConfig.orig_argv` list is empty and :c:member:`~PyConfig.argv` is not a list only containing an empty - string, :c:func:`PyConfig_Read()` copies :c:member:`~PyConfig.argv` into + string, :c:func:`PyConfig_Read` copies :c:member:`~PyConfig.argv` into :c:member:`~PyConfig.orig_argv` before modifying :c:member:`~PyConfig.argv` (if :c:member:`~PyConfig.parse_argv` is non-zero). @@ -847,12 +887,22 @@ PyConfig Parse command line arguments? - If non-zero, parse :c:member:`~PyConfig.argv` the same way the regular + If equals to ``1``, parse :c:member:`~PyConfig.argv` the same way the regular Python parses :ref:`command line arguments <using-on-cmdline>`, and strip Python arguments from :c:member:`~PyConfig.argv`. + The :c:func:`PyConfig_Read` function only parses + :c:member:`PyConfig.argv` arguments once: :c:member:`PyConfig.parse_argv` + is set to ``2`` after arguments are parsed. Since Python arguments are + strippped from :c:member:`PyConfig.argv`, parsing arguments twice would + parse the application options as Python options. + Default: ``1`` in Python mode, ``0`` in isolated mode. + .. versionchanged:: 3.10 + The :c:member:`PyConfig.argv` arguments are now only parsed if + :c:member:`PyConfig.parse_argv` equals to ``1``. + .. c:member:: int parser_debug Parser debug mode. If greater than 0, turn on parser debugging output (for expert only, depending @@ -865,7 +915,7 @@ PyConfig .. c:member:: int pathconfig_warnings - On Unix, if non-zero, calculating the :ref:`Path Configuration + On Unix, if non-zero, calculating the :ref:`Python Path Configuration <init-path-config>` can log warnings into ``stderr``. If equals to 0, suppress these warnings. @@ -873,7 +923,7 @@ PyConfig Default: ``1`` in Python mode, ``0`` in isolated mode. - Part of the :ref:`Path Configuration <init-path-config>` input. + Part of the :ref:`Python Path Configuration <init-path-config>` input. .. c:member:: wchar_t* prefix @@ -882,7 +932,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` output. + Part of the :ref:`Python Path Configuration <init-path-config>` output. .. c:member:: wchar_t* program_name @@ -900,7 +950,7 @@ PyConfig Default: ``NULL``. - Part of the :ref:`Path Configuration <init-path-config>` input. + Part of the :ref:`Python Path Configuration <init-path-config>` input. .. c:member:: wchar_t* pycache_prefix @@ -1106,7 +1156,7 @@ the :option:`-X` command line option. Initialization with PyConfig ----------------------------- +============================ Function to initialize Python: @@ -1121,6 +1171,9 @@ If :c:func:`PyImport_FrozenModules`, :c:func:`PyImport_AppendInittab` or :c:func:`PyImport_ExtendInittab` are used, they must be set or called after Python preinitialization and before the Python initialization. +The current configuration (``PyConfig`` type) is stored in +``PyInterpreterState.config``. + Example setting the program name:: void init_python(void) @@ -1134,17 +1187,17 @@ Example setting the program name:: status = PyConfig_SetString(&config, &config.program_name, L"/path/to/my_program"); if (PyStatus_Exception(status)) { - goto fail; + goto exception; } status = Py_InitializeFromConfig(&config); if (PyStatus_Exception(status)) { - goto fail; + goto exception; } PyConfig_Clear(&config); return; - fail: + exception: PyConfig_Clear(&config); Py_ExitStatusException(status); } @@ -1200,7 +1253,7 @@ configuration, and then override some parameters:: .. _init-isolated-conf: Isolated Configuration ----------------------- +====================== :c:func:`PyPreConfig_InitIsolatedConfig` and :c:func:`PyConfig_InitIsolatedConfig` functions create a configuration to @@ -1213,7 +1266,7 @@ and user site directory. The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left unchanged. Signal handlers are not installed. Configuration files are still used with this configuration. Set the -:ref:`Path Configuration <init-path-config>` ("output fields") to ignore these +:ref:`Python Path Configuration <init-path-config>` ("output fields") to ignore these configuration files and avoid the function computing the default path configuration. @@ -1221,7 +1274,7 @@ configuration. .. _init-python-config: Python Configuration --------------------- +==================== :c:func:`PyPreConfig_InitPythonConfig` and :c:func:`PyConfig_InitPythonConfig` functions create a configuration to build a customized Python which behaves as @@ -1235,46 +1288,11 @@ and :ref:`Python UTF-8 Mode <utf8-mode>` (:pep:`540`) depending on the LC_CTYPE locale, :envvar:`PYTHONUTF8` and :envvar:`PYTHONCOERCECLOCALE` environment variables. -Example of customized Python always running in isolated mode:: - - int main(int argc, char **argv) - { - PyStatus status; - - PyConfig config; - PyConfig_InitPythonConfig(&config); - config.isolated = 1; - - /* Decode command line arguments. - Implicitly preinitialize Python (in isolated mode). */ - status = PyConfig_SetBytesArgv(&config, argc, argv); - if (PyStatus_Exception(status)) { - goto fail; - } - - status = Py_InitializeFromConfig(&config); - if (PyStatus_Exception(status)) { - goto fail; - } - PyConfig_Clear(&config); - - return Py_RunMain(); - - fail: - PyConfig_Clear(&config); - if (PyStatus_IsExit(status)) { - return status.exitcode; - } - /* Display the error message and exit the process with - non-zero exit code */ - Py_ExitStatusException(status); - } - .. _init-path-config: -Path Configuration ------------------- +Python Path Configuration +========================= :c:type:`PyConfig` contains multiple fields for the path configuration: @@ -1354,7 +1372,7 @@ The ``__PYVENV_LAUNCHER__`` environment variable is used to set Py_RunMain() ------------- +============ .. c:function:: int Py_RunMain(void) @@ -1374,7 +1392,7 @@ customized Python always running in isolated mode using Py_GetArgcArgv() ----------------- +================ .. c:function:: void Py_GetArgcArgv(int *argc, wchar_t ***argv) @@ -1384,7 +1402,7 @@ Py_GetArgcArgv() Multi-Phase Initialization Private Provisional API --------------------------------------------------- +================================================== This section is a private provisional API introducing multi-phase initialization, the core feature of the :pep:`432`: diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst index 7ca8693afa..bae5ce11b7 100644 --- a/Doc/c-api/intro.rst +++ b/Doc/c-api/intro.rst @@ -326,7 +326,7 @@ when it's no longer needed---or passing on this responsibility (usually to its caller). When a function passes ownership of a reference on to its caller, the caller is said to receive a *new* reference. When no ownership is transferred, the caller is said to *borrow* the reference. Nothing needs to be done for a -borrowed reference. +:term:`borrowed reference`. Conversely, when a calling function passes in a reference to an object, there are two possibilities: the function *steals* a reference to the object, or it diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 6e9474bfa4..41a705d9e9 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -264,7 +264,7 @@ of the following two module creation functions: instead; only use this if you are sure you need it. Before it is returned from in the initialization function, the resulting module -object is typically populated using functions like :c:func:`PyModule_AddObject`. +object is typically populated using functions like :c:func:`PyModule_AddObjectRef`. .. _multi-phase-initialization: @@ -437,26 +437,102 @@ a function called from a module execution slot (if using multi-phase initialization), can use the following functions to help initialize the module state: +.. c:function:: int PyModule_AddObjectRef(PyObject *module, const char *name, PyObject *value) + + Add an object to *module* as *name*. This is a convenience function which + can be used from the module's initialization function. + + On success, return ``0``. On error, raise an exception and return ``-1``. + + Return ``NULL`` if *value* is ``NULL``. It must be called with an exception + raised in this case. + + Example usage:: + + static int + add_spam(PyObject *module, int value) + { + PyObject *obj = PyLong_FromLong(value); + if (obj == NULL) { + return -1; + } + int res = PyModule_AddObjectRef(module, "spam", obj); + Py_DECREF(obj); + return res; + } + + The example can also be written without checking explicitly if *obj* is + ``NULL``:: + + static int + add_spam(PyObject *module, int value) + { + PyObject *obj = PyLong_FromLong(value); + int res = PyModule_AddObjectRef(module, "spam", obj); + Py_XDECREF(obj); + return res; + } + + Note that ``Py_XDECREF()`` should be used instead of ``Py_DECREF()`` in + this case, since *obj* can be ``NULL``. + + .. versionadded:: 3.10 + + .. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value) - Add an object to *module* as *name*. This is a convenience function which can - be used from the module's initialization function. This steals a reference to - *value* on success. Return ``-1`` on error, ``0`` on success. + Similar to :c:func:`PyModule_AddObjectRef`, but steals a reference to + *value* on success (if it returns ``0``). + + The new :c:func:`PyModule_AddObjectRef` function is recommended, since it is + easy to introduce reference leaks by misusing the + :c:func:`PyModule_AddObject` function. .. note:: - Unlike other functions that steal references, ``PyModule_AddObject()`` only - decrements the reference count of *value* **on success**. + Unlike other functions that steal references, ``PyModule_AddObject()`` + only decrements the reference count of *value* **on success**. This means that its return value must be checked, and calling code must - :c:func:`Py_DECREF` *value* manually on error. Example usage:: - - Py_INCREF(spam); - if (PyModule_AddObject(module, "spam", spam) < 0) { - Py_DECREF(module); - Py_DECREF(spam); - return NULL; - } + :c:func:`Py_DECREF` *value* manually on error. + + Example usage:: + + static int + add_spam(PyObject *module, int value) + { + PyObject *obj = PyLong_FromLong(value); + if (obj == NULL) { + return -1; + } + if (PyModule_AddObject(module, "spam", obj) < 0) { + Py_DECREF(obj); + return -1; + } + // PyModule_AddObject() stole a reference to obj: + // Py_DECREF(obj) is not needed here + return 0; + } + + The example can also be written without checking explicitly if *obj* is + ``NULL``:: + + static int + add_spam(PyObject *module, int value) + { + PyObject *obj = PyLong_FromLong(value); + if (PyModule_AddObject(module, "spam", obj) < 0) { + Py_XDECREF(obj); + return -1; + } + // PyModule_AddObject() stole a reference to obj: + // Py_DECREF(obj) is not needed here + return 0; + } + + Note that ``Py_XDECREF()`` should be used instead of ``Py_DECREF()`` in + this case, since *obj* can be ``NULL``. + .. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value) diff --git a/Doc/c-api/refcounting.rst b/Doc/c-api/refcounting.rst index 0df12c67f4..391907c8c2 100644 --- a/Doc/c-api/refcounting.rst +++ b/Doc/c-api/refcounting.rst @@ -13,8 +13,14 @@ objects. .. c:function:: void Py_INCREF(PyObject *o) - Increment the reference count for object *o*. The object must not be ``NULL``; if - you aren't sure that it isn't ``NULL``, use :c:func:`Py_XINCREF`. + Increment the reference count for object *o*. + + This function is usually used to convert a :term:`borrowed reference` to a + :term:`strong reference` in-place. The :c:func:`Py_NewRef` function can be + used to create a new :term:`strong reference`. + + The object must not be ``NULL``; if you aren't sure that it isn't + ``NULL``, use :c:func:`Py_XINCREF`. .. c:function:: void Py_XINCREF(PyObject *o) @@ -22,13 +28,55 @@ objects. Increment the reference count for object *o*. The object may be ``NULL``, in which case the macro has no effect. + See also :c:func:`Py_XNewRef`. + + +.. c:function:: PyObject* Py_NewRef(PyObject *o) + + Create a new :term:`strong reference` to an object: increment the reference + count of the object *o* and return the object *o*. + + When the :term:`strong reference` is no longer needed, :c:func:`Py_DECREF` + should be called on it to decrement the object reference count. + + The object *o* must not be ``NULL``; use :c:func:`Py_XNewRef` if *o* can be + ``NULL``. + + For example:: + + Py_INCREF(obj); + self->attr = obj; + + can be written as:: + + self->attr = Py_NewRef(obj); + + See also :c:func:`Py_INCREF`. + + .. versionadded:: 3.10 + + +.. c:function:: PyObject* Py_XNewRef(PyObject *o) + + Similar to :c:func:`Py_NewRef`, but the object *o* can be NULL. + + If the object *o* is ``NULL``, the function just returns ``NULL``. + + .. versionadded:: 3.10 + .. c:function:: void Py_DECREF(PyObject *o) - Decrement the reference count for object *o*. The object must not be ``NULL``; if - you aren't sure that it isn't ``NULL``, use :c:func:`Py_XDECREF`. If the reference - count reaches zero, the object's type's deallocation function (which must not be - ``NULL``) is invoked. + Decrement the reference count for object *o*. + + If the reference count reaches zero, the object's type's deallocation + function (which must not be ``NULL``) is invoked. + + This function is usually used to delete a :term:`strong reference` before + exiting its scope. + + The object must not be ``NULL``; if you aren't sure that it isn't ``NULL``, + use :c:func:`Py_XDECREF`. .. warning:: diff --git a/Doc/c-api/reflection.rst b/Doc/c-api/reflection.rst index 9207d86012..64ce4d1d0c 100644 --- a/Doc/c-api/reflection.rst +++ b/Doc/c-api/reflection.rst @@ -35,7 +35,7 @@ Reflection Get the *frame* next outer frame. - Return a strong reference, or ``NULL`` if *frame* has no outer frame. + Return a :term:`strong reference`, or ``NULL`` if *frame* has no outer frame. *frame* must not be ``NULL``. @@ -46,7 +46,7 @@ Reflection Get the *frame* code. - Return a strong reference. + Return a :term:`strong reference`. *frame* must not be ``NULL``. The result (frame code) cannot be ``NULL``. diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index a9e1c6fbcc..37072d30d5 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -66,11 +66,9 @@ the definition of all other Python objects. Get the type of the Python object *o*. - Return a borrowed reference. + Return a :term:`borrowed reference`. - .. versionchanged:: 3.10 - :c:func:`Py_TYPE()` is changed to the inline static function. - Use :c:func:`Py_SET_TYPE()` to set an object type. + The :c:func:`Py_SET_TYPE` function must be used to set an object type. .. c:function:: int Py_IS_TYPE(PyObject *o, PyTypeObject *type) @@ -108,9 +106,7 @@ the definition of all other Python objects. Get the size of the Python object *o*. - .. versionchanged:: 3.10 - :c:func:`Py_SIZE()` is changed to the inline static function. - Use :c:func:`Py_SET_SIZE()` to set an object size. + The :c:func:`Py_SET_SIZE` function must be used to set an object size. .. c:function:: void Py_SET_SIZE(PyVarObject *o, Py_ssize_t size) @@ -263,10 +259,12 @@ There are these calling conventions: of :c:type:`PyObject*` values indicating the arguments and the third parameter is the number of arguments (the length of the array). - This is not part of the :ref:`limited API <stable>`. - .. versionadded:: 3.7 + .. versionchanged:: 3.10 + + ``METH_FASTCALL`` is now part of the stable ABI. + .. data:: METH_FASTCALL | METH_KEYWORDS diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst index bf751e44ac..c0c14f480d 100644 --- a/Doc/c-api/tuple.rst +++ b/Doc/c-api/tuple.rst @@ -57,7 +57,7 @@ Tuple Objects .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos) Return the object at position *pos* in the tuple pointed to by *p*. If *pos* is - out of bounds, return ``NULL`` and set an :exc:`IndexError` exception. + negative or out of bounds, return ``NULL`` and set an :exc:`IndexError` exception. .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos) diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst index 73f26875d8..a869859dd4 100644 --- a/Doc/c-api/type.rst +++ b/Doc/c-api/type.rst @@ -105,10 +105,12 @@ Type Objects See :c:member:`PyType_Slot.slot` for possible values of the *slot* argument. - An exception is raised if *type* is not a heap type. - .. versionadded:: 3.4 + .. versionchanged:: 3.10 + :c:func:`PyType_GetSlot` can now accept all types. + Previously, it was limited to heap types. + .. c:function:: PyObject* PyType_GetModule(PyTypeObject *type) Return the module object associated with the given type when the type was @@ -152,10 +154,10 @@ The following functions and structs are used to create Creates and returns a heap type object from the *spec* (:const:`Py_TPFLAGS_HEAPTYPE`). - If *bases* is a tuple, the created heap type contains all types contained - in it as base types. - - If *bases* is ``NULL``, the *Py_tp_base* slot is used instead. + The *bases* argument can be used to specify base classes; it can either + be only one class or a tuple of classes. + If *bases* is ``NULL``, the *Py_tp_bases* slot is used instead. + If that also is ``NULL``, the *Py_tp_base* slot is used instead. If that also is ``NULL``, the new type derives from :class:`object`. The *module* argument can be used to record the module in which the new @@ -169,6 +171,11 @@ The following functions and structs are used to create .. versionadded:: 3.9 + .. versionchanged:: 3.10 + + The function now accepts a single class as the *bases* argument and + ``NULL`` as the ``tp_doc`` slot. + .. c:function:: PyObject* PyType_FromSpecWithBases(PyType_Spec *spec, PyObject *bases) Equivalent to ``PyType_FromModuleAndSpec(NULL, spec, bases)``. @@ -247,7 +254,8 @@ The following functions and structs are used to create * :c:member:`~PyBufferProcs.bf_getbuffer` * :c:member:`~PyBufferProcs.bf_releasebuffer` - Setting :c:data:`Py_tp_bases` may be problematic on some platforms. + Setting :c:data:`Py_tp_bases` or :c:data:`Py_tp_base` may be + problematic on some platforms. To avoid issues, use the *bases* argument of :py:func:`PyType_FromSpecWithBases` instead. @@ -260,4 +268,4 @@ The following functions and structs are used to create The desired value of the slot. In most cases, this is a pointer to a function. - May not be ``NULL``. + Slots other than ``Py_tp_doc`` may not be ``NULL``. diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index ddcb8ae3d0..9efe3aac2e 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -199,6 +199,8 @@ sub-slots +---------------------------------------------------------+-----------------------------------+--------------+ | :c:member:`~PyAsyncMethods.am_anext` | :c:type:`unaryfunc` | __anext__ | +---------------------------------------------------------+-----------------------------------+--------------+ + | :c:member:`~PyAsyncMethods.am_send` | :c:type:`sendfunc` | | + +---------------------------------------------------------+-----------------------------------+--------------+ | | +---------------------------------------------------------+-----------------------------------+--------------+ | :c:member:`~PyNumberMethods.nb_add` | :c:type:`binaryfunc` | __add__ | @@ -1168,6 +1170,14 @@ and :c:type:`PyType_Type` effectively act as defaults.) .. versionadded:: 3.9 + .. data:: Py_TPFLAGS_HAVE_AM_SEND + + This bit is set when the :c:member:`~PyAsyncMethods.am_send` entry is present in the + :c:member:`~PyTypeObject.tp_as_async` slot of type structure. + + .. versionadded:: 3.10 + + .. c:member:: const char* PyTypeObject.tp_doc An optional pointer to a NUL-terminated C string giving the docstring for this @@ -1213,8 +1223,9 @@ and :c:type:`PyType_Type` effectively act as defaults.) :func:`~gc.get_referents` function will include it. .. warning:: - When implementing :c:member:`~PyTypeObject.tp_traverse`, only the members - that the instance *owns* (by having strong references to them) must be + When implementing :c:member:`~PyTypeObject.tp_traverse`, only the + members that the instance *owns* (by having :term:`strong references + <strong reference>` to them) must be visited. For instance, if an object supports weak references via the :c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting the linked list (what *tp_weaklist* points to) must **not** be @@ -2303,6 +2314,7 @@ Async Object Structures unaryfunc am_await; unaryfunc am_aiter; unaryfunc am_anext; + sendfunc am_send; } PyAsyncMethods; .. c:member:: unaryfunc PyAsyncMethods.am_await @@ -2336,6 +2348,15 @@ Async Object Structures Must return an :term:`awaitable` object. See :meth:`__anext__` for details. This slot may be set to ``NULL``. +.. c:member:: sendfunc PyAsyncMethods.am_send + + The signature of this function is:: + + PySendResult am_send(PyObject *self, PyObject *arg, PyObject **result); + + See :c:func:`PyIter_Send` for details. + This slot may be set to ``NULL``. + .. _slot-typedefs: @@ -2431,6 +2452,10 @@ Slot Type typedefs .. c:type:: PyObject *(*binaryfunc)(PyObject *, PyObject *) +.. c:type:: PySendResult (*sendfunc)(PyObject *, PyObject *, PyObject **) + + See :c:member:`~PyAsyncMethods.am_send`. + .. c:type:: PyObject *(*ternaryfunc)(PyObject *, PyObject *, PyObject *) .. c:type:: PyObject *(*ssizeargfunc)(PyObject *, Py_ssize_t) diff --git a/Doc/c-api/weakref.rst b/Doc/c-api/weakref.rst index e3a9bda54d..9205ca620a 100644 --- a/Doc/c-api/weakref.rst +++ b/Doc/c-api/weakref.rst @@ -57,10 +57,10 @@ as much as it can. .. note:: - This function returns a **borrowed reference** to the referenced object. + This function returns a :term:`borrowed reference` to the referenced object. This means that you should always call :c:func:`Py_INCREF` on the object - except if you know that it cannot be destroyed while you are still - using it. + except when it cannot be destroyed before the last usage of the borrowed + reference. .. c:function:: PyObject* PyWeakref_GET_OBJECT(PyObject *ref) diff --git a/Doc/conf.py b/Doc/conf.py index 079d17717f..6b88c23a44 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -15,8 +15,7 @@ sys.path.append(os.path.abspath('includes')) extensions = ['sphinx.ext.coverage', 'sphinx.ext.doctest', 'pyspecific', 'c_annotations', 'escape4chm', - 'asdl_highlight', 'peg_highlight'] - + 'asdl_highlight', 'peg_highlight', 'glossary_search'] doctest_global_setup = ''' try: diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index d01e99ca5e..8a6ee718a0 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -3007,6 +3007,9 @@ Py_GetVersion:const char*::: Py_INCREF:void::: Py_INCREF:PyObject*:o:+1: +Py_NewRef:void::: +Py_NewRef:PyObject*:o:+1: + Py_Initialize:void::: Py_IsInitialized:int::: @@ -3028,6 +3031,9 @@ Py_XDECREF:PyObject*:o:-1:if o is not NULL Py_XINCREF:void::: Py_XINCREF:PyObject*:o:+1:if o is not NULL +Py_XNewRef:void::: +Py_XNewRef:PyObject*:o:+1:if o is not NULL + _PyImport_Fini:void::: _PyObject_New:PyObject*::+1: diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat new file mode 100644 index 0000000000..c2c9c6e11e --- /dev/null +++ b/Doc/data/stable_abi.dat @@ -0,0 +1,780 @@ +# File generated by 'make regen-limited-abi' +# This is NOT an authoritative list of stable ABI symbols +PyArg_Parse +PyArg_ParseTuple +PyArg_ParseTupleAndKeywords +PyArg_UnpackTuple +PyArg_VaParse +PyArg_VaParseTupleAndKeywords +PyArg_ValidateKeywordArguments +PyBaseObject_Type +PyBool_FromLong +PyBool_Type +PyByteArrayIter_Type +PyByteArray_AsString +PyByteArray_Concat +PyByteArray_FromObject +PyByteArray_FromStringAndSize +PyByteArray_Resize +PyByteArray_Size +PyByteArray_Type +PyBytesIter_Type +PyBytes_AsString +PyBytes_AsStringAndSize +PyBytes_Concat +PyBytes_ConcatAndDel +PyBytes_DecodeEscape +PyBytes_FromFormat +PyBytes_FromFormatV +PyBytes_FromObject +PyBytes_FromString +PyBytes_FromStringAndSize +PyBytes_Repr +PyBytes_Size +PyBytes_Type +PyCFunction_Call +PyCFunction_GetFlags +PyCFunction_GetFunction +PyCFunction_GetSelf +PyCFunction_NewEx +PyCFunction_Type +PyCMethod_New +PyCallIter_New +PyCallIter_Type +PyCallable_Check +PyCapsule_GetContext +PyCapsule_GetDestructor +PyCapsule_GetName +PyCapsule_GetPointer +PyCapsule_Import +PyCapsule_IsValid +PyCapsule_New +PyCapsule_SetContext +PyCapsule_SetDestructor +PyCapsule_SetName +PyCapsule_SetPointer +PyCapsule_Type +PyClassMethodDescr_Type +PyCodec_BackslashReplaceErrors +PyCodec_Decode +PyCodec_Decoder +PyCodec_Encode +PyCodec_Encoder +PyCodec_IgnoreErrors +PyCodec_IncrementalDecoder +PyCodec_IncrementalEncoder +PyCodec_KnownEncoding +PyCodec_LookupError +PyCodec_NameReplaceErrors +PyCodec_Register +PyCodec_RegisterError +PyCodec_ReplaceErrors +PyCodec_StreamReader +PyCodec_StreamWriter +PyCodec_StrictErrors +PyCodec_Unregister +PyCodec_XMLCharRefReplaceErrors +PyComplex_FromDoubles +PyComplex_ImagAsDouble +PyComplex_RealAsDouble +PyComplex_Type +PyDescr_NewClassMethod +PyDescr_NewGetSet +PyDescr_NewMember +PyDescr_NewMethod +PyDictItems_Type +PyDictIterItem_Type +PyDictIterKey_Type +PyDictIterValue_Type +PyDictKeys_Type +PyDictProxy_New +PyDictProxy_Type +PyDictRevIterItem_Type +PyDictRevIterKey_Type +PyDictRevIterValue_Type +PyDictValues_Type +PyDict_Clear +PyDict_Contains +PyDict_Copy +PyDict_DelItem +PyDict_DelItemString +PyDict_GetItem +PyDict_GetItemString +PyDict_GetItemWithError +PyDict_Items +PyDict_Keys +PyDict_Merge +PyDict_MergeFromSeq2 +PyDict_New +PyDict_Next +PyDict_SetItem +PyDict_SetItemString +PyDict_Size +PyDict_Type +PyDict_Update +PyDict_Values +PyEllipsis_Type +PyEnum_Type +PyErr_BadArgument +PyErr_BadInternalCall +PyErr_CheckSignals +PyErr_Clear +PyErr_Display +PyErr_ExceptionMatches +PyErr_Fetch +PyErr_Format +PyErr_FormatV +PyErr_GetExcInfo +PyErr_GivenExceptionMatches +PyErr_NewException +PyErr_NewExceptionWithDoc +PyErr_NoMemory +PyErr_NormalizeException +PyErr_Occurred +PyErr_Print +PyErr_PrintEx +PyErr_ProgramText +PyErr_ResourceWarning +PyErr_Restore +PyErr_SetExcInfo +PyErr_SetFromErrno +PyErr_SetFromErrnoWithFilename +PyErr_SetFromErrnoWithFilenameObject +PyErr_SetFromErrnoWithFilenameObjects +PyErr_SetImportError +PyErr_SetImportErrorSubclass +PyErr_SetInterrupt +PyErr_SetNone +PyErr_SetObject +PyErr_SetString +PyErr_SyntaxLocation +PyErr_SyntaxLocationEx +PyErr_WarnEx +PyErr_WarnExplicit +PyErr_WarnFormat +PyErr_WriteUnraisable +PyEval_AcquireLock +PyEval_AcquireThread +PyEval_CallFunction +PyEval_CallMethod +PyEval_CallObjectWithKeywords +PyEval_EvalCode +PyEval_EvalCodeEx +PyEval_EvalFrame +PyEval_EvalFrameEx +PyEval_GetBuiltins +PyEval_GetFrame +PyEval_GetFuncDesc +PyEval_GetFuncName +PyEval_GetGlobals +PyEval_GetLocals +PyEval_InitThreads +PyEval_ReleaseLock +PyEval_ReleaseThread +PyEval_RestoreThread +PyEval_SaveThread +PyEval_ThreadsInitialized +PyExc_ArithmeticError +PyExc_AssertionError +PyExc_AttributeError +PyExc_BaseException +PyExc_BlockingIOError +PyExc_BrokenPipeError +PyExc_BufferError +PyExc_BytesWarning +PyExc_ChildProcessError +PyExc_ConnectionAbortedError +PyExc_ConnectionError +PyExc_ConnectionRefusedError +PyExc_ConnectionResetError +PyExc_DeprecationWarning +PyExc_EOFError +PyExc_EnvironmentError +PyExc_Exception +PyExc_FileExistsError +PyExc_FileNotFoundError +PyExc_FloatingPointError +PyExc_FutureWarning +PyExc_GeneratorExit +PyExc_IOError +PyExc_ImportError +PyExc_ImportWarning +PyExc_IndentationError +PyExc_IndexError +PyExc_InterruptedError +PyExc_IsADirectoryError +PyExc_KeyError +PyExc_KeyboardInterrupt +PyExc_LookupError +PyExc_MemoryError +PyExc_ModuleNotFoundError +PyExc_NameError +PyExc_NotADirectoryError +PyExc_NotImplementedError +PyExc_OSError +PyExc_OverflowError +PyExc_PendingDeprecationWarning +PyExc_PermissionError +PyExc_ProcessLookupError +PyExc_RecursionError +PyExc_ReferenceError +PyExc_ResourceWarning +PyExc_RuntimeError +PyExc_RuntimeWarning +PyExc_StopAsyncIteration +PyExc_StopIteration +PyExc_SyntaxError +PyExc_SyntaxWarning +PyExc_SystemError +PyExc_SystemExit +PyExc_TabError +PyExc_TimeoutError +PyExc_TypeError +PyExc_UnboundLocalError +PyExc_UnicodeDecodeError +PyExc_UnicodeEncodeError +PyExc_UnicodeError +PyExc_UnicodeTranslateError +PyExc_UnicodeWarning +PyExc_UserWarning +PyExc_ValueError +PyExc_Warning +PyExc_ZeroDivisionError +PyExceptionClass_Name +PyException_GetCause +PyException_GetContext +PyException_GetTraceback +PyException_SetCause +PyException_SetContext +PyException_SetTraceback +PyFile_FromFd +PyFile_GetLine +PyFile_WriteObject +PyFile_WriteString +PyFilter_Type +PyFloat_AsDouble +PyFloat_FromDouble +PyFloat_FromString +PyFloat_GetInfo +PyFloat_GetMax +PyFloat_GetMin +PyFloat_Type +PyFrame_GetCode +PyFrame_GetLineNumber +PyFrozenSet_New +PyFrozenSet_Type +PyGC_Collect +PyGILState_Ensure +PyGILState_GetThisThreadState +PyGILState_Release +PyGetSetDescr_Type +PyImport_AddModule +PyImport_AddModuleObject +PyImport_AppendInittab +PyImport_ExecCodeModule +PyImport_ExecCodeModuleEx +PyImport_ExecCodeModuleObject +PyImport_ExecCodeModuleWithPathnames +PyImport_GetImporter +PyImport_GetMagicNumber +PyImport_GetMagicTag +PyImport_GetModule +PyImport_GetModuleDict +PyImport_Import +PyImport_ImportFrozenModule +PyImport_ImportFrozenModuleObject +PyImport_ImportModule +PyImport_ImportModuleLevel +PyImport_ImportModuleLevelObject +PyImport_ImportModuleNoBlock +PyImport_ReloadModule +PyIndex_Check +PyInterpreterState_Clear +PyInterpreterState_Delete +PyInterpreterState_Get +PyInterpreterState_GetDict +PyInterpreterState_GetID +PyInterpreterState_New +PyIter_Check +PyIter_Next +PyIter_Send +PyListIter_Type +PyListRevIter_Type +PyList_Append +PyList_AsTuple +PyList_GetItem +PyList_GetSlice +PyList_Insert +PyList_New +PyList_Reverse +PyList_SetItem +PyList_SetSlice +PyList_Size +PyList_Sort +PyList_Type +PyLongRangeIter_Type +PyLong_AsDouble +PyLong_AsLong +PyLong_AsLongAndOverflow +PyLong_AsLongLong +PyLong_AsLongLongAndOverflow +PyLong_AsSize_t +PyLong_AsSsize_t +PyLong_AsUnsignedLong +PyLong_AsUnsignedLongLong +PyLong_AsUnsignedLongLongMask +PyLong_AsUnsignedLongMask +PyLong_AsVoidPtr +PyLong_FromDouble +PyLong_FromLong +PyLong_FromLongLong +PyLong_FromSize_t +PyLong_FromSsize_t +PyLong_FromString +PyLong_FromUnsignedLong +PyLong_FromUnsignedLongLong +PyLong_FromVoidPtr +PyLong_GetInfo +PyLong_Type +PyMap_Type +PyMapping_Check +PyMapping_GetItemString +PyMapping_HasKey +PyMapping_HasKeyString +PyMapping_Items +PyMapping_Keys +PyMapping_Length +PyMapping_SetItemString +PyMapping_Size +PyMapping_Values +PyMarshal_ReadObjectFromString +PyMarshal_WriteLongToFile +PyMarshal_WriteObjectToFile +PyMarshal_WriteObjectToString +PyMem_Free +PyMem_Malloc +PyMem_Realloc +PyMemberDescr_Type +PyMember_GetOne +PyMember_SetOne +PyMemoryView_FromMemory +PyMemoryView_FromObject +PyMemoryView_GetContiguous +PyMemoryView_Type +PyMethodDescr_Type +PyModuleDef_Init +PyModuleDef_Type +PyModule_AddFunctions +PyModule_AddIntConstant +PyModule_AddObject +PyModule_AddObjectRef +PyModule_AddStringConstant +PyModule_AddType +PyModule_Create2 +PyModule_ExecDef +PyModule_FromDefAndSpec2 +PyModule_GetDef +PyModule_GetDict +PyModule_GetFilename +PyModule_GetFilenameObject +PyModule_GetName +PyModule_GetNameObject +PyModule_GetState +PyModule_New +PyModule_NewObject +PyModule_SetDocString +PyModule_Type +PyNumber_Absolute +PyNumber_Add +PyNumber_And +PyNumber_AsSsize_t +PyNumber_Check +PyNumber_Divmod +PyNumber_Float +PyNumber_FloorDivide +PyNumber_InPlaceAdd +PyNumber_InPlaceAnd +PyNumber_InPlaceFloorDivide +PyNumber_InPlaceLshift +PyNumber_InPlaceMatrixMultiply +PyNumber_InPlaceMultiply +PyNumber_InPlaceOr +PyNumber_InPlacePower +PyNumber_InPlaceRemainder +PyNumber_InPlaceRshift +PyNumber_InPlaceSubtract +PyNumber_InPlaceTrueDivide +PyNumber_InPlaceXor +PyNumber_Index +PyNumber_Invert +PyNumber_Long +PyNumber_Lshift +PyNumber_MatrixMultiply +PyNumber_Multiply +PyNumber_Negative +PyNumber_Or +PyNumber_Positive +PyNumber_Power +PyNumber_Remainder +PyNumber_Rshift +PyNumber_Subtract +PyNumber_ToBase +PyNumber_TrueDivide +PyNumber_Xor +PyOS_AfterFork +PyOS_AfterFork_Child +PyOS_AfterFork_Parent +PyOS_BeforeFork +PyOS_FSPath +PyOS_InterruptOccurred +PyOS_double_to_string +PyOS_getsig +PyOS_mystricmp +PyOS_mystrnicmp +PyOS_setsig +PyOS_snprintf +PyOS_string_to_double +PyOS_strtol +PyOS_strtoul +PyOS_vsnprintf +PyObject_ASCII +PyObject_AsFileDescriptor +PyObject_Bytes +PyObject_Call +PyObject_CallFunction +PyObject_CallFunctionObjArgs +PyObject_CallMethod +PyObject_CallMethodObjArgs +PyObject_CallNoArgs +PyObject_CallObject +PyObject_Calloc +PyObject_ClearWeakRefs +PyObject_DelItem +PyObject_DelItemString +PyObject_Dir +PyObject_Format +PyObject_Free +PyObject_GC_Del +PyObject_GC_IsFinalized +PyObject_GC_IsTracked +PyObject_GC_Track +PyObject_GC_UnTrack +PyObject_GenericGetAttr +PyObject_GenericGetDict +PyObject_GenericSetAttr +PyObject_GenericSetDict +PyObject_GetAttr +PyObject_GetAttrString +PyObject_GetItem +PyObject_GetIter +PyObject_HasAttr +PyObject_HasAttrString +PyObject_Hash +PyObject_HashNotImplemented +PyObject_Init +PyObject_InitVar +PyObject_IsInstance +PyObject_IsSubclass +PyObject_IsTrue +PyObject_Length +PyObject_Malloc +PyObject_Not +PyObject_Realloc +PyObject_Repr +PyObject_RichCompare +PyObject_RichCompareBool +PyObject_SelfIter +PyObject_SetAttr +PyObject_SetAttrString +PyObject_SetItem +PyObject_Size +PyObject_Str +PyObject_Type +PyProperty_Type +PyRangeIter_Type +PyRange_Type +PyReversed_Type +PySeqIter_New +PySeqIter_Type +PySequence_Check +PySequence_Concat +PySequence_Contains +PySequence_Count +PySequence_DelItem +PySequence_DelSlice +PySequence_Fast +PySequence_GetItem +PySequence_GetSlice +PySequence_In +PySequence_InPlaceConcat +PySequence_InPlaceRepeat +PySequence_Index +PySequence_Length +PySequence_List +PySequence_Repeat +PySequence_SetItem +PySequence_SetSlice +PySequence_Size +PySequence_Tuple +PySetIter_Type +PySet_Add +PySet_Clear +PySet_Contains +PySet_Discard +PySet_New +PySet_Pop +PySet_Size +PySet_Type +PySlice_AdjustIndices +PySlice_GetIndices +PySlice_GetIndicesEx +PySlice_New +PySlice_Type +PySlice_Unpack +PyState_AddModule +PyState_FindModule +PyState_RemoveModule +PyStructSequence_GetItem +PyStructSequence_New +PyStructSequence_NewType +PyStructSequence_SetItem +PySuper_Type +PySys_AddWarnOption +PySys_AddWarnOptionUnicode +PySys_AddXOption +PySys_FormatStderr +PySys_FormatStdout +PySys_GetObject +PySys_GetXOptions +PySys_HasWarnOptions +PySys_ResetWarnOptions +PySys_SetArgv +PySys_SetArgvEx +PySys_SetObject +PySys_SetPath +PySys_WriteStderr +PySys_WriteStdout +PyThreadState_Clear +PyThreadState_Delete +PyThreadState_Get +PyThreadState_GetDict +PyThreadState_GetFrame +PyThreadState_GetID +PyThreadState_GetInterpreter +PyThreadState_New +PyThreadState_SetAsyncExc +PyThreadState_Swap +PyThread_GetInfo +PyThread_ReInitTLS +PyThread_acquire_lock +PyThread_acquire_lock_timed +PyThread_allocate_lock +PyThread_create_key +PyThread_delete_key +PyThread_delete_key_value +PyThread_exit_thread +PyThread_free_lock +PyThread_get_key_value +PyThread_get_stacksize +PyThread_get_thread_ident +PyThread_get_thread_native_id +PyThread_init_thread +PyThread_release_lock +PyThread_set_key_value +PyThread_set_stacksize +PyThread_start_new_thread +PyThread_tss_alloc +PyThread_tss_create +PyThread_tss_delete +PyThread_tss_free +PyThread_tss_get +PyThread_tss_is_created +PyThread_tss_set +PyTraceBack_Here +PyTraceBack_Print +PyTraceBack_Type +PyTupleIter_Type +PyTuple_GetItem +PyTuple_GetSlice +PyTuple_New +PyTuple_Pack +PyTuple_SetItem +PyTuple_Size +PyTuple_Type +PyType_ClearCache +PyType_FromModuleAndSpec +PyType_FromSpec +PyType_FromSpecWithBases +PyType_GenericAlloc +PyType_GenericNew +PyType_GetFlags +PyType_GetModule +PyType_GetModuleState +PyType_GetSlot +PyType_IsSubtype +PyType_Modified +PyType_Ready +PyType_Type +PyUnicodeDecodeError_Create +PyUnicodeDecodeError_GetEncoding +PyUnicodeDecodeError_GetEnd +PyUnicodeDecodeError_GetObject +PyUnicodeDecodeError_GetReason +PyUnicodeDecodeError_GetStart +PyUnicodeDecodeError_SetEnd +PyUnicodeDecodeError_SetReason +PyUnicodeDecodeError_SetStart +PyUnicodeEncodeError_GetEncoding +PyUnicodeEncodeError_GetEnd +PyUnicodeEncodeError_GetObject +PyUnicodeEncodeError_GetReason +PyUnicodeEncodeError_GetStart +PyUnicodeEncodeError_SetEnd +PyUnicodeEncodeError_SetReason +PyUnicodeEncodeError_SetStart +PyUnicodeIter_Type +PyUnicodeTranslateError_GetEnd +PyUnicodeTranslateError_GetObject +PyUnicodeTranslateError_GetReason +PyUnicodeTranslateError_GetStart +PyUnicodeTranslateError_SetEnd +PyUnicodeTranslateError_SetReason +PyUnicodeTranslateError_SetStart +PyUnicode_Append +PyUnicode_AppendAndDel +PyUnicode_AsASCIIString +PyUnicode_AsCharmapString +PyUnicode_AsDecodedObject +PyUnicode_AsDecodedUnicode +PyUnicode_AsEncodedObject +PyUnicode_AsEncodedString +PyUnicode_AsEncodedUnicode +PyUnicode_AsLatin1String +PyUnicode_AsRawUnicodeEscapeString +PyUnicode_AsUCS4 +PyUnicode_AsUCS4Copy +PyUnicode_AsUTF16String +PyUnicode_AsUTF32String +PyUnicode_AsUTF8AndSize +PyUnicode_AsUTF8String +PyUnicode_AsUnicodeEscapeString +PyUnicode_AsWideChar +PyUnicode_AsWideCharString +PyUnicode_BuildEncodingMap +PyUnicode_Compare +PyUnicode_CompareWithASCIIString +PyUnicode_Concat +PyUnicode_Contains +PyUnicode_Count +PyUnicode_Decode +PyUnicode_DecodeASCII +PyUnicode_DecodeCharmap +PyUnicode_DecodeFSDefault +PyUnicode_DecodeFSDefaultAndSize +PyUnicode_DecodeLatin1 +PyUnicode_DecodeLocale +PyUnicode_DecodeLocaleAndSize +PyUnicode_DecodeRawUnicodeEscape +PyUnicode_DecodeUTF16 +PyUnicode_DecodeUTF16Stateful +PyUnicode_DecodeUTF32 +PyUnicode_DecodeUTF32Stateful +PyUnicode_DecodeUTF7 +PyUnicode_DecodeUTF7Stateful +PyUnicode_DecodeUTF8 +PyUnicode_DecodeUTF8Stateful +PyUnicode_DecodeUnicodeEscape +PyUnicode_EncodeFSDefault +PyUnicode_EncodeLocale +PyUnicode_FSConverter +PyUnicode_FSDecoder +PyUnicode_Find +PyUnicode_FindChar +PyUnicode_Format +PyUnicode_FromEncodedObject +PyUnicode_FromFormat +PyUnicode_FromFormatV +PyUnicode_FromObject +PyUnicode_FromOrdinal +PyUnicode_FromString +PyUnicode_FromStringAndSize +PyUnicode_FromWideChar +PyUnicode_GetDefaultEncoding +PyUnicode_GetLength +PyUnicode_GetSize +PyUnicode_InternFromString +PyUnicode_InternImmortal +PyUnicode_InternInPlace +PyUnicode_IsIdentifier +PyUnicode_Join +PyUnicode_Partition +PyUnicode_RPartition +PyUnicode_RSplit +PyUnicode_ReadChar +PyUnicode_Replace +PyUnicode_Resize +PyUnicode_RichCompare +PyUnicode_Split +PyUnicode_Splitlines +PyUnicode_Substring +PyUnicode_Tailmatch +PyUnicode_Translate +PyUnicode_Type +PyUnicode_WriteChar +PyWeakref_GetObject +PyWeakref_NewProxy +PyWeakref_NewRef +PyWrapperDescr_Type +PyWrapper_New +PyZip_Type +Py_AddPendingCall +Py_AtExit +Py_BuildValue +Py_BytesMain +Py_CompileString +Py_DecRef +Py_DecodeLocale +Py_EncodeLocale +Py_EndInterpreter +Py_EnterRecursiveCall +Py_Exit +Py_FatalError +Py_FileSystemDefaultEncodeErrors +Py_FileSystemDefaultEncoding +Py_Finalize +Py_FinalizeEx +Py_FrozenMain +Py_GenericAlias +Py_GenericAliasType +Py_GetBuildInfo +Py_GetCompiler +Py_GetCopyright +Py_GetExecPrefix +Py_GetPath +Py_GetPlatform +Py_GetPrefix +Py_GetProgramFullPath +Py_GetProgramName +Py_GetPythonHome +Py_GetRecursionLimit +Py_GetVersion +Py_HasFileSystemDefaultEncoding +Py_IncRef +Py_Initialize +Py_InitializeEx +Py_IsInitialized +Py_LeaveRecursiveCall +Py_Main +Py_MakePendingCalls +Py_NewInterpreter +Py_NewRef +Py_ReprEnter +Py_ReprLeave +Py_SetPath +Py_SetProgramName +Py_SetPythonHome +Py_SetRecursionLimit +Py_SymtableString +Py_UTF8Mode +Py_VaBuildValue +Py_XNewRef diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 506973e964..0661c82832 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -158,6 +158,18 @@ Glossary See also :term:`text file` for a file object able to read and write :class:`str` objects. + borrowed reference + In Python's C API, a borrowed reference is a reference to an object. + It does not modify the object reference count. It becomes a dangling + pointer if the object is destroyed. For example, a garbage collection can + remove the last :term:`strong reference` to the object and so destroy it. + + Calling :c:func:`Py_INCREF` on the :term:`borrowed reference` is + recommended to convert it to a :term:`strong reference` in-place, except + when the object cannot be destroyed before the last usage of the borrowed + reference. The :c:func:`Py_NewRef` function can be used to create a new + :term:`strong reference`. + bytes-like object An object that supports the :ref:`bufferobjects` and can export a C-:term:`contiguous` buffer. This includes all :class:`bytes`, @@ -1100,6 +1112,18 @@ Glossary an :term:`expression` or one of several constructs with a keyword, such as :keyword:`if`, :keyword:`while` or :keyword:`for`. + strong reference + In Python's C API, a strong reference is a reference to an object + which increments the object's reference count when it is created and + decrements the object's reference count when it is deleted. + + The :c:func:`Py_NewRef` function can be used to create a strong reference + to an object. Usually, the :c:func:`Py_DECREF` function must be called on + the strong reference before exiting the scope of the strong reference, to + avoid leaking one reference. + + See also :term:`borrowed reference`. + text encoding A codec which encodes Unicode strings to bytes. diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst index 76d8e6be42..a97d10cfe6 100644 --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -83,7 +83,7 @@ Following is a result of running the code: $ python3 prog.py --help usage: prog.py [-h] - optional arguments: + options: -h, --help show this help message and exit $ python3 prog.py --verbose usage: prog.py [-h] @@ -130,7 +130,7 @@ And running the code: positional arguments: echo - optional arguments: + options: -h, --help show this help message and exit $ python3 prog.py foo foo @@ -172,7 +172,7 @@ And we get: positional arguments: echo echo the string you use here - optional arguments: + options: -h, --help show this help message and exit Now, how about doing something even more useful:: @@ -241,7 +241,7 @@ And the output: $ python3 prog.py --help usage: prog.py [-h] [--verbosity VERBOSITY] - optional arguments: + options: -h, --help show this help message and exit --verbosity VERBOSITY increase output verbosity @@ -289,7 +289,7 @@ And the output: $ python3 prog.py --help usage: prog.py [-h] [--verbose] - optional arguments: + options: -h, --help show this help message and exit --verbose increase output verbosity @@ -332,7 +332,7 @@ And here goes: $ python3 prog.py --help usage: prog.py [-h] [-v] - optional arguments: + options: -h, --help show this help message and exit -v, --verbose increase output verbosity @@ -440,7 +440,7 @@ And the output: positional arguments: square display a square of a given number - optional arguments: + options: -h, --help show this help message and exit -v {0,1,2}, --verbosity {0,1,2} increase output verbosity @@ -468,7 +468,8 @@ verbosity argument (check the output of ``python --help``):: print(answer) We have introduced another action, "count", -to count the number of occurrences of a specific optional arguments: +to count the number of occurrences of specific options. + .. code-block:: shell-session @@ -489,7 +490,7 @@ to count the number of occurrences of a specific optional arguments: positional arguments: square display a square of a given number - optional arguments: + options: -h, --help show this help message and exit -v, --verbosity increase output verbosity $ python3 prog.py 4 -vvv @@ -626,7 +627,7 @@ Output: x the base y the exponent - optional arguments: + options: -h, --help show this help message and exit -v, --verbosity $ python3 prog.py 4 2 -v @@ -750,7 +751,7 @@ but not both at the same time: x the base y the exponent - optional arguments: + options: -h, --help show this help message and exit -v, --verbose -q, --quiet diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst index fedf8a8c09..ab5a573c6a 100644 --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -16,7 +16,7 @@ storage, and deletion. This guide has four major sections: 1) The "primer" gives a basic overview, moving gently from simple examples, - adding one feature at a time. It is a great place to start. + adding one feature at a time. Start here if you're new to descriptors. 2) The second section shows a complete, practical descriptor example. If you already know the basics, start there. @@ -42,21 +42,27 @@ add new capabilities one by one. Simple example: A descriptor that returns a constant ---------------------------------------------------- -The :class:`Ten` class is a descriptor that always returns the constant ``10``:: +The :class:`Ten` class is a descriptor that always returns the constant ``10`` +from its :meth:`__get__` method: +.. testcode:: class Ten: def __get__(self, obj, objtype=None): return 10 -To use the descriptor, it must be stored as a class variable in another class:: +To use the descriptor, it must be stored as a class variable in another class: + +.. testcode:: class A: x = 5 # Regular class attribute y = Ten() # Descriptor instance An interactive session shows the difference between normal attribute lookup -and descriptor lookup:: +and descriptor lookup: + +.. doctest:: >>> a = A() # Make an instance of class A >>> a.x # Normal attribute lookup @@ -64,9 +70,11 @@ and descriptor lookup:: >>> a.y # Descriptor lookup 10 -In the ``a.x`` attribute lookup, the dot operator finds the value ``5`` stored -in the class dictionary. In the ``a.y`` descriptor lookup, the dot operator -calls the descriptor's :meth:`__get__()` method. That method returns ``10``. +In the ``a.x`` attribute lookup, the dot operator finds the key ``x`` and the +value ``5`` in the class dictionary. In the ``a.y`` lookup, the dot operator +finds a descriptor instance, recognized by its ``__get__`` method, and calls +that method which returns ``10``. + Note that the value ``10`` is not stored in either the class dictionary or the instance dictionary. Instead, the value ``10`` is computed on demand. @@ -79,7 +87,10 @@ In the next section, we'll create something more useful, a dynamic lookup. Dynamic lookups --------------- -Interesting descriptors typically run computations instead of doing lookups:: +Interesting descriptors typically run computations instead of returning +constants: + +.. testcode:: import os @@ -98,21 +109,20 @@ Interesting descriptors typically run computations instead of doing lookups:: An interactive session shows that the lookup is dynamic — it computes different, updated answers each time:: - >>> g = Directory('games') >>> s = Directory('songs') + >>> g = Directory('games') + >>> s.size # The songs directory has twenty files + 20 >>> g.size # The games directory has three files 3 - >>> os.system('touch games/newfile') # Add a fourth file to the directory - 0 - >>> g.size # Automatically updated + >>> open('games/newfile').close() # Add a fourth file to the directory + >>> g.size # File count is automatically updated 4 - >>> s.size # The songs directory has twenty files - 20 Besides showing how descriptors can run computations, this example also reveals the purpose of the parameters to :meth:`__get__`. The *self* parameter is *size*, an instance of *DirectorySize*. The *obj* parameter is -either *g* or *s*, an instance of *Directory*. It is *obj* parameter that +either *g* or *s*, an instance of *Directory*. It is the *obj* parameter that lets the :meth:`__get__` method learn the target directory. The *objtype* parameter is the class *Directory*. @@ -128,7 +138,9 @@ the public attribute is accessed. In the following example, *age* is the public attribute and *_age* is the private attribute. When the public attribute is accessed, the descriptor logs -the lookup or update:: +the lookup or update: + +.. testcode:: import logging @@ -158,7 +170,15 @@ the lookup or update:: An interactive session shows that all access to the managed attribute *age* is -logged, but that the regular attribute *name* is not logged:: +logged, but that the regular attribute *name* is not logged: + +.. testcode:: + :hide: + + import logging, sys + logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True) + +.. doctest:: >>> mary = Person('Mary M', 30) # The initial age update is logged INFO:root:Updating 'age' to 30 @@ -183,7 +203,7 @@ logged, but that the regular attribute *name* is not logged:: INFO:root:Accessing 'age' giving 40 40 -One major issue with this example is the private name *_age* is hardwired in +One major issue with this example is that the private name *_age* is hardwired in the *LoggedAgeAccess* class. That means that each instance can only have one logged attribute and that its name is unchangeable. In the next example, we'll fix that problem. @@ -192,13 +212,15 @@ we'll fix that problem. Customized names ---------------- -When a class uses descriptors, it can inform each descriptor about what +When a class uses descriptors, it can inform each descriptor about which variable name was used. In this example, the :class:`Person` class has two descriptor instances, *name* and *age*. When the :class:`Person` class is defined, it makes a callback to :meth:`__set_name__` in *LoggedAccess* so that the field names can -be recorded, giving each descriptor its own *public_name* and *private_name*:: +be recorded, giving each descriptor its own *public_name* and *private_name*: + +.. testcode:: import logging @@ -208,7 +230,7 @@ be recorded, giving each descriptor its own *public_name* and *private_name*:: def __set_name__(self, owner, name): self.public_name = name - self.private_name = f'_{name}' + self.private_name = '_' + name def __get__(self, obj, objtype=None): value = getattr(obj, self.private_name) @@ -233,14 +255,24 @@ be recorded, giving each descriptor its own *public_name* and *private_name*:: An interactive session shows that the :class:`Person` class has called :meth:`__set_name__` so that the field names would be recorded. Here -we call :func:`vars` to lookup the descriptor without triggering it:: +we call :func:`vars` to look up the descriptor without triggering it: + +.. doctest:: >>> vars(vars(Person)['name']) {'public_name': 'name', 'private_name': '_name'} >>> vars(vars(Person)['age']) {'public_name': 'age', 'private_name': '_age'} -The new class now logs access to both *name* and *age*:: +The new class now logs access to both *name* and *age*: + +.. testcode:: + :hide: + + import logging, sys + logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True) + +.. doctest:: >>> pete = Person('Peter P', 10) INFO:root:Updating 'name' to 'Peter P' @@ -265,9 +297,10 @@ A :term:`descriptor` is what we call any object that defines :meth:`__get__`, Optionally, descriptors can have a :meth:`__set_name__` method. This is only used in cases where a descriptor needs to know either the class where it was -created or the name of class variable it was assigned to. +created or the name of class variable it was assigned to. (This method, if +present, is called even if the class is not a descriptor.) -Descriptors get invoked by the dot operator during attribute lookup. If a +Descriptors get invoked by the dot "operator" during attribute lookup. If a descriptor is accessed indirectly with ``vars(some_class)[descriptor_name]``, the descriptor instance is returned without invoking it. @@ -275,7 +308,7 @@ Descriptors only work when used as class variables. When put in instances, they have no effect. The main motivation for descriptors is to provide a hook allowing objects -stored in class variables to control what happens during dotted lookup. +stored in class variables to control what happens during attribute lookup. Traditionally, the calling class controls what happens during lookup. Descriptors invert that relationship and allow the data being looked-up to @@ -303,14 +336,16 @@ restrictions. If those restrictions aren't met, it raises an exception to prevent data corruption at its source. This :class:`Validator` class is both an :term:`abstract base class` and a -managed attribute descriptor:: +managed attribute descriptor: + +.. testcode:: from abc import ABC, abstractmethod class Validator(ABC): def __set_name__(self, owner, name): - self.private_name = f'_{name}' + self.private_name = '_' + name def __get__(self, obj, objtype=None): return getattr(obj, self.private_name) @@ -343,7 +378,7 @@ Here are three practical data validation utilities: user-defined `predicate <https://en.wikipedia.org/wiki/Predicate_(mathematical_logic)>`_ as well. -:: +.. testcode:: class OneOf(Validator): @@ -396,10 +431,12 @@ Here are three practical data validation utilities: ) -Practical use -------------- +Practical application +--------------------- + +Here's how the data validators can be used in a real class: -Here's how the data validators can be used in a real class:: +.. testcode:: class Component: @@ -412,13 +449,30 @@ Here's how the data validators can be used in a real class:: self.kind = kind self.quantity = quantity -The descriptors prevent invalid instances from being created:: +The descriptors prevent invalid instances from being created: + +.. doctest:: + + >>> Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase + Traceback (most recent call last): + ... + ValueError: Expected <method 'isupper' of 'str' objects> to be true for 'Widget' - Component('WIDGET', 'metal', 5) # Allowed. - Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase - Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled - Component('WIDGET', 'metal', -5) # Blocked: -5 is negative - Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number + >>> Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled + Traceback (most recent call last): + ... + ValueError: Expected 'metle' to be one of {'metal', 'plastic', 'wood'} + + >>> Component('WIDGET', 'metal', -5) # Blocked: -5 is negative + Traceback (most recent call last): + ... + ValueError: Expected -5 to be at least 0 + >>> Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number + Traceback (most recent call last): + ... + TypeError: Expected 'V' to be an int or float + + >>> c = Component('WIDGET', 'metal', 5) # Allowed: The inputs are valid Technical Tutorial @@ -435,23 +489,21 @@ Defines descriptors, summarizes the protocol, and shows how descriptors are called. Provides an example showing how object relational mappings work. Learning about descriptors not only provides access to a larger toolset, it -creates a deeper understanding of how Python works and an appreciation for the -elegance of its design. +creates a deeper understanding of how Python works. Definition and introduction --------------------------- -In general, a descriptor is an object attribute with "binding behavior", one -whose attribute access has been overridden by methods in the descriptor -protocol. Those methods are :meth:`__get__`, :meth:`__set__`, and -:meth:`__delete__`. If any of those methods are defined for an object, it is -said to be a :term:`descriptor`. +In general, a descriptor is an attribute value that has one of the methods in +the descriptor protocol. Those methods are :meth:`__get__`, :meth:`__set__`, +and :meth:`__delete__`. If any of those methods are defined for an the +attribute, it is said to be a :term:`descriptor`. The default behavior for attribute access is to get, set, or delete the attribute from an object's dictionary. For instance, ``a.x`` has a lookup chain starting with ``a.__dict__['x']``, then ``type(a).__dict__['x']``, and -continuing through the base classes of ``type(a)``. If the +continuing through the method resolution order of ``type(a)``. If the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which descriptor methods @@ -479,7 +531,7 @@ as an attribute. If an object defines :meth:`__set__` or :meth:`__delete__`, it is considered a data descriptor. Descriptors that only define :meth:`__get__` are called -non-data descriptors (they are typically used for methods but other uses are +non-data descriptors (they are often used for methods but other uses are possible). Data and non-data descriptors differ in how overrides are calculated with @@ -504,8 +556,9 @@ But it is more common for a descriptor to be invoked automatically from attribute access. The expression ``obj.x`` looks up the attribute ``x`` in the chain of -namespaces for ``obj``. If the search finds a descriptor, its :meth:`__get__` -method is invoked according to the precedence rules listed below. +namespaces for ``obj``. If the search finds a descriptor outside of the +instance ``__dict__``, its :meth:`__get__` method is invoked according to the +precedence rules listed below. The details of invocation depend on whether ``obj`` is an object, class, or instance of super. @@ -523,31 +576,145 @@ If a descriptor is found for ``a.x``, then it is invoked with: ``desc.__get__(a, type(a))``. The logic for a dotted lookup is in :meth:`object.__getattribute__`. Here is -a pure Python equivalent:: +a pure Python equivalent: + +.. testcode:: def object_getattribute(obj, name): "Emulate PyObject_GenericGetAttr() in Objects/object.c" null = object() objtype = type(obj) - value = getattr(objtype, name, null) - if value is not null and hasattr(value, '__get__'): - if hasattr(value, '__set__') or hasattr(value, '__delete__'): - return value.__get__(obj, objtype) # data descriptor - try: - return vars(obj)[name] # instance variable - except (KeyError, TypeError): - pass - if hasattr(value, '__get__'): - return value.__get__(obj, objtype) # non-data descriptor - if value is not null: - return value # class variable - # Emulate slot_tp_getattr_hook() in Objects/typeobject.c - if hasattr(objtype, '__getattr__'): - return objtype.__getattr__(obj, name) # __getattr__ hook + cls_var = getattr(objtype, name, null) + descr_get = getattr(type(cls_var), '__get__', null) + if descr_get is not null: + if (hasattr(type(cls_var), '__set__') + or hasattr(type(cls_var), '__delete__')): + return descr_get(cls_var, obj, objtype) # data descriptor + if hasattr(obj, '__dict__') and name in vars(obj): + return vars(obj)[name] # instance variable + if descr_get is not null: + return descr_get(cls_var, obj, objtype) # non-data descriptor + if cls_var is not null: + return cls_var # class variable raise AttributeError(name) -The :exc:`TypeError` exception handler is needed because the instance dictionary -doesn't exist when its class defines :term:`__slots__`. + +.. testcode:: + :hide: + + # Test the fidelity of object_getattribute() by comparing it with the + # normal object.__getattribute__(). The former will be accessed by + # square brackets and the latter by the dot operator. + + class Object: + + def __getitem__(obj, name): + try: + return object_getattribute(obj, name) + except AttributeError: + if not hasattr(type(obj), '__getattr__'): + raise + return type(obj).__getattr__(obj, name) # __getattr__ + + class DualOperator(Object): + + x = 10 + + def __init__(self, z): + self.z = z + + @property + def p2(self): + return 2 * self.x + + @property + def p3(self): + return 3 * self.x + + def m5(self, y): + return 5 * y + + def m7(self, y): + return 7 * y + + def __getattr__(self, name): + return ('getattr_hook', self, name) + + class DualOperatorWithSlots: + + __getitem__ = Object.__getitem__ + + __slots__ = ['z'] + + x = 15 + + def __init__(self, z): + self.z = z + + @property + def p2(self): + return 2 * self.x + + def m5(self, y): + return 5 * y + + def __getattr__(self, name): + return ('getattr_hook', self, name) + + +.. doctest:: + :hide: + + >>> a = DualOperator(11) + >>> vars(a).update(p3 = '_p3', m7 = '_m7') + >>> a.x == a['x'] == 10 + True + >>> a.z == a['z'] == 11 + True + >>> a.p2 == a['p2'] == 20 + True + >>> a.p3 == a['p3'] == 30 + True + >>> a.m5(100) == a.m5(100) == 500 + True + >>> a.m7 == a['m7'] == '_m7' + True + >>> a.g == a['g'] == ('getattr_hook', a, 'g') + True + + >>> b = DualOperatorWithSlots(22) + >>> b.x == b['x'] == 15 + True + >>> b.z == b['z'] == 22 + True + >>> b.p2 == b['p2'] == 30 + True + >>> b.m5(200) == b['m5'](200) == 1000 + True + >>> b.g == b['g'] == ('getattr_hook', b, 'g') + True + + +Interestingly, attribute lookup doesn't call :meth:`object.__getattribute__` +directly. Instead, both the dot operator and the :func:`getattr` function +perform attribute lookup by way of a helper function: + +.. testcode:: + + def getattr_hook(obj, name): + "Emulate slot_tp_getattr_hook() in Objects/typeobject.c" + try: + return obj.__getattribute__(name) + except AttributeError: + if not hasattr(type(obj), '__getattr__'): + raise + return type(obj).__getattr__(obj, name) # __getattr__ + +So if :meth:`__getattr__` exists, it is called whenever :meth:`__getattribute__` +raises :exc:`AttributeError` (either directly or in one of the descriptor calls). + +Also, if a user calls :meth:`object.__getattribute__` directly, the +:meth:`__getattr__` hook is bypassed entirely. Invocation from a class @@ -614,8 +781,8 @@ Sometimes it is desirable for a descriptor to know what class variable name it was assigned to. When a new class is created, the :class:`type` metaclass scans the dictionary of the new class. If any of the entries are descriptors and if they define :meth:`__set_name__`, that method is called with two -arguments. The *owner* is the class where the descriptor is used, the *name* -is class variable the descriptor was assigned to. +arguments. The *owner* is the class where the descriptor is used, and the +*name* is the class variable the descriptor was assigned to. The implementation details are in :c:func:`type_new()` and :c:func:`set_names()` in :source:`Objects/typeobject.c`. @@ -634,7 +801,9 @@ be used to implement an `object relational mapping The essential idea is that the data is stored in an external database. The Python instances only hold keys to the database's tables. Descriptors take -care of lookups or updates:: +care of lookups or updates: + +.. testcode:: class Field: @@ -649,8 +818,11 @@ care of lookups or updates:: conn.execute(self.store, [value, obj.key]) conn.commit() -We can use the :class:`Field` class to define "models" that describe the schema -for each table in a database:: +We can use the :class:`Field` class to define `models +<https://en.wikipedia.org/wiki/Database_model>`_ that describe the schema for +each table in a database: + +.. testcode:: class Movie: table = 'Movies' # Table name @@ -671,12 +843,41 @@ for each table in a database:: def __init__(self, key): self.key = key -An interactive session shows how data is retrieved from the database and how -it can be updated:: +To use the models, first connect to the database:: >>> import sqlite3 >>> conn = sqlite3.connect('entertainment.db') +An interactive session shows how data is retrieved from the database and how +it can be updated: + +.. testsetup:: + + song_data = [ + ('Country Roads', 'John Denver', 1972), + ('Me and Bobby McGee', 'Janice Joplin', 1971), + ('Coal Miners Daughter', 'Loretta Lynn', 1970), + ] + + movie_data = [ + ('Star Wars', 'George Lucas', 1977), + ('Jaws', 'Steven Spielberg', 1975), + ('Aliens', 'James Cameron', 1986), + ] + + import sqlite3 + + conn = sqlite3.connect(':memory:') + conn.execute('CREATE TABLE Music (title text, artist text, year integer);') + conn.execute('CREATE INDEX MusicNdx ON Music (title);') + conn.executemany('INSERT INTO Music VALUES (?, ?, ?);', song_data) + conn.execute('CREATE TABLE Movies (title text, director text, year integer);') + conn.execute('CREATE INDEX MovieNdx ON Music (title);') + conn.executemany('INSERT INTO Movies VALUES (?, ?, ?);', movie_data) + conn.commit() + +.. doctest:: + >>> Movie('Star Wars').director 'George Lucas' >>> jaws = Movie('Jaws') @@ -690,6 +891,7 @@ it can be updated:: >>> Movie('Star Wars').director 'J.J. Abrams' + Pure Python Equivalents ^^^^^^^^^^^^^^^^^^^^^^^ @@ -703,11 +905,13 @@ Properties ---------- Calling :func:`property` is a succinct way of building a data descriptor that -triggers function calls upon access to an attribute. Its signature is:: +triggers a function call upon access to an attribute. Its signature is:: property(fget=None, fset=None, fdel=None, doc=None) -> property -The documentation shows a typical use to define a managed attribute ``x``:: +The documentation shows a typical use to define a managed attribute ``x``: + +.. testcode:: class C: def getx(self): return self.__x @@ -716,7 +920,9 @@ The documentation shows a typical use to define a managed attribute ``x``:: x = property(getx, setx, delx, "I'm the 'x' property.") To see how :func:`property` is implemented in terms of the descriptor protocol, -here is a pure Python equivalent:: +here is a pure Python equivalent: + +.. testcode:: class Property: "Emulate PyProperty_Type() in Objects/descrobject.c" @@ -755,6 +961,57 @@ here is a pure Python equivalent:: def deleter(self, fdel): return type(self)(self.fget, self.fset, fdel, self.__doc__) +.. testcode:: + :hide: + + # Verify the Property() emulation + + class CC: + def getx(self): + return self.__x + def setx(self, value): + self.__x = value + def delx(self): + del self.__x + x = Property(getx, setx, delx, "I'm the 'x' property.") + + # Now do it again but use the decorator style + + class CCC: + @Property + def x(self): + return self.__x + @x.setter + def x(self, value): + self.__x = value + @x.deleter + def x(self): + del self.__x + + +.. doctest:: + :hide: + + >>> cc = CC() + >>> hasattr(cc, 'x') + False + >>> cc.x = 33 + >>> cc.x + 33 + >>> del cc.x + >>> hasattr(cc, 'x') + False + + >>> ccc = CCC() + >>> hasattr(ccc, 'x') + False + >>> ccc.x = 333 + >>> ccc.x == 333 + True + >>> del ccc.x + >>> hasattr(ccc, 'x') + False + The :func:`property` builtin helps whenever a user interface has granted attribute access and then subsequent changes require the intervention of a method. @@ -763,7 +1020,9 @@ For instance, a spreadsheet class may grant access to a cell value through ``Cell('b10').value``. Subsequent improvements to the program require the cell to be recalculated on every access; however, the programmer does not want to affect existing client code accessing the attribute directly. The solution is -to wrap access to the value attribute in a property data descriptor:: +to wrap access to the value attribute in a property data descriptor: + +.. testcode:: class Cell: ... @@ -774,6 +1033,9 @@ to wrap access to the value attribute in a property data descriptor:: self.recalc() return self._value +Either the built-in :func:`property` or our :func:`Property` equivalent would +work in this example. + Functions and methods --------------------- @@ -787,7 +1049,9 @@ prepended to the other arguments. By convention, the instance is called *self* but could be called *this* or any other variable name. Methods can be created manually with :class:`types.MethodType` which is -roughly equivalent to:: +roughly equivalent to: + +.. testcode:: class MethodType: "Emulate Py_MethodType in Objects/classobject.c" @@ -803,8 +1067,10 @@ roughly equivalent to:: To support automatic creation of methods, functions include the :meth:`__get__` method for binding methods during attribute access. This -means that functions are non-data descriptors which return bound methods -during dotted lookup from an instance. Here's how it works:: +means that functions are non-data descriptors that return bound methods +during dotted lookup from an instance. Here's how it works: + +.. testcode:: class Function: ... @@ -816,13 +1082,17 @@ during dotted lookup from an instance. Here's how it works:: return MethodType(self, obj) Running the following class in the interpreter shows how the function -descriptor works in practice:: +descriptor works in practice: + +.. testcode:: class D: def f(self, x): return x -The function has a :term:`qualified name` attribute to support introspection:: +The function has a :term:`qualified name` attribute to support introspection: + +.. doctest:: >>> D.f.__qualname__ 'D.f' @@ -850,7 +1120,7 @@ Internally, the bound method stores the underlying function and the bound instance:: >>> d.f.__func__ - <function D.f at 0x1012e5ae8> + <function D.f at 0x00C45070> >>> d.f.__self__ <__main__.D object at 0x1012e1f98> @@ -902,20 +1172,26 @@ It can be called either from an object or the class: ``s.erf(1.5) --> .9332`` o ``Sample.erf(1.5) --> .9332``. Since static methods return the underlying function with no changes, the -example calls are unexciting:: +example calls are unexciting: + +.. testcode:: class E: @staticmethod def f(x): print(x) +.. doctest:: + >>> E.f(3) 3 >>> E().f(3) 3 Using the non-data descriptor protocol, a pure Python version of -:func:`staticmethod` would look like this:: +:func:`staticmethod` would look like this: + +.. doctest:: class StaticMethod: "Emulate PyStaticMethod_Type() in Objects/funcobject.c" @@ -932,27 +1208,31 @@ Class methods Unlike static methods, class methods prepend the class reference to the argument list before calling the function. This format is the same -for whether the caller is an object or a class:: +for whether the caller is an object or a class: + +.. testcode:: class F: @classmethod def f(cls, x): return cls.__name__, x - >>> print(F.f(3)) +.. doctest:: + + >>> F.f(3) ('F', 3) - >>> print(F().f(3)) + >>> F().f(3) ('F', 3) This behavior is useful whenever the method only needs to have a class -reference and does rely on data stored in a specific instance. One use for +reference and does not rely on data stored in a specific instance. One use for class methods is to create alternate class constructors. For example, the classmethod :func:`dict.fromkeys` creates a new dictionary from a list of -keys. The pure Python equivalent is:: +keys. The pure Python equivalent is: - class Dict: - ... +.. testcode:: + class Dict(dict): @classmethod def fromkeys(cls, iterable, value=None): "Emulate dict_fromkeys() in Objects/dictobject.c" @@ -961,13 +1241,20 @@ keys. The pure Python equivalent is:: d[key] = value return d -Now a new dictionary of unique keys can be constructed like this:: +Now a new dictionary of unique keys can be constructed like this: + +.. doctest:: - >>> Dict.fromkeys('abracadabra') - {'a': None, 'r': None, 'b': None, 'c': None, 'd': None} + >>> d = Dict.fromkeys('abracadabra') + >>> type(d) is Dict + True + >>> d + {'a': None, 'b': None, 'r': None, 'c': None, 'd': None} Using the non-data descriptor protocol, a pure Python version of -:func:`classmethod` would look like this:: +:func:`classmethod` would look like this: + +.. testcode:: class ClassMethod: "Emulate PyClassMethod_Type() in Objects/funcobject.c" @@ -982,9 +1269,31 @@ Using the non-data descriptor protocol, a pure Python version of return self.f.__get__(cls) return MethodType(self.f, cls) +.. testcode:: + :hide: + + # Verify the emulation works + class T: + @ClassMethod + def cm(cls, x, y): + return (cls, x, y) + +.. doctest:: + :hide: + + >>> T.cm(11, 22) + (<class 'T'>, 11, 22) + + # Also call it from an instance + >>> t = T() + >>> t.cm(11, 22) + (<class 'T'>, 11, 22) + The code path for ``hasattr(obj, '__get__')`` was added in Python 3.9 and makes it possible for :func:`classmethod` to support chained decorators. -For example, a classmethod and property could be chained together:: +For example, a classmethod and property could be chained together: + +.. testcode:: class G: @classmethod @@ -992,6 +1301,12 @@ For example, a classmethod and property could be chained together:: def __doc__(cls): return f'A doc for {cls.__name__!r}' +.. doctest:: + + >>> G.__doc__ + "A doc for 'G'" + + Member objects and __slots__ ---------------------------- @@ -1000,11 +1315,15 @@ fixed-length array of slot values. From a user point of view that has several effects: 1. Provides immediate detection of bugs due to misspelled attribute -assignments. Only attribute names specified in ``__slots__`` are allowed:: +assignments. Only attribute names specified in ``__slots__`` are allowed: + +.. testcode:: class Vehicle: __slots__ = ('id_number', 'make', 'model') +.. doctest:: + >>> auto = Vehicle() >>> auto.id_nubmer = 'VYE483814LQEX' Traceback (most recent call last): @@ -1012,11 +1331,13 @@ assignments. Only attribute names specified in ``__slots__`` are allowed:: AttributeError: 'Vehicle' object has no attribute 'id_nubmer' 2. Helps create immutable objects where descriptors manage access to private -attributes stored in ``__slots__``:: +attributes stored in ``__slots__``: + +.. testcode:: class Immutable: - __slots__ = ('_dept', '_name') # Replace instance dictionary + __slots__ = ('_dept', '_name') # Replace the instance dictionary def __init__(self, dept, name): self._dept = dept # Store to private attribute @@ -1030,7 +1351,19 @@ attributes stored in ``__slots__``:: def name(self): # Read-only descriptor return self._name - mark = Immutable('Botany', 'Mark Watney') # Create an immutable instance +.. doctest:: + + >>> mark = Immutable('Botany', 'Mark Watney') + >>> mark.dept + 'Botany' + >>> mark.dept = 'Space Pirate' + Traceback (most recent call last): + ... + AttributeError: can't set attribute + >>> mark.location = 'Mars' + Traceback (most recent call last): + ... + AttributeError: 'Immutable' object has no attribute 'location' 3. Saves memory. On a 64-bit Linux build, an instance with two attributes takes 48 bytes with ``__slots__`` and 152 bytes without. This `flyweight @@ -1038,7 +1371,9 @@ design pattern <https://en.wikipedia.org/wiki/Flyweight_pattern>`_ likely only matters when a large number of instances are going to be created. 4. Blocks tools like :func:`functools.cached_property` which require an -instance dictionary to function correctly:: +instance dictionary to function correctly: + +.. testcode:: from functools import cached_property @@ -1050,17 +1385,23 @@ instance dictionary to function correctly:: return 4 * sum((-1.0)**n / (2.0*n + 1.0) for n in reversed(range(100_000))) +.. doctest:: + >>> CP().pi Traceback (most recent call last): ... TypeError: No '__dict__' attribute on 'CP' instance to cache 'pi' property. -It's not possible to create an exact drop-in pure Python version of +It is not possible to create an exact drop-in pure Python version of ``__slots__`` because it requires direct access to C structures and control over object memory allocation. However, we can build a mostly faithful simulation where the actual C structure for slots is emulated by a private ``_slotvalues`` list. Reads and writes to that private structure are managed -by member descriptors:: +by member descriptors: + +.. testcode:: + + null = object() class Member: @@ -1074,20 +1415,30 @@ by member descriptors:: def __get__(self, obj, objtype=None): 'Emulate member_get() in Objects/descrobject.c' # Also see PyMember_GetOne() in Python/structmember.c - return obj._slotvalues[self.offset] + value = obj._slotvalues[self.offset] + if value is null: + raise AttributeError(self.name) + return value def __set__(self, obj, value): 'Emulate member_set() in Objects/descrobject.c' obj._slotvalues[self.offset] = value + def __delete__(self, obj): + 'Emulate member_delete() in Objects/descrobject.c' + value = obj._slotvalues[self.offset] + if value is null: + raise AttributeError(self.name) + obj._slotvalues[self.offset] = null + def __repr__(self): 'Emulate member_repr() in Objects/descrobject.c' return f'<Member {self.name!r} of {self.clsname!r}>' The :meth:`type.__new__` method takes care of adding member objects to class -variables. The :meth:`object.__new__` method takes care of creating instances -that have slots instead of a instance dictionary. Here is a rough equivalent -in pure Python:: +variables: + +.. testcode:: class Type(type): 'Simulate how the type metaclass adds member objects for slots' @@ -1100,6 +1451,12 @@ in pure Python:: mapping[name] = Member(name, clsname, offset) return type.__new__(mcls, clsname, bases, mapping) +The :meth:`object.__new__` method takes care of creating instances that have +slots instead of an instance dictionary. Here is a rough simulation in pure +Python: + +.. testcode:: + class Object: 'Simulate how object.__new__() allocates memory for __slots__' @@ -1107,13 +1464,35 @@ in pure Python:: 'Emulate object_new() in Objects/typeobject.c' inst = super().__new__(cls) if hasattr(cls, 'slot_names'): - inst._slotvalues = [None] * len(cls.slot_names) + empty_slots = [null] * len(cls.slot_names) + object.__setattr__(inst, '_slotvalues', empty_slots) return inst + def __setattr__(self, name, value): + 'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c' + cls = type(self) + if hasattr(cls, 'slot_names') and name not in cls.slot_names: + raise AttributeError( + f'{type(self).__name__!r} object has no attribute {name!r}' + ) + super().__setattr__(name, value) + + def __delattr__(self, name): + 'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c' + cls = type(self) + if hasattr(cls, 'slot_names') and name not in cls.slot_names: + raise AttributeError( + f'{type(self).__name__!r} object has no attribute {name!r}' + ) + super().__delattr__(name) + To use the simulation in a real class, just inherit from :class:`Object` and -set the :term:`metaclass` to :class:`Type`:: +set the :term:`metaclass` to :class:`Type`: + +.. testcode:: class H(Object, metaclass=Type): + 'Instance variables stored in slots' slot_names = ['x', 'y'] @@ -1123,17 +1502,29 @@ set the :term:`metaclass` to :class:`Type`:: At this point, the metaclass has loaded member objects for *x* and *y*:: - >>> import pprint - >>> pprint.pp(dict(vars(H))) + >>> from pprint import pp + >>> pp(dict(vars(H))) {'__module__': '__main__', + '__doc__': 'Instance variables stored in slots', 'slot_names': ['x', 'y'], '__init__': <function H.__init__ at 0x7fb5d302f9d0>, 'x': <Member 'x' of 'H'>, - 'y': <Member 'y' of 'H'>, - '__doc__': None} + 'y': <Member 'y' of 'H'>} + +.. doctest:: + :hide: + + # We test this separately because the preceding section is not + # doctestable due to the hex memory address for the __init__ function + >>> isinstance(vars(H)['x'], Member) + True + >>> isinstance(vars(H)['y'], Member) + True When instances are created, they have a ``slot_values`` list where the -attributes are stored:: +attributes are stored: + +.. doctest:: >>> h = H(10, 20) >>> vars(h) @@ -1142,8 +1533,30 @@ attributes are stored:: >>> vars(h) {'_slotvalues': [55, 20]} -Unlike the real ``__slots__``, this simulation does have an instance -dictionary just to hold the ``_slotvalues`` array. So, unlike the real code, -this simulation doesn't block assignments to misspelled attributes:: +Misspelled or unassigned attributes will raise an exception: + +.. doctest:: - >>> h.xz = 30 # For actual __slots__ this would raise an AttributeError + >>> h.xz + Traceback (most recent call last): + ... + AttributeError: 'H' object has no attribute 'xz' + +.. doctest:: + :hide: + + # Examples for deleted attributes are not shown because this section + # is already a bit lengthy. We still test that code here. + >>> del h.x + >>> hasattr(h, 'x') + False + + # Also test the code for uninitialized slots + >>> class HU(Object, metaclass=Type): + ... slot_names = ['x', 'y'] + ... + >>> hu = HU() + >>> hasattr(hu, 'x') + False + >>> hasattr(hu, 'y') + False diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst index 3a7414d735..1a6ed474ff 100644 --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -336,6 +336,7 @@ The :mod:`abc` module also provides the following functions: .. versionadded:: 3.4 .. function:: update_abstractmethods(cls) + A function to recalculate an abstract class's abstraction status. This function should be called if a class's abstract methods have been implemented or changed after it was created. Usually, this function should @@ -343,7 +344,7 @@ The :mod:`abc` module also provides the following functions: Returns *cls*, to allow usage as a class decorator. - If *cls* is not an instance of ABCMeta, does nothing. + If *cls* is not an instance of :class:`ABCMeta`, does nothing. .. note:: diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 7a7a4cf949..80e0f013df 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -57,7 +57,7 @@ be run at the command line and provides useful help messages: positional arguments: N an integer for the accumulator - optional arguments: + options: -h, --help show this help message and exit --sum sum the integers (default: find the max) @@ -217,14 +217,14 @@ The help for this program will display ``myprogram.py`` as the program name $ python myprogram.py --help usage: myprogram.py [-h] [--foo FOO] - optional arguments: + options: -h, --help show this help message and exit --foo FOO foo help $ cd .. $ python subdir/myprogram.py --help usage: myprogram.py [-h] [--foo FOO] - optional arguments: + options: -h, --help show this help message and exit --foo FOO foo help @@ -235,7 +235,7 @@ To change this default behavior, another value can be supplied using the >>> parser.print_help() usage: myprogram [-h] - optional arguments: + options: -h, --help show this help message and exit Note that the program name, whether determined from ``sys.argv[0]`` or from the @@ -249,7 +249,7 @@ specifier. >>> parser.print_help() usage: myprogram [-h] [--foo FOO] - optional arguments: + options: -h, --help show this help message and exit --foo FOO foo of the myprogram program @@ -269,7 +269,7 @@ arguments it contains:: positional arguments: bar bar help - optional arguments: + options: -h, --help show this help message and exit --foo [FOO] foo help @@ -284,7 +284,7 @@ The default message can be overridden with the ``usage=`` keyword argument:: positional arguments: bar bar help - optional arguments: + options: -h, --help show this help message and exit --foo [FOO] foo help @@ -307,7 +307,7 @@ various arguments:: A foo that bars - optional arguments: + options: -h, --help show this help message and exit By default, the description will be line-wrapped so that it fits within the @@ -329,7 +329,7 @@ argument to :class:`ArgumentParser`:: A foo that bars - optional arguments: + options: -h, --help show this help message and exit And that's how you'd foo a bar @@ -403,7 +403,7 @@ epilog_ texts in command-line help messages:: this description was indented weird but that is okay - optional arguments: + options: -h, --help show this help message and exit likewise for this epilog whose whitespace will be cleaned up and whose words @@ -432,7 +432,7 @@ should not be line-wrapped:: exactly the way I want it - optional arguments: + options: -h, --help show this help message and exit :class:`RawTextHelpFormatter` maintains whitespace for all sorts of help text, @@ -454,7 +454,7 @@ default values to each of the argument help messages:: positional arguments: bar BAR! (default: [1, 2, 3]) - optional arguments: + options: -h, --help show this help message and exit --foo FOO FOO! (default: 42) @@ -473,7 +473,7 @@ as the regular formatter does):: positional arguments: float - optional arguments: + options: -h, --help show this help message and exit --foo int @@ -592,7 +592,7 @@ older arguments with the same option string. To get this behavior, the value >>> parser.print_help() usage: PROG [-h] [-f FOO] [--foo FOO] - optional arguments: + options: -h, --help show this help message and exit -f FOO old foo help --foo FOO new foo help @@ -623,7 +623,7 @@ help will be printed: $ python myprogram.py --help usage: myprogram.py [-h] [--foo FOO] - optional arguments: + options: -h, --help show this help message and exit --foo FOO foo help @@ -636,7 +636,7 @@ This can be achieved by passing ``False`` as the ``add_help=`` argument to >>> parser.print_help() usage: PROG [--foo FOO] - optional arguments: + options: --foo FOO foo help The help option is typically ``-h/--help``. The exception to this is @@ -649,7 +649,7 @@ the help options:: >>> parser.print_help() usage: PROG [+h] - optional arguments: + options: +h, ++help show this help message and exit @@ -696,7 +696,7 @@ The add_argument() method * const_ - A constant value required by some action_ and nargs_ selections. * default_ - The value produced if the argument is absent from the - command line. + command line and if it is absent from the namespace object. * type_ - The type to which the command-line argument should be converted. @@ -1006,6 +1006,14 @@ was not present at the command line:: >>> parser.parse_args([]) Namespace(foo=42) +If the target namespace already has an attribute set, the action *default* +will not over write it:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default=42) + >>> parser.parse_args([], namespace=argparse.Namespace(foo=101)) + Namespace(foo=101) + If the ``default`` value is a string, the parser parses the value as if it were a command-line argument. In particular, the parser applies any type_ conversion argument, if provided, before setting the attribute on the @@ -1042,63 +1050,70 @@ command-line argument was not present:: type ^^^^ -By default, :class:`ArgumentParser` objects read command-line arguments in as simple +By default, the parser reads command-line arguments in as simple strings. However, quite often the command-line string should instead be -interpreted as another type, like a :class:`float` or :class:`int`. The -``type`` keyword argument of :meth:`~ArgumentParser.add_argument` allows any -necessary type-checking and type conversions to be performed. Common built-in -types and functions can be used directly as the value of the ``type`` argument:: +interpreted as another type, such as a :class:`float` or :class:`int`. The +``type`` keyword for :meth:`~ArgumentParser.add_argument` allows any +necessary type-checking and type conversions to be performed. - >>> parser = argparse.ArgumentParser() - >>> parser.add_argument('foo', type=int) - >>> parser.add_argument('bar', type=open) - >>> parser.parse_args('2 temp.txt'.split()) - Namespace(bar=<_io.TextIOWrapper name='temp.txt' encoding='UTF-8'>, foo=2) +If the type_ keyword is used with the default_ keyword, the type converter +is only applied if the default is a string. -See the section on the default_ keyword argument for information on when the -``type`` argument is applied to default arguments. +The argument to ``type`` can be any callable that accepts a single string. +If the function raises :exc:`ArgumentTypeError`, :exc:`TypeError`, or +:exc:`ValueError`, the exception is caught and a nicely formatted error +message is displayed. No other exception types are handled. -To ease the use of various types of files, the argparse module provides the -factory FileType which takes the ``mode=``, ``bufsize=``, ``encoding=`` and -``errors=`` arguments of the :func:`open` function. For example, -``FileType('w')`` can be used to create a writable file:: +Common built-in types and functions can be used as type converters: - >>> parser = argparse.ArgumentParser() - >>> parser.add_argument('bar', type=argparse.FileType('w')) - >>> parser.parse_args(['out.txt']) - Namespace(bar=<_io.TextIOWrapper name='out.txt' encoding='UTF-8'>) - -``type=`` can take any callable that takes a single string argument and returns -the converted value:: - - >>> def perfect_square(string): - ... value = int(string) - ... sqrt = math.sqrt(value) - ... if sqrt != int(sqrt): - ... msg = "%r is not a perfect square" % string - ... raise argparse.ArgumentTypeError(msg) - ... return value +.. testcode:: + + import argparse + import pathlib + + parser = argparse.ArgumentParser() + parser.add_argument('count', type=int) + parser.add_argument('distance', type=float) + parser.add_argument('street', type=ascii) + parser.add_argument('code_point', type=ord) + parser.add_argument('source_file', type=open) + parser.add_argument('dest_file', type=argparse.FileType('w', encoding='latin-1')) + parser.add_argument('datapath', type=pathlib.Path) + +User defined functions can be used as well: + +.. doctest:: + + >>> def hyphenated(string): + ... return '-'.join([word[:4] for word in string.casefold().split()]) ... - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', type=perfect_square) - >>> parser.parse_args(['9']) - Namespace(foo=9) - >>> parser.parse_args(['7']) - usage: PROG [-h] foo - PROG: error: argument foo: '7' is not a perfect square + >>> parser = argparse.ArgumentParser() + >>> _ = parser.add_argument('short_title', type=hyphenated) + >>> parser.parse_args(['"The Tale of Two Cities"']) + Namespace(short_title='"the-tale-of-two-citi') -The choices_ keyword argument may be more convenient for type checkers that -simply check against a range of values:: +The :func:`bool` function is not recommended as a type converter. All it does +is convert empty strings to ``False`` and non-empty strings to ``True``. +This is usually not what is desired. - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', type=int, choices=range(5, 10)) - >>> parser.parse_args(['7']) - Namespace(foo=7) - >>> parser.parse_args(['11']) - usage: PROG [-h] {5,6,7,8,9} - PROG: error: argument foo: invalid choice: 11 (choose from 5, 6, 7, 8, 9) +In general, the ``type`` keyword is a convenience that should only be used for +simple conversions that can only raise one of the three supported exceptions. +Anything with more interesting error-handling or resource management should be +done downstream after the arguments are parsed. + +For example, JSON or YAML conversions have complex error cases that require +better reporting than can be given by the ``type`` keyword. An +:exc:`~json.JSONDecodeError` would not be well formatted and a +:exc:`FileNotFound` exception would not be handled at all. + +Even :class:`~argparse.FileType` has its limitations for use with the ``type`` +keyword. If one argument uses *FileType* and then a subsequent argument fails, +an error is reported but the file is not automatically closed. In this case, it +would be better to wait until after the parser has run and then use the +:keyword:`with`-statement to manage the files. -See the choices_ section for more details. +For type checkers that simply check against a fixed set of values, consider +using the choices_ keyword instead. choices @@ -1133,20 +1148,14 @@ container should match the type_ specified:: Any container can be passed as the *choices* value, so :class:`list` objects, :class:`set` objects, and custom containers are all supported. -This includes :class:`enum.Enum`, which could be used to restrain -argument's choices; if we reuse previous rock/paper/scissors game example, -this could be as follows:: - - >>> from enum import Enum - >>> class GameMove(Enum): - ... ROCK = 'rock' - ... PAPER = 'paper' - ... SCISSORS = 'scissors' - ... - >>> parser = argparse.ArgumentParser(prog='game.py') - >>> parser.add_argument('move', type=GameMove, choices=GameMove) - >>> parser.parse_args(['rock']) - Namespace(move=<GameMove.ROCK: 'rock'>) + +Use of :class:`enum.Enum` is not recommended because it is difficult to +control its appearance in usage, help, and error messages. + +Formatted choices overrides the default *metavar* which is normally derived +from *dest*. This is usually what you want because the user never sees the +*dest* parameter. If this display isn't desirable (perhaps because there are +many choices), just specify an explicit metavar_. required @@ -1194,7 +1203,7 @@ argument:: positional arguments: bar one of the bars to be frobbled - optional arguments: + options: -h, --help show this help message and exit --foo foo the bars before frobbling @@ -1212,7 +1221,7 @@ specifiers include the program name, ``%(prog)s`` and most keyword arguments to positional arguments: bar the bar to frobble (default: 42) - optional arguments: + options: -h, --help show this help message and exit As the help string supports %-formatting, if you want a literal ``%`` to appear @@ -1226,7 +1235,7 @@ setting the ``help`` value to ``argparse.SUPPRESS``:: >>> parser.print_help() usage: frobble [-h] - optional arguments: + options: -h, --help show this help message and exit @@ -1253,7 +1262,7 @@ will be referred to as ``FOO``. An example:: positional arguments: bar - optional arguments: + options: -h, --help show this help message and exit --foo FOO @@ -1270,7 +1279,7 @@ An alternative name can be specified with ``metavar``:: positional arguments: XXX - optional arguments: + options: -h, --help show this help message and exit --foo YYY @@ -1288,7 +1297,7 @@ arguments:: >>> parser.print_help() usage: PROG [-h] [-x X X] [--foo bar baz] - optional arguments: + options: -h, --help show this help message and exit -x X X --foo bar baz @@ -1692,7 +1701,7 @@ Sub-commands a a help b b help - optional arguments: + options: -h, --help show this help message and exit --foo foo help @@ -1702,13 +1711,13 @@ Sub-commands positional arguments: bar bar help - optional arguments: + options: -h, --help show this help message and exit >>> parser.parse_args(['b', '--help']) usage: PROG b [-h] [--baz {X,Y,Z}] - optional arguments: + options: -h, --help show this help message and exit --baz {X,Y,Z} baz help @@ -1725,7 +1734,7 @@ Sub-commands >>> parser.parse_args(['-h']) usage: [-h] {foo,bar} ... - optional arguments: + options: -h, --help show this help message and exit subcommands: diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index b8688ae61a..9149a53e0d 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -80,12 +80,13 @@ Node classes end_col_offset Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have - :attr:`lineno`, :attr:`col_offset`, :attr:`lineno`, and :attr:`col_offset` - attributes. The :attr:`lineno` and :attr:`end_lineno` are the first and - last line numbers of source text span (1-indexed so the first line is line 1) - and the :attr:`col_offset` and :attr:`end_col_offset` are the corresponding - UTF-8 byte offsets of the first and last tokens that generated the node. - The UTF-8 offset is recorded because the parser uses UTF-8 internally. + :attr:`lineno`, :attr:`col_offset`, :attr:`end_lineno`, and + :attr:`end_col_offset` attributes. The :attr:`lineno` and :attr:`end_lineno` + are the first and last line numbers of source text span (1-indexed so the + first line is line 1) and the :attr:`col_offset` and :attr:`end_col_offset` + are the corresponding UTF-8 byte offsets of the first and last tokens that + generated the node. The UTF-8 offset is recorded because the parser uses + UTF-8 internally. Note that the end positions are not required by the compiler and are therefore optional. The end offset is *after* the last symbol, for example @@ -1575,7 +1576,7 @@ and classes for traversing abstract syntax trees: Safely evaluate an expression node or a string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, - dicts, sets, booleans, and ``None``. + dicts, sets, booleans, ``None`` and ``Ellipsis``. This can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself. It is not @@ -1587,6 +1588,10 @@ and classes for traversing abstract syntax trees: sufficiently large/complex string due to stack depth limitations in Python's AST compiler. + It can raise :exc:`ValueError`, :exc:`TypeError`, :exc:`SyntaxError`, + :exc:`MemoryError` and :exc:`RecursionError` depending on the malformed + input. + .. versionchanged:: 3.2 Now allows bytes and set literals. diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index b1e73189a7..7de5a0ab25 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -321,7 +321,7 @@ Creating Futures and Tasks .. versionadded:: 3.5.2 -.. method:: loop.create_task(coro, \*, name=None) +.. method:: loop.create_task(coro, *, name=None) Schedule the execution of a :ref:`coroutine`. Return a :class:`Task` object. @@ -356,7 +356,7 @@ Opening network connections ^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. coroutinemethod:: loop.create_connection(protocol_factory, \ - host=None, port=None, \*, ssl=None, \ + host=None, port=None, *, ssl=None, \ family=0, proto=0, flags=0, sock=None, \ local_addr=None, server_hostname=None, \ ssl_handshake_timeout=None, \ @@ -482,7 +482,7 @@ Opening network connections that can be used directly in async/await code. .. coroutinemethod:: loop.create_datagram_endpoint(protocol_factory, \ - local_addr=None, remote_addr=None, \*, \ + local_addr=None, remote_addr=None, *, \ family=0, proto=0, flags=0, \ reuse_address=None, reuse_port=None, \ allow_broadcast=None, sock=None) @@ -559,7 +559,7 @@ Opening network connections Added support for Windows. .. coroutinemethod:: loop.create_unix_connection(protocol_factory, \ - path=None, \*, ssl=None, sock=None, \ + path=None, *, ssl=None, sock=None, \ server_hostname=None, ssl_handshake_timeout=None) Create a Unix connection. @@ -592,7 +592,7 @@ Creating network servers ^^^^^^^^^^^^^^^^^^^^^^^^ .. coroutinemethod:: loop.create_server(protocol_factory, \ - host=None, port=None, \*, \ + host=None, port=None, *, \ family=socket.AF_UNSPEC, \ flags=socket.AI_PASSIVE, \ sock=None, backlog=100, ssl=None, \ @@ -683,7 +683,7 @@ Creating network servers .. coroutinemethod:: loop.create_unix_server(protocol_factory, path=None, \ - \*, sock=None, backlog=100, ssl=None, \ + *, sock=None, backlog=100, ssl=None, \ ssl_handshake_timeout=None, start_serving=True) Similar to :meth:`loop.create_server` but works with the @@ -708,7 +708,7 @@ Creating network servers The *path* parameter can now be a :class:`~pathlib.Path` object. .. coroutinemethod:: loop.connect_accepted_socket(protocol_factory, \ - sock, \*, ssl=None, ssl_handshake_timeout=None) + sock, *, ssl=None, ssl_handshake_timeout=None) Wrap an already accepted connection into a transport/protocol pair. @@ -773,7 +773,7 @@ TLS Upgrade ^^^^^^^^^^^ .. coroutinemethod:: loop.start_tls(transport, protocol, \ - sslcontext, \*, server_side=False, \ + sslcontext, *, server_side=False, \ server_hostname=None, ssl_handshake_timeout=None) Upgrade an existing transport-based connection to TLS. @@ -806,7 +806,7 @@ TLS Upgrade Watching file descriptors ^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: loop.add_reader(fd, callback, \*args) +.. method:: loop.add_reader(fd, callback, *args) Start monitoring the *fd* file descriptor for read availability and invoke *callback* with the specified arguments once *fd* is available for @@ -816,7 +816,7 @@ Watching file descriptors Stop monitoring the *fd* file descriptor for read availability. -.. method:: loop.add_writer(fd, callback, \*args) +.. method:: loop.add_writer(fd, callback, *args) Start monitoring the *fd* file descriptor for write availability and invoke *callback* with the specified arguments once *fd* is available for @@ -930,7 +930,7 @@ convenient. :meth:`loop.create_server` and :func:`start_server`. .. coroutinemethod:: loop.sock_sendfile(sock, file, offset=0, count=None, \ - \*, fallback=True) + *, fallback=True) Send a file using high-performance :mod:`os.sendfile` if possible. Return the total number of bytes sent. @@ -964,7 +964,7 @@ convenient. DNS ^^^ -.. coroutinemethod:: loop.getaddrinfo(host, port, \*, family=0, \ +.. coroutinemethod:: loop.getaddrinfo(host, port, *, family=0, \ type=0, proto=0, flags=0) Asynchronous version of :meth:`socket.getaddrinfo`. @@ -1029,7 +1029,7 @@ Working with pipes Unix signals ^^^^^^^^^^^^ -.. method:: loop.add_signal_handler(signum, callback, \*args) +.. method:: loop.add_signal_handler(signum, callback, *args) Set *callback* as the handler for the *signum* signal. @@ -1064,7 +1064,7 @@ Unix signals Executing code in thread or process pools ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. awaitablemethod:: loop.run_in_executor(executor, func, \*args) +.. awaitablemethod:: loop.run_in_executor(executor, func, *args) Arrange for *func* to be called in the specified executor. @@ -1184,10 +1184,13 @@ Allows customizing how exceptions are handled in the event loop. * 'message': Error message; * 'exception' (optional): Exception object; * 'future' (optional): :class:`asyncio.Future` instance; + * 'task' (optional): :class:`asyncio.Task` instance; * 'handle' (optional): :class:`asyncio.Handle` instance; * 'protocol' (optional): :ref:`Protocol <asyncio-protocol>` instance; * 'transport' (optional): :ref:`Transport <asyncio-transport>` instance; - * 'socket' (optional): :class:`socket.socket` instance. + * 'socket' (optional): :class:`socket.socket` instance; + * 'asyncgen' (optional): Asynchronous generator that caused + the exception. .. note:: @@ -1234,9 +1237,9 @@ async/await code consider using the high-level subprocesses. See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>` for details. -.. coroutinemethod:: loop.subprocess_exec(protocol_factory, \*args, \ +.. coroutinemethod:: loop.subprocess_exec(protocol_factory, *args, \ stdin=subprocess.PIPE, stdout=subprocess.PIPE, \ - stderr=subprocess.PIPE, \*\*kwargs) + stderr=subprocess.PIPE, **kwargs) Create a subprocess from one or more string arguments specified by *args*. @@ -1316,9 +1319,9 @@ async/await code consider using the high-level conforms to the :class:`asyncio.SubprocessTransport` base class and *protocol* is an object instantiated by the *protocol_factory*. -.. coroutinemethod:: loop.subprocess_shell(protocol_factory, cmd, \*, \ +.. coroutinemethod:: loop.subprocess_shell(protocol_factory, cmd, *, \ stdin=subprocess.PIPE, stdout=subprocess.PIPE, \ - stderr=subprocess.PIPE, \*\*kwargs) + stderr=subprocess.PIPE, **kwargs) Create a subprocess from *cmd*, which can be a :class:`str` or a :class:`bytes` string encoded to the diff --git a/Doc/library/asyncio-future.rst b/Doc/library/asyncio-future.rst index e1ac18eaf0..939d4c1a84 100644 --- a/Doc/library/asyncio-future.rst +++ b/Doc/library/asyncio-future.rst @@ -31,7 +31,7 @@ Future Functions .. versionadded:: 3.5 -.. function:: ensure_future(obj, \*, loop=None) +.. function:: ensure_future(obj, *, loop=None) Return: @@ -58,7 +58,7 @@ Future Functions The function accepts any :term:`awaitable` object. -.. function:: wrap_future(future, \*, loop=None) +.. function:: wrap_future(future, *, loop=None) Wrap a :class:`concurrent.futures.Future` object in a :class:`asyncio.Future` object. @@ -67,7 +67,7 @@ Future Functions Future Object ============= -.. class:: Future(\*, loop=None) +.. class:: Future(*, loop=None) A Future represents an eventual result of an asynchronous operation. Not thread-safe. diff --git a/Doc/library/asyncio-policy.rst b/Doc/library/asyncio-policy.rst index 88e69ceff9..ef6a058850 100644 --- a/Doc/library/asyncio-policy.rst +++ b/Doc/library/asyncio-policy.rst @@ -159,7 +159,7 @@ implementation used by the asyncio event loop: .. class:: AbstractChildWatcher - .. method:: add_child_handler(pid, callback, \*args) + .. method:: add_child_handler(pid, callback, *args) Register a new child handler. @@ -209,7 +209,7 @@ implementation used by the asyncio event loop: It works reliably even when the asyncio event loop is run in a non-main OS thread. There is no noticeable overhead when handling a big number of children (*O(1)* each - time a child terminates), but stating a thread per process requires extra memory. + time a child terminates), but starting a thread per process requires extra memory. This watcher is used by default. diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 524560b691..289ad1b014 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -23,7 +23,7 @@ See also the `Examples`_ section below. Queue ===== -.. class:: Queue(maxsize=0, \*, loop=None) +.. class:: Queue(maxsize=0) A first in, first out (FIFO) queue. @@ -36,9 +36,6 @@ Queue the queue is always known and can be returned by calling the :meth:`qsize` method. - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - This class is :ref:`not thread safe <asyncio-multithreading>`. diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index b76ed379c7..9b456c1435 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -48,9 +48,9 @@ The following top-level asyncio functions can be used to create and work with streams: -.. coroutinefunction:: open_connection(host=None, port=None, \*, \ - loop=None, limit=None, ssl=None, family=0, \ - proto=0, flags=0, sock=None, local_addr=None, \ +.. coroutinefunction:: open_connection(host=None, port=None, *, \ + limit=None, ssl=None, family=0, proto=0, \ + flags=0, sock=None, local_addr=None, \ server_hostname=None, ssl_handshake_timeout=None) Establish a network connection and return a pair of @@ -59,9 +59,6 @@ and work with streams: The returned *reader* and *writer* objects are instances of :class:`StreamReader` and :class:`StreamWriter` classes. - The *loop* argument is optional and can always be determined - automatically when this function is awaited from a coroutine. - *limit* determines the buffer size limit used by the returned :class:`StreamReader` instance. By default the *limit* is set to 64 KiB. @@ -74,7 +71,7 @@ and work with streams: The *ssl_handshake_timeout* parameter. .. coroutinefunction:: start_server(client_connected_cb, host=None, \ - port=None, \*, loop=None, limit=None, \ + port=None, *, limit=None, \ family=socket.AF_UNSPEC, \ flags=socket.AI_PASSIVE, sock=None, \ backlog=100, ssl=None, reuse_address=None, \ @@ -92,9 +89,6 @@ and work with streams: :ref:`coroutine function <coroutine>`; if it is a coroutine function, it will be automatically scheduled as a :class:`Task`. - The *loop* argument is optional and can always be determined - automatically when this method is awaited from a coroutine. - *limit* determines the buffer size limit used by the returned :class:`StreamReader` instance. By default the *limit* is set to 64 KiB. @@ -109,9 +103,9 @@ and work with streams: .. rubric:: Unix Sockets -.. coroutinefunction:: open_unix_connection(path=None, \*, loop=None, \ - limit=None, ssl=None, sock=None, \ - server_hostname=None, ssl_handshake_timeout=None) +.. coroutinefunction:: open_unix_connection(path=None, *, limit=None, \ + ssl=None, sock=None, server_hostname=None, \ + ssl_handshake_timeout=None) Establish a Unix socket connection and return a pair of ``(reader, writer)``. @@ -132,9 +126,8 @@ and work with streams: .. coroutinefunction:: start_unix_server(client_connected_cb, path=None, \ - \*, loop=None, limit=None, sock=None, \ - backlog=100, ssl=None, ssl_handshake_timeout=None, \ - start_serving=True) + *, limit=None, sock=None, backlog=100, ssl=None, \ + ssl_handshake_timeout=None, start_serving=True) Start a Unix socket server. @@ -192,7 +185,7 @@ StreamReader can be read. Use the :attr:`IncompleteReadError.partial` attribute to get the partially read data. - .. coroutinemethod:: readuntil(separator=b'\\n') + .. coroutinemethod:: readuntil(separator=b'\n') Read data from the stream until *separator* is found. diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst index b0330349df..f955a292e1 100644 --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -61,9 +61,8 @@ See also the `Examples`_ subsection. Creating Subprocesses ===================== -.. coroutinefunction:: create_subprocess_exec(program, \*args, stdin=None, \ - stdout=None, stderr=None, loop=None, \ - limit=None, \*\*kwds) +.. coroutinefunction:: create_subprocess_exec(program, *args, stdin=None, \ + stdout=None, stderr=None, limit=None, **kwds) Create a subprocess. @@ -76,13 +75,9 @@ Creating Subprocesses See the documentation of :meth:`loop.subprocess_exec` for other parameters. - .. deprecated-removed:: 3.8 3.10 - - The *loop* parameter. .. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \ - stdout=None, stderr=None, loop=None, \ - limit=None, \*\*kwds) + stdout=None, stderr=None, limit=None, **kwds) Run the *cmd* shell command. @@ -104,10 +99,6 @@ Creating Subprocesses escape whitespace and special shell characters in strings that are going to be used to construct shell commands. - .. deprecated-removed:: 3.8 3.10 - - The *loop* parameter. - .. note:: Subprocesses are available for Windows if a :class:`ProactorEventLoop` is diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst index 84a52cb2d5..d12630afc6 100644 --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -36,7 +36,7 @@ asyncio has the following basic synchronization primitives: Lock ==== -.. class:: Lock(\*, loop=None) +.. class:: Lock() Implements a mutex lock for asyncio tasks. Not thread-safe. @@ -63,9 +63,6 @@ Lock finally: lock.release() - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - .. coroutinemethod:: acquire() Acquire the lock. @@ -96,7 +93,7 @@ Lock Event ===== -.. class:: Event(\*, loop=None) +.. class:: Event() An event object. Not thread-safe. @@ -104,14 +101,10 @@ Event that some event has happened. An Event object manages an internal flag that can be set to *true* - with the :meth:`set` method and reset to *false* with the - :meth:`clear` method. The :meth:`wait` method blocks until the + with the :meth:`~Event.set` method and reset to *false* with the + :meth:`clear` method. The :meth:`~Event.wait` method blocks until the flag is set to *true*. The flag is set to *false* initially. - - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - .. _asyncio_example_sync_event: Example:: @@ -142,7 +135,7 @@ Event Wait until the event is set. If the event is set, return ``True`` immediately. - Otherwise block until another task calls :meth:`set`. + Otherwise block until another task calls :meth:`~Event.set`. .. method:: set() @@ -155,8 +148,8 @@ Event Clear (unset) the event. - Tasks awaiting on :meth:`wait` will now block until the - :meth:`set` method is called again. + Tasks awaiting on :meth:`~Event.wait` will now block until the + :meth:`~Event.set` method is called again. .. method:: is_set() @@ -166,7 +159,7 @@ Event Condition ========= -.. class:: Condition(lock=None, \*, loop=None) +.. class:: Condition(lock=None) A Condition object. Not thread-safe. @@ -184,10 +177,6 @@ Condition ``None``. In the latter case a new Lock object is created automatically. - - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - The preferred way to use a Condition is an :keyword:`async with` statement:: @@ -270,7 +259,7 @@ Condition Semaphore ========= -.. class:: Semaphore(value=1, \*, loop=None) +.. class:: Semaphore(value=1) A Semaphore object. Not thread-safe. @@ -284,10 +273,6 @@ Semaphore internal counter (``1`` by default). If the given value is less than ``0`` a :exc:`ValueError` is raised. - - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - The preferred way to use a Semaphore is an :keyword:`async with` statement:: @@ -332,7 +317,7 @@ Semaphore BoundedSemaphore ================ -.. class:: BoundedSemaphore(value=1, \*, loop=None) +.. class:: BoundedSemaphore(value=1) A bounded semaphore object. Not thread-safe. @@ -340,10 +325,6 @@ BoundedSemaphore a :exc:`ValueError` in :meth:`~Semaphore.release` if it increases the internal counter above the initial *value*. - - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - --------- diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index eb51c704ce..45e58437e0 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -210,7 +210,7 @@ is :meth:`loop.run_in_executor`. Running an asyncio Program ========================== -.. function:: run(coro, \*, debug=False) +.. function:: run(coro, *, debug=False) Execute the :term:`coroutine` *coro* and return the result. @@ -247,7 +247,7 @@ Running an asyncio Program Creating Tasks ============== -.. function:: create_task(coro, \*, name=None) +.. function:: create_task(coro, *, name=None) Wrap the *coro* :ref:`coroutine <coroutine>` into a :class:`Task` and schedule its execution. Return the Task object. @@ -283,7 +283,7 @@ Creating Tasks Sleeping ======== -.. coroutinefunction:: sleep(delay, result=None, \*, loop=None) +.. coroutinefunction:: sleep(delay, result=None) Block for *delay* seconds. @@ -293,9 +293,6 @@ Sleeping ``sleep()`` always suspends the current task, allowing other tasks to run. - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - .. _asyncio_example_sleep: Example of coroutine displaying the current date every second @@ -319,7 +316,7 @@ Sleeping Running Tasks Concurrently ========================== -.. awaitablefunction:: gather(\*aws, loop=None, return_exceptions=False) +.. awaitablefunction:: gather(*aws, return_exceptions=False) Run :ref:`awaitable objects <asyncio-awaitables>` in the *aws* sequence *concurrently*. @@ -348,9 +345,6 @@ Running Tasks Concurrently cancellation of one submitted Task/Future to cause other Tasks/Futures to be cancelled. - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - .. _asyncio_example_gather: Example:: @@ -403,7 +397,7 @@ Running Tasks Concurrently Shielding From Cancellation =========================== -.. awaitablefunction:: shield(aw, \*, loop=None) +.. awaitablefunction:: shield(aw) Protect an :ref:`awaitable object <asyncio-awaitables>` from being :meth:`cancelled <Task.cancel>`. @@ -436,14 +430,11 @@ Shielding From Cancellation except CancelledError: res = None - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - Timeouts ======== -.. coroutinefunction:: wait_for(aw, timeout, \*, loop=None) +.. coroutinefunction:: wait_for(aw, timeout) Wait for the *aw* :ref:`awaitable <asyncio-awaitables>` to complete with a timeout. @@ -466,9 +457,6 @@ Timeouts If the wait is cancelled, the future *aw* is also cancelled. - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - .. _asyncio_example_waitfor: Example:: @@ -500,8 +488,7 @@ Timeouts Waiting Primitives ================== -.. coroutinefunction:: wait(aws, \*, loop=None, timeout=None,\ - return_when=ALL_COMPLETED) +.. coroutinefunction:: wait(aws, *, timeout=None, return_when=ALL_COMPLETED) Run :ref:`awaitable objects <asyncio-awaitables>` in the *aws* iterable concurrently and block until the condition specified @@ -553,10 +540,6 @@ Waiting Primitives ``wait()`` directly is deprecated as it leads to :ref:`confusing behavior <asyncio_example_wait_coroutine>`. - .. deprecated-removed:: 3.8 3.10 - - The *loop* parameter. - .. _asyncio_example_wait_coroutine: .. note:: @@ -590,7 +573,7 @@ Waiting Primitives deprecated. -.. function:: as_completed(aws, \*, loop=None, timeout=None) +.. function:: as_completed(aws, *, timeout=None) Run :ref:`awaitable objects <asyncio-awaitables>` in the *aws* iterable concurrently. Return an iterator of coroutines. @@ -600,9 +583,6 @@ Waiting Primitives Raises :exc:`asyncio.TimeoutError` if the timeout occurs before all Futures are done. - .. deprecated-removed:: 3.8 3.10 - The *loop* parameter. - Example:: for coro in as_completed(aws): @@ -613,12 +593,12 @@ Waiting Primitives Running in Threads ================== -.. coroutinefunction:: to_thread(func, /, \*args, \*\*kwargs) +.. coroutinefunction:: to_thread(func, /, *args, **kwargs) Asynchronously run function *func* in a separate thread. Any \*args and \*\*kwargs supplied for this function are directly passed - to *func*. Also, the current :class:`contextvars.Context` is propogated, + to *func*. Also, the current :class:`contextvars.Context` is propagated, allowing context variables from the event loop thread to be accessed in the separate thread. @@ -743,7 +723,7 @@ Introspection Task Object =========== -.. class:: Task(coro, \*, loop=None, name=None) +.. class:: Task(coro, *, loop=None, name=None) A :class:`Future-like <Future>` object that runs a Python :ref:`coroutine <coroutine>`. Not thread-safe. @@ -909,7 +889,7 @@ Task Object See the documentation of :meth:`Future.remove_done_callback` for more details. - .. method:: get_stack(\*, limit=None) + .. method:: get_stack(*, limit=None) Return the list of stack frames for this Task. @@ -930,7 +910,7 @@ Task Object stack are returned, but the oldest frames of a traceback are returned. (This matches the behavior of the traceback module.) - .. method:: print_stack(\*, limit=None, file=None) + .. method:: print_stack(*, limit=None, file=None) Print the stack or traceback for this Task. diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index 2f24bb6391..25b3a4ca29 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -199,7 +199,7 @@ The modern interface provides: .. versionadded:: 3.4 -.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \\t\\n\\r\\v') +.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \t\n\r\v') Decode the Ascii85 encoded :term:`bytes-like object` or ASCII string *b* and return the decoded :class:`bytes`. diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 85cdc16a7d..637baf49da 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -266,7 +266,6 @@ Below are some examples of typical usage of the :mod:`bz2` module. Using :func:`compress` and :func:`decompress` to demonstrate round-trip compression: >>> import bz2 - >>> data = b"""\ ... Donec rhoncus quis sapien sit amet molestie. Fusce scelerisque vel augue ... nec ullamcorper. Nam rutrum pretium placerat. Aliquam vel tristique lorem, @@ -275,11 +274,9 @@ Using :func:`compress` and :func:`decompress` to demonstrate round-trip compress ... Aliquam pharetra lacus non risus vehicula rutrum. Maecenas aliquam leo ... felis. Pellentesque semper nunc sit amet nibh ullamcorper, ac elementum ... dolor luctus. Curabitur lacinia mi ornare consectetur vestibulum.""" - >>> c = bz2.compress(data) >>> len(data) / len(c) # Data compression ratio 1.513595166163142 - >>> d = bz2.decompress(c) >>> data == d # Check equality to original object after round-trip True @@ -287,7 +284,6 @@ Using :func:`compress` and :func:`decompress` to demonstrate round-trip compress Using :class:`BZ2Compressor` for incremental compression: >>> import bz2 - >>> def gen_data(chunks=10, chunksize=1000): ... """Yield incremental blocks of chunksize bytes.""" ... for _ in range(chunks): @@ -310,7 +306,6 @@ while ordered, repetitive data usually yields a high compression ratio. Writing and reading a bzip2-compressed file in binary mode: >>> import bz2 - >>> data = b"""\ ... Donec rhoncus quis sapien sit amet molestie. Fusce scelerisque vel augue ... nec ullamcorper. Nam rutrum pretium placerat. Aliquam vel tristique lorem, @@ -319,14 +314,11 @@ Writing and reading a bzip2-compressed file in binary mode: ... Aliquam pharetra lacus non risus vehicula rutrum. Maecenas aliquam leo ... felis. Pellentesque semper nunc sit amet nibh ullamcorper, ac elementum ... dolor luctus. Curabitur lacinia mi ornare consectetur vestibulum.""" - >>> with bz2.open("myfile.bz2", "wb") as f: ... # Write compressed data to file ... unused = f.write(data) - >>> with bz2.open("myfile.bz2", "rb") as f: ... # Decompress data from file ... content = f.read() - >>> content == data # Check equality to original object after round-trip True diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index db0e25bb07..2345e78a17 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -291,7 +291,7 @@ Notes on using :class:`Set` and :class:`MutableSet` as a mixin: :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set. If the :class:`Set` mixin is being used in a class with a different constructor signature, you will need to override :meth:`_from_iterable` - with a classmethod that can construct new instances from + with a classmethod or regular method that can construct new instances from an iterable argument. (2) diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index f538da5e1c..1293f542b0 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -862,6 +862,9 @@ they add the ability to access fields by name instead of position index. Named tuple instances do not have per-instance dictionaries, so they are lightweight and require no more memory than regular tuples. + To support pickling, the named tuple class should be assigned to a variable + that matches *typename*. + .. versionchanged:: 3.1 Added support for *rename*. diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 9b914b1f0d..5c6e68f930 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -148,7 +148,7 @@ runtime. Public functions ---------------- -.. function:: compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, \*, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False) +.. function:: compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` files along the way. Return a true value if all the files compiled successfully, @@ -231,7 +231,7 @@ Public functions Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments. Default value of *maxlevels* was changed from ``10`` to ``sys.getrecursionlimit()`` -.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, \*, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False) +.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False) Compile the file with path *fullname*. Return a true value if the file compiled successfully, and a false value otherwise. diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 675a9ffdd0..d57f8ce23d 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -67,7 +67,7 @@ Executor Objects .. versionchanged:: 3.5 Added the *chunksize* argument. - .. method:: shutdown(wait=True, \*, cancel_futures=False) + .. method:: shutdown(wait=True, *, cancel_futures=False) Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to @@ -236,9 +236,9 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. An :class:`Executor` subclass that executes calls asynchronously using a pool of at most *max_workers* processes. If *max_workers* is ``None`` or not given, it will default to the number of processors on the machine. - If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError` + If *max_workers* is less than or equal to ``0``, then a :exc:`ValueError` will be raised. - On Windows, *max_workers* must be equal or lower than ``61``. If it is not + On Windows, *max_workers* must be less than or equal to ``61``. If it is not then :exc:`ValueError` will be raised. If *max_workers* is ``None``, then the default chosen will be at most ``61``, even if more processors are available. @@ -250,7 +250,7 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. each worker process; *initargs* is a tuple of arguments passed to the initializer. Should *initializer* raise an exception, all currently pending jobs will raise a :exc:`~concurrent.futures.process.BrokenProcessPool`, - as well any attempt to submit more jobs to the pool. + as well as any attempt to submit more jobs to the pool. .. versionchanged:: 3.3 When one of the worker processes terminates abruptly, a diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index e42f5a9328..91edbba728 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -126,6 +126,31 @@ Functions and classes provided: .. versionadded:: 3.7 + Context managers defined with :func:`asynccontextmanager` can be used + either as decorators or with :keyword:`async with` statements:: + + import time + + async def timeit(): + now = time.monotonic() + try: + yield + finally: + print(f'it took {time.monotonic() - now}s to run') + + @timeit() + async def main(): + # ... async code ... + + When used as a decorator, a new generator instance is implicitly created on + each function call. This allows the otherwise "one-shot" context managers + created by :func:`asynccontextmanager` to meet the requirement that context + managers support multiple invocations in order to be used as decorators. + + .. versionchanged:: 3.10 + Async context managers created with :func:`asynccontextmanager` can + be used as decorators. + .. function:: closing(thing) @@ -218,8 +243,26 @@ Functions and classes provided: with cm as file: # Perform processing on the file + It can also be used as a stand-in for + :ref:`asynchronous context managers <async-context-managers>`:: + + async def send_http(session=None): + if not session: + # If no http session, create it with aiohttp + cm = aiohttp.ClientSession() + else: + # Caller is responsible for closing the session + cm = nullcontext(session) + + async with cm as session: + # Send http requests with session + .. versionadded:: 3.7 + .. versionchanged:: 3.10 + :term:`asynchronous context manager` support was added. + + .. function:: suppress(*exceptions) @@ -384,6 +427,45 @@ Functions and classes provided: .. versionadded:: 3.2 +.. class:: AsyncContextDecorator + + Similar to :class:`ContextDecorator` but only for asynchronous functions. + + Example of ``AsyncContextDecorator``:: + + from asyncio import run + from contextlib import AsyncContextDecorator + + class mycontext(AsyncContextDecorator): + async def __aenter__(self): + print('Starting') + return self + + async def __aexit__(self, *exc): + print('Finishing') + return False + + >>> @mycontext() + ... async def function(): + ... print('The bit in the middle') + ... + >>> run(function()) + Starting + The bit in the middle + Finishing + + >>> async def function(): + ... async with mycontext(): + ... print('The bit in the middle') + ... + >>> run(function()) + Starting + The bit in the middle + Finishing + + .. versionadded:: 3.10 + + .. class:: ExitStack() A context manager that is designed to make it easy to programmatically diff --git a/Doc/library/contextvars.rst b/Doc/library/contextvars.rst index 8805661c45..14ac47f4c9 100644 --- a/Doc/library/contextvars.rst +++ b/Doc/library/contextvars.rst @@ -26,7 +26,7 @@ See also :pep:`567` for additional details. Context Variables ----------------- -.. class:: ContextVar(name, [\*, default]) +.. class:: ContextVar(name, [*, default]) This class is used to declare a new Context Variable, e.g.:: @@ -146,7 +146,7 @@ Manual Context Management Context implements the :class:`collections.abc.Mapping` interface. - .. method:: run(callable, \*args, \*\*kwargs) + .. method:: run(callable, *args, **kwargs) Execute ``callable(*args, **kwargs)`` code in the context object the *run* method is called on. Return the result of the execution diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index bf32d3e549..7313148721 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -2508,7 +2508,7 @@ other data types containing pointer type fields. Arrays and pointers ^^^^^^^^^^^^^^^^^^^ -.. class:: Array(\*args) +.. class:: Array(*args) Abstract base class for arrays. diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 508bc88e7f..dae0dd7aa5 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -1219,7 +1219,7 @@ Instance methods: .. method:: datetime.replace(year=self.year, month=self.month, day=self.day, \ hour=self.hour, minute=self.minute, second=self.second, microsecond=self.microsecond, \ - tzinfo=self.tzinfo, * fold=0) + tzinfo=self.tzinfo, *, fold=0) Return a datetime with the same attributes, except for those attributes given new values by whichever keyword arguments are specified. Note that @@ -1783,7 +1783,7 @@ Other constructor: Instance methods: .. method:: time.replace(hour=self.hour, minute=self.minute, second=self.second, \ - microsecond=self.microsecond, tzinfo=self.tzinfo, * fold=0) + microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=0) Return a :class:`.time` with the same value, except for those attributes given new values by whichever keyword arguments are specified. Note that diff --git a/Doc/library/dialog.rst b/Doc/library/dialog.rst index dc82a974ce..53f98c1018 100644 --- a/Doc/library/dialog.rst +++ b/Doc/library/dialog.rst @@ -198,7 +198,7 @@ These do not emulate the native look-and-feel of the platform. A subclass of FileDialog that creates a dialog window for selecting a destination file. - .. method:: ok_command() + .. method:: ok_command() Test whether or not the selection points to a valid file that is not a directory. Confirmation is required if an already existing file is diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index aa08988c8b..a5ee0fb538 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -149,7 +149,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. contains a good example of its use. -.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n') +.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') Compare *a* and *b* (lists of strings); return a delta (a :term:`generator` generating the delta lines) in context diff format. @@ -279,7 +279,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. emu -.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n') +.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') Compare *a* and *b* (lists of strings); return a delta (a :term:`generator` generating the delta lines) in unified diff format. @@ -321,7 +321,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. -.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n') +.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\n') Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a sequence of delta lines (also bytes) in the format returned by *dfunc*. diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index d0307bd8e9..c09d8338d2 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -708,7 +708,8 @@ iterations of the loop. .. opcode:: RERAISE - Re-raises the exception currently on top of the stack. + Re-raises the exception currently on top of the stack. If oparg is non-zero, + restores ``f_lasti`` of the current frame to its value when the exception was raised. .. versionadded:: 3.9 @@ -861,7 +862,7 @@ All of the following opcodes use their arguments. .. opcode:: LIST_TO_TUPLE - Pops a list from the stack and pushes a tuple containing the same values. + Pops a list from the stack and pushes a tuple containing the same values. .. versionadded:: 3.9 @@ -889,7 +890,7 @@ All of the following opcodes use their arguments. .. opcode:: DICT_MERGE - Like :opcode:`DICT_UPDATE` but raises an exception for duplicate keys. + Like :opcode:`DICT_UPDATE` but raises an exception for duplicate keys. .. versionadded:: 3.9 @@ -907,14 +908,14 @@ All of the following opcodes use their arguments. .. opcode:: IS_OP (invert) - Performs ``is`` comparison, or ``is not`` if ``invert`` is 1. + Performs ``is`` comparison, or ``is not`` if ``invert`` is 1. .. versionadded:: 3.9 .. opcode:: CONTAINS_OP (invert) - Performs ``in`` comparison, or ``not in`` if ``invert`` is 1. + Performs ``in`` comparison, or ``not in`` if ``invert`` is 1. .. versionadded:: 3.9 @@ -955,8 +956,8 @@ All of the following opcodes use their arguments. .. opcode:: JUMP_IF_NOT_EXC_MATCH (target) - Tests whether the second value on the stack is an exception matching TOS, - and jumps if it is not. Pops two values from the stack. + Tests whether the second value on the stack is an exception matching TOS, + and jumps if it is not. Pops two values from the stack. .. versionadded:: 3.9 @@ -1144,11 +1145,13 @@ All of the following opcodes use their arguments. * ``0x01`` a tuple of default values for positional-only and positional-or-keyword parameters in positional order * ``0x02`` a dictionary of keyword-only parameters' default values - * ``0x04`` an annotation dictionary + * ``0x04`` a tuple of strings containing parameters' annotations * ``0x08`` a tuple containing cells for free variables, making a closure * the code associated with the function (at TOS1) * the :term:`qualified name` of the function (at TOS) + .. versionchanged:: 3.10 + Flag value ``0x04`` is a tuple of strings instead of dictionary .. opcode:: BUILD_SLICE (argc) diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index a77322f83a..42ad0c9f06 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -719,36 +719,51 @@ above. An example's doctest directives modify doctest's behavior for that single example. Use ``+`` to enable the named behavior, or ``-`` to disable it. -For example, this test passes:: +For example, this test passes: - >>> print(list(range(20))) # doctest: +NORMALIZE_WHITESPACE +.. doctest:: + :no-trim-doctest-flags: + + >>> print(list(range(20))) # doctest: +NORMALIZE_WHITESPACE [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] Without the directive it would fail, both because the actual output doesn't have two blanks before the single-digit list elements, and because the actual output is on a single line. This test also passes, and also requires a directive to do -so:: +so: + +.. doctest:: + :no-trim-doctest-flags: - >>> print(list(range(20))) # doctest: +ELLIPSIS + >>> print(list(range(20))) # doctest: +ELLIPSIS [0, 1, ..., 18, 19] Multiple directives can be used on a single physical line, separated by -commas:: +commas: - >>> print(list(range(20))) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE +.. doctest:: + :no-trim-doctest-flags: + + >>> print(list(range(20))) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE [0, 1, ..., 18, 19] If multiple directive comments are used for a single example, then they are -combined:: +combined: + +.. doctest:: + :no-trim-doctest-flags: - >>> print(list(range(20))) # doctest: +ELLIPSIS - ... # doctest: +NORMALIZE_WHITESPACE + >>> print(list(range(20))) # doctest: +ELLIPSIS + ... # doctest: +NORMALIZE_WHITESPACE [0, 1, ..., 18, 19] As the previous example shows, you can add ``...`` lines to your example containing only directives. This can be useful when an example is too long for -a directive to comfortably fit on the same line:: +a directive to comfortably fit on the same line: + +.. doctest:: + :no-trim-doctest-flags: >>> print(list(range(5)) + list(range(10, 20)) + list(range(30, 40))) ... # doctest: +ELLIPSIS @@ -793,18 +808,23 @@ instead. Another is to do :: There are others, but you get the idea. -Another bad idea is to print things that embed an object address, like :: +Another bad idea is to print things that embed an object address, like + +.. doctest:: - >>> id(1.0) # certain to fail some of the time + >>> id(1.0) # certain to fail some of the time # doctest: +SKIP 7948648 >>> class C: pass - >>> C() # the default repr() for instances embeds an address - <__main__.C instance at 0x00AC18F0> + >>> C() # the default repr() for instances embeds an address # doctest: +SKIP + <C object at 0x00AC18F0> + +The :const:`ELLIPSIS` directive gives a nice approach for the last example: -The :const:`ELLIPSIS` directive gives a nice approach for the last example:: +.. doctest:: + :no-trim-doctest-flags: - >>> C() #doctest: +ELLIPSIS - <__main__.C instance at 0x...> + >>> C() # doctest: +ELLIPSIS + <C object at 0x...> Floating-point numbers are also subject to small output variations across platforms, because Python defers to the platform C library for float formatting, diff --git a/Doc/library/email.contentmanager.rst b/Doc/library/email.contentmanager.rst index e09c7c0e40..918fc55677 100644 --- a/Doc/library/email.contentmanager.rst +++ b/Doc/library/email.contentmanager.rst @@ -116,7 +116,7 @@ Currently the email package provides only one concrete content manager, decoding the payload to unicode. The default error handler is ``replace``. - .. method:: set_content(msg, <'str'>, subtype="plain", charset='utf-8' \ + .. method:: set_content(msg, <'str'>, subtype="plain", charset='utf-8', \ cte=None, \ disposition=None, filename=None, cid=None, \ params=None, headers=None) diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst index 07152c224f..e093f13893 100644 --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -116,7 +116,7 @@ Here is the :class:`Header` class description: if *s* is a byte string. - .. method:: encode(splitchars=';, \\t', maxlinelen=None, linesep='\\n') + .. method:: encode(splitchars=';, \t', maxlinelen=None, linesep='\n') Encode a message header into an RFC-compliant format, possibly wrapping long lines and encapsulating non-ASCII parts in base64 or quoted-printable diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index 8e70762598..bf53b9520f 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -210,7 +210,7 @@ added matters. To illustrate:: :meth:`register_defect` method. - .. attribute:: mangle_from\_ + .. attribute:: mangle_from_ If :const:`True`, lines starting with *"From "* in the body are escaped by putting a ``>`` in front of them. This parameter is used when diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index 118002bef1..c532e2caec 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -61,16 +61,18 @@ helper, :class:`auto`. the bitwise operations without losing their :class:`Flag` membership. .. function:: unique - :noindex: + :noindex: Enum class decorator that ensures only one name is bound to any one value. .. class:: auto - Instances are replaced with an appropriate value for Enum members. By default, the initial value starts at 1. + Instances are replaced with an appropriate value for Enum members. + :class:`StrEnum` defaults to the lower-cased version of the member name, + while other Enums default to 1 and increase from there. .. versionadded:: 3.6 ``Flag``, ``IntFlag``, ``auto`` - +.. versionadded:: 3.10 ``StrEnum`` Creating an Enum ---------------- @@ -1164,6 +1166,15 @@ and raise an error if the two do not match:: In Python 2 code the :attr:`_order_` attribute is necessary as definition order is lost before it can be recorded. + +_Private__names +""""""""""""""" + +Private names are not converted to Enum members, but remain normal attributes. + +.. versionchanged:: 3.10 + + ``Enum`` member type """""""""""""""""""" diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 8fb25a50e2..1028213699 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -90,8 +90,13 @@ The following exceptions are used mostly as base classes for other exceptions. .. method:: with_traceback(tb) This method sets *tb* as the new traceback for the exception and returns - the exception object. It is usually used in exception handling code like - this:: + the exception object. It was more commonly used before the exception + chaining features of :pep:`3134` became available. The following example + shows how we can convert an instance of ``SomeException`` into an + instance of ``OtherException`` while preserving the traceback. Once + raised, the current frame is pushed onto the traceback of the + ``OtherException``, as would have happened to the traceback of the + original ``SomeException`` had we allowed it to propagate to the caller. try: ... diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst index 31b9b4afab..c60603b30a 100644 --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -173,7 +173,13 @@ The :class:`dircmp` class .. attribute:: subdirs A dictionary mapping names in :attr:`common_dirs` to :class:`dircmp` - objects. + instances (or MyDirCmp instances if this instance is of type MyDirCmp, a + subclass of :class:`dircmp`). + + .. versionchanged:: 3.10 + Previously entries were always :class:`dircmp` instances. Now entries + are the same type as *self*, if *self* is a subclass of + :class:`dircmp`. .. attribute:: DEFAULT_IGNORES diff --git a/Doc/library/fnmatch.rst b/Doc/library/fnmatch.rst index ce07d326b3..925f08e914 100644 --- a/Doc/library/fnmatch.rst +++ b/Doc/library/fnmatch.rst @@ -75,7 +75,7 @@ patterns. .. function:: filter(names, pattern) - Return the subset of the list of *names* that match *pattern*. It is the same as + Construct a list from those elements of the iterable *names* that match *pattern*. It is the same as ``[n for n in names if fnmatch(n, pattern)]``, but implemented more efficiently. diff --git a/Doc/library/formatter.rst b/Doc/library/formatter.rst deleted file mode 100644 index 6c10ac6fab..0000000000 --- a/Doc/library/formatter.rst +++ /dev/null @@ -1,351 +0,0 @@ -:mod:`formatter` --- Generic output formatting -============================================== - -.. module:: formatter - :synopsis: Generic output formatter and device interface. - :deprecated: - -.. deprecated:: 3.4 - Due to lack of usage, the formatter module has been deprecated. - --------------- - -This module supports two interface definitions, each with multiple -implementations: The *formatter* interface, and the *writer* interface which is -required by the formatter interface. - -Formatter objects transform an abstract flow of formatting events into specific -output events on writer objects. Formatters manage several stack structures to -allow various properties of a writer object to be changed and restored; writers -need not be able to handle relative changes nor any sort of "change back" -operation. Specific writer properties which may be controlled via formatter -objects are horizontal alignment, font, and left margin indentations. A -mechanism is provided which supports providing arbitrary, non-exclusive style -settings to a writer as well. Additional interfaces facilitate formatting -events which are not reversible, such as paragraph separation. - -Writer objects encapsulate device interfaces. Abstract devices, such as file -formats, are supported as well as physical devices. The provided -implementations all work with abstract devices. The interface makes available -mechanisms for setting the properties which formatter objects manage and -inserting data into the output. - - -.. _formatter-interface: - -The Formatter Interface ------------------------ - -Interfaces to create formatters are dependent on the specific formatter class -being instantiated. The interfaces described below are the required interfaces -which all formatters must support once initialized. - -One data element is defined at the module level: - - -.. data:: AS_IS - - Value which can be used in the font specification passed to the ``push_font()`` - method described below, or as the new value to any other ``push_property()`` - method. Pushing the ``AS_IS`` value allows the corresponding ``pop_property()`` - method to be called without having to track whether the property was changed. - -The following attributes are defined for formatter instance objects: - - -.. attribute:: formatter.writer - - The writer instance with which the formatter interacts. - - -.. method:: formatter.end_paragraph(blanklines) - - Close any open paragraphs and insert at least *blanklines* before the next - paragraph. - - -.. method:: formatter.add_line_break() - - Add a hard line break if one does not already exist. This does not break the - logical paragraph. - - -.. method:: formatter.add_hor_rule(*args, **kw) - - Insert a horizontal rule in the output. A hard break is inserted if there is - data in the current paragraph, but the logical paragraph is not broken. The - arguments and keywords are passed on to the writer's :meth:`send_line_break` - method. - - -.. method:: formatter.add_flowing_data(data) - - Provide data which should be formatted with collapsed whitespace. Whitespace - from preceding and successive calls to :meth:`add_flowing_data` is considered as - well when the whitespace collapse is performed. The data which is passed to - this method is expected to be word-wrapped by the output device. Note that any - word-wrapping still must be performed by the writer object due to the need to - rely on device and font information. - - -.. method:: formatter.add_literal_data(data) - - Provide data which should be passed to the writer unchanged. Whitespace, - including newline and tab characters, are considered legal in the value of - *data*. - - -.. method:: formatter.add_label_data(format, counter) - - Insert a label which should be placed to the left of the current left margin. - This should be used for constructing bulleted or numbered lists. If the - *format* value is a string, it is interpreted as a format specification for - *counter*, which should be an integer. The result of this formatting becomes the - value of the label; if *format* is not a string it is used as the label value - directly. The label value is passed as the only argument to the writer's - :meth:`send_label_data` method. Interpretation of non-string label values is - dependent on the associated writer. - - Format specifications are strings which, in combination with a counter value, - are used to compute label values. Each character in the format string is copied - to the label value, with some characters recognized to indicate a transform on - the counter value. Specifically, the character ``'1'`` represents the counter - value formatter as an Arabic number, the characters ``'A'`` and ``'a'`` - represent alphabetic representations of the counter value in upper and lower - case, respectively, and ``'I'`` and ``'i'`` represent the counter value in Roman - numerals, in upper and lower case. Note that the alphabetic and roman - transforms require that the counter value be greater than zero. - - -.. method:: formatter.flush_softspace() - - Send any pending whitespace buffered from a previous call to - :meth:`add_flowing_data` to the associated writer object. This should be called - before any direct manipulation of the writer object. - - -.. method:: formatter.push_alignment(align) - - Push a new alignment setting onto the alignment stack. This may be - :const:`AS_IS` if no change is desired. If the alignment value is changed from - the previous setting, the writer's :meth:`new_alignment` method is called with - the *align* value. - - -.. method:: formatter.pop_alignment() - - Restore the previous alignment. - - -.. method:: formatter.push_font((size, italic, bold, teletype)) - - Change some or all font properties of the writer object. Properties which are - not set to :const:`AS_IS` are set to the values passed in while others are - maintained at their current settings. The writer's :meth:`new_font` method is - called with the fully resolved font specification. - - -.. method:: formatter.pop_font() - - Restore the previous font. - - -.. method:: formatter.push_margin(margin) - - Increase the number of left margin indentations by one, associating the logical - tag *margin* with the new indentation. The initial margin level is ``0``. - Changed values of the logical tag must be true values; false values other than - :const:`AS_IS` are not sufficient to change the margin. - - -.. method:: formatter.pop_margin() - - Restore the previous margin. - - -.. method:: formatter.push_style(*styles) - - Push any number of arbitrary style specifications. All styles are pushed onto - the styles stack in order. A tuple representing the entire stack, including - :const:`AS_IS` values, is passed to the writer's :meth:`new_styles` method. - - -.. method:: formatter.pop_style(n=1) - - Pop the last *n* style specifications passed to :meth:`push_style`. A tuple - representing the revised stack, including :const:`AS_IS` values, is passed to - the writer's :meth:`new_styles` method. - - -.. method:: formatter.set_spacing(spacing) - - Set the spacing style for the writer. - - -.. method:: formatter.assert_line_data(flag=1) - - Inform the formatter that data has been added to the current paragraph - out-of-band. This should be used when the writer has been manipulated - directly. The optional *flag* argument can be set to false if the writer - manipulations produced a hard line break at the end of the output. - - -.. _formatter-impls: - -Formatter Implementations -------------------------- - -Two implementations of formatter objects are provided by this module. Most -applications may use one of these classes without modification or subclassing. - - -.. class:: NullFormatter(writer=None) - - A formatter which does nothing. If *writer* is omitted, a :class:`NullWriter` - instance is created. No methods of the writer are called by - :class:`NullFormatter` instances. Implementations should inherit from this - class if implementing a writer interface but don't need to inherit any - implementation. - - -.. class:: AbstractFormatter(writer) - - The standard formatter. This implementation has demonstrated wide applicability - to many writers, and may be used directly in most circumstances. It has been - used to implement a full-featured World Wide Web browser. - - -.. _writer-interface: - -The Writer Interface --------------------- - -Interfaces to create writers are dependent on the specific writer class being -instantiated. The interfaces described below are the required interfaces which -all writers must support once initialized. Note that while most applications can -use the :class:`AbstractFormatter` class as a formatter, the writer must -typically be provided by the application. - - -.. method:: writer.flush() - - Flush any buffered output or device control events. - - -.. method:: writer.new_alignment(align) - - Set the alignment style. The *align* value can be any object, but by convention - is a string or ``None``, where ``None`` indicates that the writer's "preferred" - alignment should be used. Conventional *align* values are ``'left'``, - ``'center'``, ``'right'``, and ``'justify'``. - - -.. method:: writer.new_font(font) - - Set the font style. The value of *font* will be ``None``, indicating that the - device's default font should be used, or a tuple of the form ``(size, - italic, bold, teletype)``. Size will be a string indicating the size of - font that should be used; specific strings and their interpretation must be - defined by the application. The *italic*, *bold*, and *teletype* values are - Boolean values specifying which of those font attributes should be used. - - -.. method:: writer.new_margin(margin, level) - - Set the margin level to the integer *level* and the logical tag to *margin*. - Interpretation of the logical tag is at the writer's discretion; the only - restriction on the value of the logical tag is that it not be a false value for - non-zero values of *level*. - - -.. method:: writer.new_spacing(spacing) - - Set the spacing style to *spacing*. - - -.. method:: writer.new_styles(styles) - - Set additional styles. The *styles* value is a tuple of arbitrary values; the - value :const:`AS_IS` should be ignored. The *styles* tuple may be interpreted - either as a set or as a stack depending on the requirements of the application - and writer implementation. - - -.. method:: writer.send_line_break() - - Break the current line. - - -.. method:: writer.send_paragraph(blankline) - - Produce a paragraph separation of at least *blankline* blank lines, or the - equivalent. The *blankline* value will be an integer. Note that the - implementation will receive a call to :meth:`send_line_break` before this call - if a line break is needed; this method should not include ending the last line - of the paragraph. It is only responsible for vertical spacing between - paragraphs. - - -.. method:: writer.send_hor_rule(*args, **kw) - - Display a horizontal rule on the output device. The arguments to this method - are entirely application- and writer-specific, and should be interpreted with - care. The method implementation may assume that a line break has already been - issued via :meth:`send_line_break`. - - -.. method:: writer.send_flowing_data(data) - - Output character data which may be word-wrapped and re-flowed as needed. Within - any sequence of calls to this method, the writer may assume that spans of - multiple whitespace characters have been collapsed to single space characters. - - -.. method:: writer.send_literal_data(data) - - Output character data which has already been formatted for display. Generally, - this should be interpreted to mean that line breaks indicated by newline - characters should be preserved and no new line breaks should be introduced. The - data may contain embedded newline and tab characters, unlike data provided to - the :meth:`send_formatted_data` interface. - - -.. method:: writer.send_label_data(data) - - Set *data* to the left of the current left margin, if possible. The value of - *data* is not restricted; treatment of non-string values is entirely - application- and writer-dependent. This method will only be called at the - beginning of a line. - - -.. _writer-impls: - -Writer Implementations ----------------------- - -Three implementations of the writer object interface are provided as examples by -this module. Most applications will need to derive new writer classes from the -:class:`NullWriter` class. - - -.. class:: NullWriter() - - A writer which only provides the interface definition; no actions are taken on - any methods. This should be the base class for all writers which do not need to - inherit any implementation methods. - - -.. class:: AbstractWriter() - - A writer which can be used in debugging formatters, but not much else. Each - method simply announces itself by printing its name and arguments on standard - output. - - -.. class:: DumbWriter(file=None, maxcol=72) - - Simple writer class which writes output on the :term:`file object` passed - in as *file* or, if *file* is omitted, on standard output. The output is - simply word-wrapped to the number of columns specified by *maxcol*. This - class is suitable for reflowing a sequence of paragraphs. - diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index a8a4ca4200..0598a6ce94 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -478,14 +478,15 @@ are always available. They are listed here in alphabetical order. dictionaries as global and local namespace. If the *globals* dictionary is present and does not contain a value for the key ``__builtins__``, a reference to the dictionary of the built-in module :mod:`builtins` is - inserted under that key before *expression* is parsed. This means that - *expression* normally has full access to the standard :mod:`builtins` - module and restricted environments are propagated. If the *locals* - dictionary is omitted it defaults to the *globals* dictionary. If both - dictionaries are omitted, the expression is executed with the *globals* and - *locals* in the environment where :func:`eval` is called. Note, *eval()* - does not have access to the :term:`nested scopes <nested scope>` (non-locals) in the - enclosing environment. + inserted under that key before *expression* is parsed. That way you can + control what builtins are available to the executed code by inserting your + own ``__builtins__`` dictionary into *globals* before passing it to + :func:`eval`. If the *locals* dictionary is omitted it defaults to the + *globals* dictionary. If both dictionaries are omitted, the expression is + executed with the *globals* and *locals* in the environment where + :func:`eval` is called. Note, *eval()* does not have access to the + :term:`nested scopes <nested scope>` (non-locals) in the enclosing + environment. The return value is the result of the evaluated expression. Syntax errors are reported as exceptions. Example: @@ -1333,7 +1334,7 @@ are always available. They are listed here in alphabetical order. supported. -.. function:: print(*objects, sep=' ', end='\\n', file=sys.stdout, flush=False) +.. function:: print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False) Print *objects* to the text stream *file*, separated by *sep* and followed by *end*. *sep*, *end*, *file* and *flush*, if present, must be given as keyword diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 35997db2a9..56f4c0a0d7 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -99,6 +99,11 @@ The module provides the following classes: :attr:`ssl.SSLContext.post_handshake_auth` for the default *context* or when *cert_file* is passed with a custom *context*. + .. versionchanged:: 3.10 + This class now sends an ALPN extension with protocol indicator + ``http/1.1`` when no *context* is given. Custom *context* should set + ALPN protocols with :meth:`~ssl.SSLContext.set_alpn_protocol`. + .. deprecated:: 3.6 *key_file* and *cert_file* are deprecated in favor of *context*. diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 17792b2005..a2c1eb00d8 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -93,7 +93,7 @@ Cookie Objects :meth:`value_decode` are inverses on the range of *value_decode*. -.. method:: BaseCookie.output(attrs=None, header='Set-Cookie:', sep='\\r\\n') +.. method:: BaseCookie.output(attrs=None, header='Set-Cookie:', sep='\r\n') Return a string representation suitable to be sent as HTTP headers. *attrs* and *header* are sent to each :class:`Morsel`'s :meth:`output` method. *sep* is used diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index 02ecfd95d4..65681ec093 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -174,9 +174,9 @@ example of usage. .. seealso:: - Documents describing the protocol, and sources and binaries for servers - implementing it, can all be found at the University of Washington's *IMAP - Information Center* (https://www.washington.edu/imap/). + Documents describing the protocol, sources for servers + implementing it, by the University of Washington's IMAP Information Center + can all be found at (**Source Code**) https://github.com/uw-imap/imap (**Not Maintained**). .. _imap4-objects: diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 45b62aec9e..f1c444fe82 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -1138,7 +1138,7 @@ find and load modules. directory for ``''`` (i.e. the empty string). -.. class:: FileFinder(path, \*loader_details) +.. class:: FileFinder(path, *loader_details) A concrete implementation of :class:`importlib.abc.PathEntryFinder` which caches results from the file system. @@ -1181,7 +1181,7 @@ find and load modules. Clear out the internal cache. - .. classmethod:: path_hook(\*loader_details) + .. classmethod:: path_hook(*loader_details) A class method which returns a closure for use on :attr:`sys.path_hooks`. An instance of :class:`FileFinder` is returned by the closure using the diff --git a/Doc/library/index.rst b/Doc/library/index.rst index bebf7429b0..42f994f8b8 100644 --- a/Doc/library/index.rst +++ b/Doc/library/index.rst @@ -72,7 +72,6 @@ the `Python Package Index <https://pypi.org>`_. custominterp.rst modules.rst language.rst - misc.rst windows.rst unix.rst superseded.rst diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index d00a30ff00..850d6018ba 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -556,7 +556,7 @@ The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the :func:`signature` function. -.. function:: signature(callable, \*, follow_wrapped=True) +.. function:: signature(callable, *, follow_wrapped=True, globalns=None, localns=None) Return a :class:`Signature` object for the given ``callable``:: @@ -581,6 +581,9 @@ function. Raises :exc:`ValueError` if no signature can be provided, and :exc:`TypeError` if that type of object is not supported. + ``globalns`` and ``localns`` are passed into + :func:`typing.get_type_hints` when resolving the annotations. + A slash(/) in the signature of a function denotes that the parameters prior to it are positional-only. For more info, see :ref:`the FAQ entry on positional-only parameters <faq-positional-only-arguments>`. @@ -590,14 +593,23 @@ function. ``callable`` specifically (``callable.__wrapped__`` will not be used to unwrap decorated callables.) + .. versionadded:: 3.10 + ``globalns`` and ``localns`` parameters. + .. note:: Some callables may not be introspectable in certain implementations of Python. For example, in CPython, some built-in functions defined in C provide no metadata about their arguments. + .. note:: + + Will first try to resolve the annotations, but when it fails and + encounters with an error while that operation, the annotations will be + returned unchanged (as strings). + -.. class:: Signature(parameters=None, \*, return_annotation=Signature.empty) +.. class:: Signature(parameters=None, *, return_annotation=Signature.empty) A Signature object represents the call signature of a function and its return annotation. For each parameter accepted by the function it stores a @@ -668,11 +680,12 @@ function. >>> str(new_sig) "(a, b) -> 'new return anno'" - .. classmethod:: Signature.from_callable(obj, \*, follow_wrapped=True) + .. classmethod:: Signature.from_callable(obj, *, follow_wrapped=True, globalns=None, localns=None) Return a :class:`Signature` (or its subclass) object for a given callable ``obj``. Pass ``follow_wrapped=False`` to get a signature of ``obj`` - without unwrapping its ``__wrapped__`` chain. + without unwrapping its ``__wrapped__`` chain. ``globalns`` and + ``localns`` will be used as the namespaces when resolving annotations. This method simplifies subclassing of :class:`Signature`:: @@ -683,8 +696,11 @@ function. .. versionadded:: 3.5 + .. versionadded:: 3.10 + ``globalns`` and ``localns`` parameters. + -.. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) +.. class:: Parameter(name, kind, *, default=Parameter.empty, annotation=Parameter.empty) Parameter objects are *immutable*. Instead of modifying a Parameter object, you can use :meth:`Parameter.replace` to create a modified copy. diff --git a/Doc/library/io.rst b/Doc/library/io.rst index aecbec5686..048cb2a7ff 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -964,7 +964,7 @@ Text I/O .. versionadded:: 3.7 -.. class:: StringIO(initial_value='', newline='\\n') +.. class:: StringIO(initial_value='', newline='\n') A text stream using an in-memory text buffer. It inherits :class:`TextIOBase`. diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index 107bc515a6..03416a0a7f 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -55,6 +55,7 @@ Iterator Arguments Results :func:`filterfalse` pred, seq elements of seq where pred(elem) is false ``filterfalse(lambda x: x%2, range(10)) --> 0 2 4 6 8`` :func:`groupby` iterable[, key] sub-iterators grouped by value of key(v) :func:`islice` seq, [start,] stop [, step] elements from seq[start:stop:step] ``islice('ABCDEFG', 2, None) --> C D E F G`` +:func:`pairwise` iterable (p[0], p[1]), (p[1], p[2]) ``pairwise('ABCDEFG') --> AB BC CD DE EF FG`` :func:`starmap` func, seq func(\*seq[0]), func(\*seq[1]), ... ``starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000`` :func:`takewhile` pred, seq seq[0], seq[1], until pred fails ``takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4`` :func:`tee` it, n it1, it2, ... itn splits one iterator into n @@ -475,6 +476,22 @@ loops that truncate the stream. If *start* is ``None``, then iteration starts at zero. If *step* is ``None``, then the step defaults to one. +.. function:: pairwise(iterable) + + Return successive overlapping pairs taken from the input *iterable*. + + The number of 2-tuples in the output iterator will be one fewer than the + number of inputs. It will be empty if the input iterable has fewer than + two values. + + Roughly equivalent to:: + + def pairwise(iterable): + # pairwise('ABCDEFG') --> AB BC CD DE EF FG + a, b = tee(iterable) + next(b, None) + return zip(a, b) + .. function:: permutations(iterable, r=None) @@ -565,7 +582,7 @@ loops that truncate the stream. Before :func:`product` runs, it completely consumes the input iterables, keeping pools of values in memory to generate the products. Accordingly, - it only useful with finite inputs. + it is only useful with finite inputs. .. function:: repeat(object[, times]) @@ -755,7 +772,7 @@ which incur interpreter overhead. "Count how many times the predicate is true" return sum(map(pred, iterable)) - def padnone(iterable): + def pad_none(iterable): """Returns the sequence elements and then returns None indefinitely. Useful for emulating the behavior of the built-in map() function. @@ -769,6 +786,18 @@ which incur interpreter overhead. def dotproduct(vec1, vec2): return sum(map(operator.mul, vec1, vec2)) + def convolve(signal, kernel): + # See: https://betterexplained.com/articles/intuitive-convolution/ + # convolve(data, [0.25, 0.25, 0.25, 0.25]) --> Moving average (blur) + # convolve(data, [1, -1]) --> 1st finite difference (1st derivative) + # convolve(data, [1, -2, 1]) --> 2nd finite difference (2nd derivative) + kernel = list(reversed(kernel)) + n = len(kernel) + window = collections.deque([0] * n, maxlen=n) + for x in chain(signal, repeat(0, n-1)): + window.append(x) + yield sum(map(operator.mul, kernel, window)) + def flatten(list_of_lists): "Flatten one level of nesting" return chain.from_iterable(list_of_lists) @@ -782,12 +811,6 @@ which incur interpreter overhead. return starmap(func, repeat(args)) return starmap(func, repeat(args, times)) - def pairwise(iterable): - "s -> (s0,s1), (s1,s2), (s2, s3), ..." - a, b = tee(iterable) - next(b, None) - return zip(a, b) - def grouper(iterable, n, fillvalue=None): "Collect data into fixed-length chunks or blocks" # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx" @@ -809,7 +832,7 @@ which incur interpreter overhead. nexts = cycle(islice(nexts, num_active)) def partition(pred, iterable): - 'Use a predicate to partition entries into false entries and true entries' + "Use a predicate to partition entries into false entries and true entries" # partition(is_odd, range(10)) --> 0 2 4 6 8 and 1 3 5 7 9 t1, t2 = tee(iterable) return filterfalse(pred, t1), filter(pred, t2) @@ -881,7 +904,7 @@ which incur interpreter overhead. def random_product(*args, repeat=1): "Random selection from itertools.product(*args, **kwds)" pools = [tuple(pool) for pool in args] * repeat - return tuple(random.choice(pool) for pool in pools) + return tuple(map(random.choice, pools)) def random_permutation(iterable, r=None): "Random selection from itertools.permutations(iterable, r)" @@ -900,11 +923,11 @@ which incur interpreter overhead. "Random selection from itertools.combinations_with_replacement(iterable, r)" pool = tuple(iterable) n = len(pool) - indices = sorted(random.randrange(n) for i in range(r)) + indices = sorted(random.choices(range(n), k=r)) return tuple(pool[i] for i in indices) def nth_combination(iterable, r, index): - 'Equivalent to list(combinations(iterable, r))[index]' + "Equivalent to list(combinations(iterable, r))[index]" pool = tuple(iterable) n = len(pool) if r < 0 or r > n: diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst index 683d6ed5e8..0b5e2fc2a6 100644 --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -35,45 +35,45 @@ in :mod:`logging` itself) and defining handlers which are declared either in .. function:: dictConfig(config) - Takes the logging configuration from a dictionary. The contents of - this dictionary are described in :ref:`logging-config-dictschema` - below. - - If an error is encountered during configuration, this function will - raise a :exc:`ValueError`, :exc:`TypeError`, :exc:`AttributeError` - or :exc:`ImportError` with a suitably descriptive message. The - following is a (possibly incomplete) list of conditions which will - raise an error: - - * A ``level`` which is not a string or which is a string not - corresponding to an actual logging level. - * A ``propagate`` value which is not a boolean. - * An id which does not have a corresponding destination. - * A non-existent handler id found during an incremental call. - * An invalid logger name. - * Inability to resolve to an internal or external object. - - Parsing is performed by the :class:`DictConfigurator` class, whose - constructor is passed the dictionary used for configuration, and - has a :meth:`configure` method. The :mod:`logging.config` module - has a callable attribute :attr:`dictConfigClass` - which is initially set to :class:`DictConfigurator`. - You can replace the value of :attr:`dictConfigClass` with a - suitable implementation of your own. - - :func:`dictConfig` calls :attr:`dictConfigClass` passing - the specified dictionary, and then calls the :meth:`configure` method on - the returned object to put the configuration into effect:: - - def dictConfig(config): - dictConfigClass(config).configure() - - For example, a subclass of :class:`DictConfigurator` could call - ``DictConfigurator.__init__()`` in its own :meth:`__init__()`, then - set up custom prefixes which would be usable in the subsequent - :meth:`configure` call. :attr:`dictConfigClass` would be bound to - this new subclass, and then :func:`dictConfig` could be called exactly as - in the default, uncustomized state. + Takes the logging configuration from a dictionary. The contents of + this dictionary are described in :ref:`logging-config-dictschema` + below. + + If an error is encountered during configuration, this function will + raise a :exc:`ValueError`, :exc:`TypeError`, :exc:`AttributeError` + or :exc:`ImportError` with a suitably descriptive message. The + following is a (possibly incomplete) list of conditions which will + raise an error: + + * A ``level`` which is not a string or which is a string not + corresponding to an actual logging level. + * A ``propagate`` value which is not a boolean. + * An id which does not have a corresponding destination. + * A non-existent handler id found during an incremental call. + * An invalid logger name. + * Inability to resolve to an internal or external object. + + Parsing is performed by the :class:`DictConfigurator` class, whose + constructor is passed the dictionary used for configuration, and + has a :meth:`configure` method. The :mod:`logging.config` module + has a callable attribute :attr:`dictConfigClass` + which is initially set to :class:`DictConfigurator`. + You can replace the value of :attr:`dictConfigClass` with a + suitable implementation of your own. + + :func:`dictConfig` calls :attr:`dictConfigClass` passing + the specified dictionary, and then calls the :meth:`configure` method on + the returned object to put the configuration into effect:: + + def dictConfig(config): + dictConfigClass(config).configure() + + For example, a subclass of :class:`DictConfigurator` could call + ``DictConfigurator.__init__()`` in its own :meth:`__init__()`, then + set up custom prefixes which would be usable in the subsequent + :meth:`configure` call. :attr:`dictConfigClass` would be bound to + this new subclass, and then :func:`dictConfig` could be called exactly as + in the default, uncustomized state. .. versionadded:: 3.2 diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index fb8ea705b0..431a5849fa 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -1096,8 +1096,8 @@ functions. suitable value. .. versionchanged:: 3.7 - The *level* parameter was defaulted to level ``CRITICAL``. See Issue - #28524 for more information about this change. + The *level* parameter was defaulted to level ``CRITICAL``. See + :issue:`28524` for more information about this change. .. function:: addLevelName(level, levelName) @@ -1174,7 +1174,9 @@ functions. | | to ``'a'``. | +--------------+---------------------------------------------+ | *format* | Use the specified format string for the | - | | handler. | + | | handler. Defaults to attributes | + | | ``levelname``, ``name`` and ``message`` | + | | separated by colons. | +--------------+---------------------------------------------+ | *datefmt* | Use the specified date/time format, as | | | accepted by :func:`time.strftime`. | diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst index 4bfff9c614..633c87873c 100644 --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -33,7 +33,7 @@ from multiple threads, it is necessary to protect it with a lock. Reading and writing compressed files ------------------------------------ -.. function:: open(filename, mode="rb", \*, format=None, check=-1, preset=None, filters=None, encoding=None, errors=None, newline=None) +.. function:: open(filename, mode="rb", *, format=None, check=-1, preset=None, filters=None, encoding=None, errors=None, newline=None) Open an LZMA-compressed file in binary or text mode, returning a :term:`file object`. @@ -69,7 +69,7 @@ Reading and writing compressed files Accepts a :term:`path-like object`. -.. class:: LZMAFile(filename=None, mode="r", \*, format=None, check=-1, preset=None, filters=None) +.. class:: LZMAFile(filename=None, mode="r", *, format=None, check=-1, preset=None, filters=None) Open an LZMA-compressed file in binary mode. diff --git a/Doc/library/misc.rst b/Doc/library/misc.rst deleted file mode 100644 index 0943235246..0000000000 --- a/Doc/library/misc.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. _misc: - -********************** -Miscellaneous Services -********************** - -The modules described in this chapter provide miscellaneous services that are -available in all Python versions. Here's an overview: - - -.. toctree:: - - formatter.rst diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 698c176537..d9825b47c7 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -337,6 +337,8 @@ MADV_* Constants MADV_NOCORE MADV_CORE MADV_PROTECT + MADV_FREE_REUSABLE + MADV_FREE_REUSE These options can be passed to :meth:`mmap.madvise`. Not every option will be present on every system. diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index f8e7556f25..e109b1ebe1 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -98,7 +98,7 @@ to start a process. These *start methods* are *spawn* The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run - the process objects :meth:`~Process.run` method. In particular, + the process object's :meth:`~Process.run` method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using *fork* or *forkserver*. @@ -2569,9 +2569,9 @@ Address Formats filesystem. * An ``'AF_PIPE'`` address is a string of the form - :samp:`r'\\\\.\\pipe\\{PipeName}'`. To use :func:`Client` to connect to a named - pipe on a remote computer called *ServerName* one should use an address of the - form :samp:`r'\\\\{ServerName}\\pipe\\{PipeName}'` instead. + :samp:`r'\\\\.\\pipe\\{PipeName}'`. To use :func:`Client` to connect to a named + pipe on a remote computer called *ServerName* one should use an address of the + form :samp:`r'\\\\{ServerName}\\pipe\\{PipeName}'` instead. Note that any string beginning with two backslashes is assumed by default to be an ``'AF_PIPE'`` address rather than an ``'AF_UNIX'`` address. @@ -2661,6 +2661,46 @@ The :mod:`multiprocessing.dummy` module :mod:`multiprocessing.dummy` replicates the API of :mod:`multiprocessing` but is no more than a wrapper around the :mod:`threading` module. +.. currentmodule:: multiprocessing.pool + +In particular, the ``Pool`` function provided by :mod:`multiprocessing.dummy` +returns an instance of :class:`ThreadPool`, which is a subclass of +:class:`Pool` that supports all the same method calls but uses a pool of +worker threads rather than worker processes. + + +.. class:: ThreadPool([processes[, initializer[, initargs]]]) + + A thread pool object which controls a pool of worker threads to which jobs + can be submitted. :class:`ThreadPool` instances are fully interface + compatible with :class:`Pool` instances, and their resources must also be + properly managed, either by using the pool as a context manager or by + calling :meth:`~multiprocessing.pool.Pool.close` and + :meth:`~multiprocessing.pool.Pool.terminate` manually. + + *processes* is the number of worker threads to use. If *processes* is + ``None`` then the number returned by :func:`os.cpu_count` is used. + + If *initializer* is not ``None`` then each worker process will call + ``initializer(*initargs)`` when it starts. + + Unlike :class:`Pool`, *maxtasksperchild* and *context* cannot be provided. + + .. note:: + + A :class:`ThreadPool` shares the same interface as :class:`Pool`, which + is designed around a pool of processes and predates the introduction of + the :class:`concurrent.futures` module. As such, it inherits some + operations that don't make sense for a pool backed by threads, and it + has its own type for representing the status of asynchronous jobs, + :class:`AsyncResult`, that is not understood by any other libraries. + + Users should generally prefer to use + :class:`concurrent.futures.ThreadPoolExecutor`, which has a simpler + interface that was designed around threads from the start, and which + returns :class:`concurrent.futures.Future` instances that are + compatible with many other libraries, including :mod:`asyncio`. + .. _multiprocessing-programming: diff --git a/Doc/library/os.rst b/Doc/library/os.rst index f9f35b3124..35cf7c0a0b 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -32,7 +32,7 @@ Notes on the availability of these functions: objects, and result in an object of the same type, if a path or file name is returned. -* On VxWorks, os.fork, os.execv and os.spawn*p* are not supported. +* On VxWorks, os.popen, os.fork, os.execv and os.spawn*p* are not supported. .. note:: @@ -113,6 +113,8 @@ of the UTF-8 encoding: :ref:`error handler <error-handlers>` being enabled for :data:`sys.stdin` and :data:`sys.stdout` (:data:`sys.stderr` continues to use ``backslashreplace`` as it does in the default locale-aware mode) +* On Unix, :func:`os.device_encoding` returns ``'UTF-8'``. rather than the + device encoding. Note that the standard stream settings in UTF-8 mode can be overridden by :envvar:`PYTHONIOENCODING` (just as they can be in the default locale-aware @@ -808,6 +810,12 @@ as internal buffering of data. Return a string describing the encoding of the device associated with *fd* if it is connected to a terminal; else return :const:`None`. + On Unix, if the :ref:`Python UTF-8 Mode <utf8-mode>` is enabled, return + ``'UTF-8'`` rather than the device encoding. + + .. versionchanged:: 3.10 + On Unix, the function now implements the Python UTF-8 Mode. + .. function:: dup(fd) @@ -1315,12 +1323,12 @@ or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windo .. data:: RWF_APPEND - Provide a per-write equivalent of the :data:`O_APPEND` :func:`os.open` - flag. This flag is meaningful only for :func:`os.pwritev`, and its - effect applies only to the data range written by the system call. The - *offset* argument does not affect the write operation; the data is always - appended to the end of the file. However, if the *offset* argument is - ``-1``, the current file *offset* is updated. + Provide a per-write equivalent of the :data:`O_APPEND` :func:`os.open` + flag. This flag is meaningful only for :func:`os.pwritev`, and its + effect applies only to the data range written by the system call. The + *offset* argument does not affect the write operation; the data is always + appended to the end of the file. However, if the *offset* argument is + ``-1``, the current file *offset* is updated. .. availability:: Linux 4.16 and newer. @@ -1411,6 +1419,39 @@ or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windo .. versionadded:: 3.3 +.. function:: splice(src, dst, count, offset_src=None, offset_dst=None) + + Transfer *count* bytes from file descriptor *src*, starting from offset + *offset_src*, to file descriptor *dst*, starting from offset *offset_dst*. + At least one of the file descriptors must refer to a pipe. If *offset_src* + is None, then *src* is read from the current position; respectively for + *offset_dst*. The offset associated to the file descriptor that refers to a + pipe must be ``None``. The files pointed by *src* and *dst* must reside in + the same filesystem, otherwise an :exc:`OSError` is raised with + :attr:`~OSError.errno` set to :data:`errno.EXDEV`. + + This copy is done without the additional cost of transferring data + from the kernel to user space and then back into the kernel. Additionally, + some filesystems could implement extra optimizations. The copy is done as if + both files are opened as binary. + + Upon successful completion, returns the number of bytes spliced to or from + the pipe. A return value of 0 means end of input. If *src* refers to a + pipe, then this means that there was no data to transfer, and it would not + make sense to block because there are no writers connected to the write end + of the pipe. + + .. availability:: Linux kernel >= 2.6.17 and glibc >= 2.5 + + .. versionadded:: 3.10 + + +.. data:: SPLICE_F_MOVE + SPLICE_F_NONBLOCK + SPLICE_F_MORE + + .. versionadded:: 3.10 + .. function:: readv(fd, buffers) Read from a file descriptor *fd* into a number of mutable :term:`bytes-like @@ -1950,7 +1991,7 @@ features: Accepts a :term:`path-like object`. -.. function:: lstat(path, \*, dir_fd=None) +.. function:: lstat(path, *, dir_fd=None) Perform the equivalent of an :c:func:`lstat` system call on the given path. Similar to :func:`~os.stat`, but does not follow symbolic links. Return a @@ -2457,7 +2498,7 @@ features: On the first, uncached call, a system call is required on Windows but not on Unix. - .. method:: is_dir(\*, follow_symlinks=True) + .. method:: is_dir(*, follow_symlinks=True) Return ``True`` if this entry is a directory or a symbolic link pointing to a directory; return ``False`` if the entry is or points to any other @@ -2481,7 +2522,7 @@ features: This method can raise :exc:`OSError`, such as :exc:`PermissionError`, but :exc:`FileNotFoundError` is caught and not raised. - .. method:: is_file(\*, follow_symlinks=True) + .. method:: is_file(*, follow_symlinks=True) Return ``True`` if this entry is a file or a symbolic link pointing to a file; return ``False`` if the entry is or points to a directory or other @@ -2511,7 +2552,7 @@ features: This method can raise :exc:`OSError`, such as :exc:`PermissionError`, but :exc:`FileNotFoundError` is caught and not raised. - .. method:: stat(\*, follow_symlinks=True) + .. method:: stat(*, follow_symlinks=True) Return a :class:`stat_result` object for this entry. This method follows symbolic links by default; to stat a symbolic link add the @@ -2543,7 +2584,7 @@ features: for :class:`bytes` paths on Windows. -.. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) +.. function:: stat(path, *, dir_fd=None, follow_symlinks=True) Get the status of a file or a file descriptor. Perform the equivalent of a :c:func:`stat` system call on the given path. *path* may be specified as @@ -3268,6 +3309,102 @@ features: .. versionadded:: 3.8 +.. function:: eventfd(initval[, flags=os.EFD_CLOEXEC]) + + Create and return an event file descriptor. The file descriptors supports + raw :func:`read` and :func:`write` with a buffer size of 8, + :func:`~select.select`, :func:`~select.poll` and similar. See man page + :manpage:`eventfd(2)` for more information. By default, the + new file descriptor is :ref:`non-inheritable <fd_inheritance>`. + + *initval* is the initial value of the event counter. The initial value + must be an 32 bit unsigned integer. Please note that the initial value is + limited to a 32 bit unsigned int although the event counter is an unsigned + 64 bit integer with a maximum value of 2\ :sup:`64`\ -\ 2. + + *flags* can be constructed from :const:`EFD_CLOEXEC`, + :const:`EFD_NONBLOCK`, and :const:`EFD_SEMAPHORE`. + + If :const:`EFD_SEMAPHORE` is specified and the event counter is non-zero, + :func:`eventfd_read` returns 1 and decrements the counter by one. + + If :const:`EFD_SEMAPHORE` is not specified and the event counter is + non-zero, :func:`eventfd_read` returns the current event counter value and + resets the counter to zero. + + If the event counter is zero and :const:`EFD_NONBLOCK` is not + specified, :func:`eventfd_read` blocks. + + :func:`eventfd_write` increments the event counter. Write blocks if the + write operation would increment the counter to a value larger than + 2\ :sup:`64`\ -\ 2. + + Example:: + + import os + + # semaphore with start value '1' + fd = os.eventfd(1, os.EFD_SEMAPHORE | os.EFC_CLOEXEC) + try: + # acquire semaphore + v = os.eventfd_read(fd) + try: + do_work() + finally: + # release semaphore + os.eventfd_write(fd, v) + finally: + os.close(fd) + + .. availability:: Linux 2.6.27 or newer with glibc 2.8 or newer. + + .. versionadded:: 3.10 + +.. function:: eventfd_read(fd) + + Read value from an :func:`eventfd` file descriptor and return a 64 bit + unsigned int. The function does not verify that *fd* is an :func:`eventfd`. + + .. availability:: See :func:`eventfd` + + .. versionadded:: 3.10 + +.. function:: eventfd_write(fd, value) + + Add value to an :func:`eventfd` file descriptor. *value* must be a 64 bit + unsigned int. The function does not verify that *fd* is an :func:`eventfd`. + + .. availability:: See :func:`eventfd` + + .. versionadded:: 3.10 + +.. data:: EFD_CLOEXEC + + Set close-on-exec flag for new :func:`eventfd` file descriptor. + + .. availability:: See :func:`eventfd` + + .. versionadded:: 3.10 + +.. data:: EFD_NONBLOCK + + Set :const:`O_NONBLOCK` status flag for new :func:`eventfd` file + descriptor. + + .. availability:: See :func:`eventfd` + + .. versionadded:: 3.10 + +.. data:: EFD_SEMAPHORE + + Provide semaphore-like semantics for reads from a :func:`eventfd` file + descriptor. On read the internal counter is decremented by one. + + .. availability:: Linux 2.6.30 or newer with glibc 2.8 or newer. + + .. versionadded:: 3.10 + + Linux extended attributes ~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 9de72bb725..2bbf3aad61 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -336,6 +336,8 @@ Pure paths provide the following methods and properties: >>> p.parents[2] PureWindowsPath('c:/') + .. versionchanged:: 3.10 + The parents sequence now supports :term:`slices <slice>` and negative index values. .. data:: PurePath.parent diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index b7c3452771..be48561ed1 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -213,7 +213,7 @@ The :mod:`pickle` module provides the following constants: The :mod:`pickle` module provides the following functions to make the pickling process more convenient: -.. function:: dump(obj, file, protocol=None, \*, fix_imports=True, buffer_callback=None) +.. function:: dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None) Write the pickled representation of the object *obj* to the open :term:`file object` *file*. This is equivalent to @@ -225,7 +225,7 @@ process more convenient: .. versionchanged:: 3.8 The *buffer_callback* argument was added. -.. function:: dumps(obj, protocol=None, \*, fix_imports=True, buffer_callback=None) +.. function:: dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None) Return the pickled representation of the object *obj* as a :class:`bytes` object, instead of writing it to a file. @@ -236,7 +236,7 @@ process more convenient: .. versionchanged:: 3.8 The *buffer_callback* argument was added. -.. function:: load(file, \*, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) +.. function:: load(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) Read the pickled representation of an object from the open :term:`file object` *file* and return the reconstituted object hierarchy specified therein. @@ -252,7 +252,7 @@ process more convenient: .. versionchanged:: 3.8 The *buffers* argument was added. -.. function:: loads(data, /, \*, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) +.. function:: loads(data, /, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) Return the reconstituted object hierarchy of the pickled representation *data* of an object. *data* must be a :term:`bytes-like object`. @@ -296,7 +296,7 @@ The :mod:`pickle` module defines three exceptions: The :mod:`pickle` module exports three classes, :class:`Pickler`, :class:`Unpickler` and :class:`PickleBuffer`: -.. class:: Pickler(file, protocol=None, \*, fix_imports=True, buffer_callback=None) +.. class:: Pickler(file, protocol=None, *, fix_imports=True, buffer_callback=None) This takes a binary file for writing a pickle data stream. @@ -391,7 +391,7 @@ The :mod:`pickle` module exports three classes, :class:`Pickler`, Use :func:`pickletools.optimize` if you need more compact pickles. -.. class:: Unpickler(file, \*, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) +.. class:: Unpickler(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) This takes a binary file for reading a pickle data stream. diff --git a/Doc/library/pipes.rst b/Doc/library/pipes.rst index 0a22da1f55..57e27a6acf 100644 --- a/Doc/library/pipes.rst +++ b/Doc/library/pipes.rst @@ -17,6 +17,8 @@ The :mod:`pipes` module defines a class to abstract the concept of a *pipeline* Because the module uses :program:`/bin/sh` command lines, a POSIX or compatible shell for :func:`os.system` and :func:`os.popen` is required. +.. availability:: Unix. Not available on VxWorks. + The :mod:`pipes` module defines the following class: diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index 8e8e3775aa..be86e568c1 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -209,13 +209,6 @@ Windows Platform which means the OS version uses debugging code, i.e. code that checks arguments, ranges, etc. - .. note:: - - This function works best with Mark Hammond's - :mod:`win32all` package installed, but also on Python 2.3 and - later (support for this was added in Python 2.6). It obviously - only runs on Win32 compatible platforms. - .. function:: win32_edition() Returns a string representing the current Windows edition. Possible @@ -260,3 +253,41 @@ Unix Platforms using :program:`gcc`. The file is read and scanned in chunks of *chunksize* bytes. + + +Linux Platforms +--------------- + +.. function:: freedesktop_os_release() + + Get operating system identification from ``os-release`` file and return + it as a dict. The ``os-release`` file is a `freedesktop.org standard + <https://www.freedesktop.org/software/systemd/man/os-release.html>`_ and + is available in most Linux distributions. A noticeable exception is + Android and Android-based distributions. + + Raises :exc:`OSError` or subclass when neither ``/etc/os-release`` nor + ``/usr/lib/os-release`` can be read. + + On success, the function returns a dictionary where keys and values are + strings. Values have their special characters like ``"`` and ``$`` + unquoted. The fields ``NAME``, ``ID``, and ``PRETTY_NAME`` are always + defined according to the standard. All other fields are optional. Vendors + may include additional fields. + + Note that fields like ``NAME``, ``VERSION``, and ``VARIANT`` are strings + suitable for presentation to users. Programs should use fields like + ``ID``, ``ID_LIKE``, ``VERSION_ID``, or ``VARIANT_ID`` to identify + Linux distributions. + + Example:: + + def get_like_distro(): + info = platform.freedesktop_os_release() + ids = [info["ID"]] + if "ID_LIKE" in info: + # ids are space separated and ordered by precedence + ids.extend(info["ID_LIKE"].split()) + return ids + + .. versionadded:: 3.10 diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst index 6def72b373..ce6d4a85bf 100644 --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -52,7 +52,7 @@ or :class:`datetime.datetime` objects. This module defines the following functions: -.. function:: load(fp, \*, fmt=None, dict_type=dict) +.. function:: load(fp, *, fmt=None, dict_type=dict) Read a plist file. *fp* should be a readable and binary file object. Return the unpacked root object (which usually is a @@ -80,7 +80,7 @@ This module defines the following functions: .. versionadded:: 3.4 -.. function:: loads(data, \*, fmt=None, dict_type=dict) +.. function:: loads(data, *, fmt=None, dict_type=dict) Load a plist from a bytes object. See :func:`load` for an explanation of the keyword arguments. @@ -88,7 +88,7 @@ This module defines the following functions: .. versionadded:: 3.4 -.. function:: dump(value, fp, \*, fmt=FMT_XML, sort_keys=True, skipkeys=False) +.. function:: dump(value, fp, *, fmt=FMT_XML, sort_keys=True, skipkeys=False) Write *value* to a plist file. *Fp* should be a writable, binary file object. @@ -116,7 +116,7 @@ This module defines the following functions: .. versionadded:: 3.4 -.. function:: dumps(value, \*, fmt=FMT_XML, sort_keys=True, skipkeys=False) +.. function:: dumps(value, *, fmt=FMT_XML, sort_keys=True, skipkeys=False) Return *value* as a plist-formatted bytes object. See the documentation for :func:`dump` for an explanation of the keyword diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 34525a96f5..774d46d0e9 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -525,16 +525,16 @@ Analysis of the profiler data is done using the :class:`~pstats.Stats` class. ordering are identical to the :meth:`~pstats.Stats.print_callers` method. - .. method:: get_stats_profile() + .. method:: get_stats_profile() This method returns an instance of StatsProfile, which contains a mapping of function names to instances of FunctionProfile. Each FunctionProfile instance holds information related to the function's profile such as how long the function took to run, how many times it was called, etc... - .. versionadded:: 3.9 - Added the following dataclasses: StatsProfile, FunctionProfile. - Added the following function: get_stats_profile. + .. versionadded:: 3.9 + Added the following dataclasses: StatsProfile, FunctionProfile. + Added the following function: get_stats_profile. .. _deterministic-profiling: diff --git a/Doc/library/pyclbr.rst b/Doc/library/pyclbr.rst index 36e83e85c2..1c40ba4838 100644 --- a/Doc/library/pyclbr.rst +++ b/Doc/library/pyclbr.rst @@ -97,6 +97,13 @@ statements. They have the following attributes: .. versionadded:: 3.7 +.. attribute:: Function.is_async + + ``True`` for functions that are defined with the ``async`` prefix, ``False`` otherwise. + + .. versionadded:: 3.10 + + .. _pyclbr-class-objects: Class Objects diff --git a/Doc/library/random.rst b/Doc/library/random.rst index c243aced98..fa5c74b40b 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -142,10 +142,11 @@ Functions for integers .. function:: getrandbits(k) - Returns a Python integer with *k* random bits. This method is supplied with - the MersenneTwister generator and some other generators may also provide it - as an optional part of the API. When available, :meth:`getrandbits` enables - :meth:`randrange` to handle arbitrarily large ranges. + Returns a non-negative Python integer with *k* random bits. This method + is supplied with the MersenneTwister generator and some other generators + may also provide it as an optional part of the API. When available, + :meth:`getrandbits` enables :meth:`randrange` to handle arbitrarily large + ranges. .. versionchanged:: 3.9 This method now accepts zero for *k*. diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst index 7f7f0c7f12..aab6a54379 100644 --- a/Doc/library/shlex.rst +++ b/Doc/library/shlex.rst @@ -61,6 +61,20 @@ The :mod:`shlex` module defines the following functions: string that can safely be used as one token in a shell command line, for cases where you cannot use a list. + .. _shlex-quote-warning: + + .. warning:: + + The ``shlex`` module is **only designed for Unix shells**. + + The :func:`quote` function is not guaranteed to be correct on non-POSIX + compliant shells or shells from other operating systems such as Windows. + Executing commands quoted by this module on such shells can open up the + possibility of a command injection vulnerability. + + Consider using functions that pass command arguments with lists such as + :func:`subprocess.run` with ``shell=False``. + This idiom would be unsafe: >>> filename = 'somefile; rm -rf ~' diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 3f5122760e..435787c276 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -218,7 +218,7 @@ Directory and files operations copy the file more efficiently. See :ref:`shutil-platform-dependent-efficient-copy-operations` section. -.. function:: ignore_patterns(\*patterns) +.. function:: ignore_patterns(*patterns) This factory function creates a function that can be used as a callable for :func:`copytree`\'s *ignore* argument, ignoring files and directories that diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 00a730b6b9..84a569d03e 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -117,7 +117,7 @@ The variables defined in the :mod:`signal` module are: Child process stopped or terminated. - .. availability:: Windows. + .. availability:: Unix. .. data:: SIGCLD diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index b3cc60357f..52220f7a7f 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -32,7 +32,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). than a success code, an :exc:`SMTPConnectError` is raised. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout - setting will be used). If the timeout expires, :exc:`socket.timeout` is + setting will be used). If the timeout expires, :exc:`TimeoutError` is raised. The optional source_address parameter allows binding to some specific source address in a machine with multiple network interfaces, and/or to some specific source TCP port. It takes a 2-tuple @@ -115,7 +115,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). If the *timeout* parameter is set to be zero, it will raise a :class:`ValueError` to prevent the creation of a non-blocking socket -.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, +.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, \ source_address=None[, timeout]) The LMTP protocol, which is very similar to ESMTP, is heavily based on the diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index faf8a76251..4511ff9ea4 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -56,12 +56,12 @@ created. Socket addresses are represented as follows: bytes-like object can be used for either type of address when passing it as an argument. - .. versionchanged:: 3.3 - Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 - encoding. + .. versionchanged:: 3.3 + Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 + encoding. - .. versionchanged:: 3.5 - Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. .. _host_port: @@ -283,6 +283,8 @@ Exceptions .. exception:: timeout + A deprecated alias of :exc:`TimeoutError`. + A subclass of :exc:`OSError`, this exception is raised when a timeout occurs on a socket which has had timeouts enabled via a prior call to :meth:`~socket.settimeout` (or implicitly through @@ -292,6 +294,9 @@ Exceptions .. versionchanged:: 3.3 This class was made a subclass of :exc:`OSError`. + .. versionchanged:: 3.10 + This class was made an alias of :exc:`TimeoutError`. + Constants ^^^^^^^^^ @@ -1208,7 +1213,7 @@ to sockets. address family --- see above.) If the connection is interrupted by a signal, the method waits until the - connection completes, or raise a :exc:`socket.timeout` on timeout, if the + connection completes, or raise a :exc:`TimeoutError` on timeout, if the signal handler doesn't raise an exception and the socket is blocking or has a timeout. For non-blocking sockets, the method raises an :exc:`InterruptedError` exception if the connection is interrupted by a diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 13aa8c512d..c36648775a 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -198,7 +198,9 @@ Module functions and constants *detect_types* defaults to 0 (i. e. off, no type detection), you can set it to any combination of :const:`PARSE_DECLTYPES` and :const:`PARSE_COLNAMES` to turn - type detection on. + type detection on. Due to SQLite behaviour, types can't be detected for generated + fields (for example ``max(data)``), even when *detect_types* parameter is set. In + such case, the returned type is :class:`str`. By default, *check_same_thread* is :const:`True` and only the creating thread may use the connection. If set :const:`False`, the returned connection may be shared diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst index 38a499ab37..6467704006 100644 --- a/Doc/library/statistics.rst +++ b/Doc/library/statistics.rst @@ -156,10 +156,11 @@ However, for reading convenience, most of the examples show sorted sequences. .. versionadded:: 3.8 -.. function:: harmonic_mean(data) +.. function:: harmonic_mean(data, weights=None) Return the harmonic mean of *data*, a sequence or iterable of - real-valued numbers. + real-valued numbers. If *weights* is omitted or *None*, then + equal weighting is assumed. The harmonic mean, sometimes called the subcontrary mean, is the reciprocal of the arithmetic :func:`mean` of the reciprocals of the @@ -179,17 +180,17 @@ However, for reading convenience, most of the examples show sorted sequences. >>> harmonic_mean([40, 60]) 48.0 - Suppose an investor purchases an equal value of shares in each of - three companies, with P/E (price/earning) ratios of 2.5, 3 and 10. - What is the average P/E ratio for the investor's portfolio? + Suppose a car travels 40 km/hr for 5 km, and when traffic clears, + speeds-up to 60 km/hr for the remaining 30 km of the journey. What + is the average speed? .. doctest:: - >>> harmonic_mean([2.5, 3, 10]) # For an equal investment portfolio. - 3.6 + >>> harmonic_mean([40, 60], weights=[5, 30]) + 56.0 - :exc:`StatisticsError` is raised if *data* is empty, or any element - is less than zero. + :exc:`StatisticsError` is raised if *data* is empty, any element + is less than zero, or if the weighted sum isn't positive. The current algorithm has an early-out when it encounters a zero in the input. This means that the subsequent inputs are not tested @@ -197,6 +198,8 @@ However, for reading convenience, most of the examples show sorted sequences. .. versionadded:: 3.6 + .. versionchanged:: 3.8 + Added support for *weights*. .. function:: median(data) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 84d5a3a59f..2869378bbd 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -499,7 +499,7 @@ class`. In addition, it provides a few more methods: .. versionadded:: 3.10 -.. method:: int.to_bytes(length, byteorder, \*, signed=False) +.. method:: int.to_bytes(length, byteorder, *, signed=False) Return an array of bytes representing an integer. @@ -531,7 +531,7 @@ class`. In addition, it provides a few more methods: .. versionadded:: 3.2 -.. classmethod:: int.from_bytes(bytes, byteorder, \*, signed=False) +.. classmethod:: int.from_bytes(bytes, byteorder, *, signed=False) Return the integer represented by the given array of bytes. @@ -5065,7 +5065,7 @@ instantiated from the type:: >>> class M(type): ... def __or__(self, other): - ... return "Hello" + ... return "Hello" ... >>> class C(metaclass=M): ... pass @@ -5353,8 +5353,8 @@ types, where they are relevant. Some of these are not reported by the .. method:: class.__subclasses__ Each class keeps a list of weak references to its immediate subclasses. This - method returns a list of all those references still alive. - Example:: + method returns a list of all those references still alive. The list is in + definition order. Example:: >>> int.__subclasses__() [<class 'bool'>] diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 91f43e9353..54786d0c2a 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -384,10 +384,10 @@ following: The ``'#'`` option causes the "alternate form" to be used for the conversion. The alternate form is defined differently for different -types. This option is only valid for integer, float, complex and -Decimal types. For integers, when binary, octal, or hexadecimal output +types. This option is only valid for integer, float and complex +types. For integers, when binary, octal, or hexadecimal output is used, this option adds the prefix respective ``'0b'``, ``'0o'``, or -``'0x'`` to the output value. For floats, complex and Decimal the +``'0x'`` to the output value. For float and complex the alternate form causes the result of the conversion to always contain a decimal-point character, even if no digits follow it. Normally, a decimal-point character appears in the result of these conversions @@ -476,20 +476,36 @@ with the floating point presentation types listed below (except ``'n'`` and ``None``). When doing so, :func:`float` is used to convert the integer to a floating point number before formatting. -The available presentation types for floating point and decimal values are: +The available presentation types for :class:`float` and +:class:`~decimal.Decimal` values are: +---------+----------------------------------------------------------+ | Type | Meaning | +=========+==========================================================+ - | ``'e'`` | Exponent notation. Prints the number in scientific | - | | notation using the letter 'e' to indicate the exponent. | - | | The default precision is ``6``. | + | ``'e'`` | Scientific notation. For a given precision ``p``, | + | | formats the number in scientific notation with the | + | | letter 'e' separating the coefficient from the exponent. | + | | The coefficient has one digit before and ``p`` digits | + | | after the decimal point, for a total of ``p + 1`` | + | | significant digits. With no precision given, uses a | + | | precision of ``6`` digits after the decimal point for | + | | :class:`float`, and shows all coefficient digits | + | | for :class:`~decimal.Decimal`. If no digits follow the | + | | decimal point, the decimal point is also removed unless | + | | the ``#`` option is used. | +---------+----------------------------------------------------------+ - | ``'E'`` | Exponent notation. Same as ``'e'`` except it uses an | - | | upper case 'E' as the separator character. | + | ``'E'`` | Scientific notation. Same as ``'e'`` except it uses | + | | an upper case 'E' as the separator character. | +---------+----------------------------------------------------------+ - | ``'f'`` | Fixed-point notation. Displays the number as a | - | | fixed-point number. The default precision is ``6``. | + | ``'f'`` | Fixed-point notation. For a given precision ``p``, | + | | formats the number as a decimal number with exactly | + | | ``p`` digits following the decimal point. With no | + | | precision given, uses a precision of ``6`` digits after | + | | the decimal point for :class:`float`, and uses a | + | | precision large enough to show all coefficient digits | + | | for :class:`~decimal.Decimal`. If no digits follow the | + | | decimal point, the decimal point is also removed unless | + | | the ``#`` option is used. | +---------+----------------------------------------------------------+ | ``'F'`` | Fixed-point notation. Same as ``'f'``, but converts | | | ``nan`` to ``NAN`` and ``inf`` to ``INF``. | @@ -498,6 +514,8 @@ The available presentation types for floating point and decimal values are: | | this rounds the number to ``p`` significant digits and | | | then formats the result in either fixed-point format | | | or in scientific notation, depending on its magnitude. | + | | A precision of ``0`` is treated as equivalent to a | + | | precision of ``1``. | | | | | | The precise rules are as follows: suppose that the | | | result formatted with presentation type ``'e'`` and | @@ -512,13 +530,19 @@ The available presentation types for floating point and decimal values are: | | removed if there are no remaining digits following it, | | | unless the ``'#'`` option is used. | | | | + | | With no precision given, uses a precision of ``6`` | + | | significant digits for :class:`float`. For | + | | :class:`~decimal.Decimal`, the coefficient of the result | + | | is formed from the coefficient digits of the value; | + | | scientific notation is used for values smaller than | + | | ``1e-6`` in absolute value and values where the place | + | | value of the least significant digit is larger than 1, | + | | and fixed-point notation is used otherwise. | + | | | | | Positive and negative infinity, positive and negative | | | zero, and nans, are formatted as ``inf``, ``-inf``, | | | ``0``, ``-0`` and ``nan`` respectively, regardless of | | | the precision. | - | | | - | | A precision of ``0`` is treated as equivalent to a | - | | precision of ``1``. The default precision is ``6``. | +---------+----------------------------------------------------------+ | ``'G'`` | General format. Same as ``'g'`` except switches to | | | ``'E'`` if the number gets too large. The | @@ -531,12 +555,18 @@ The available presentation types for floating point and decimal values are: | ``'%'`` | Percentage. Multiplies the number by 100 and displays | | | in fixed (``'f'``) format, followed by a percent sign. | +---------+----------------------------------------------------------+ - | None | Similar to ``'g'``, except that fixed-point notation, | - | | when used, has at least one digit past the decimal point.| - | | The default precision is as high as needed to represent | - | | the particular value. The overall effect is to match the | - | | output of :func:`str` as altered by the other format | - | | modifiers. | + | None | For :class:`float` this is the same as ``'g'``, except | + | | that when fixed-point notation is used to format the | + | | result, it always includes at least one digit past the | + | | decimal point. The precision used is as large as needed | + | | to represent the given value faithfully. | + | | | + | | For :class:`~decimal.Decimal`, this is the same as | + | | either ``'g'`` or ``'G'`` depending on the value of | + | | ``context.capitals`` for the current decimal context. | + | | | + | | The overall effect is to match the output of :func:`str` | + | | as altered by the other format modifiers. | +---------+----------------------------------------------------------+ diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 85d0f46624..07b8ad7765 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -339,7 +339,7 @@ functions. stderr=None, preexec_fn=None, close_fds=True, shell=False, \ cwd=None, env=None, universal_newlines=None, \ startupinfo=None, creationflags=0, restore_signals=True, \ - start_new_session=False, pass_fds=(), \*, group=None, \ + start_new_session=False, pass_fds=(), *, group=None, \ extra_groups=None, user=None, umask=-1, \ encoding=None, errors=None, text=None, pipesize=-1) @@ -718,11 +718,8 @@ If the shell is invoked explicitly, via ``shell=True``, it is the application's responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid `shell injection <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ -vulnerabilities. - -When using ``shell=True``, the :func:`shlex.quote` function can be -used to properly escape whitespace and shell metacharacters in strings -that are going to be used to construct shell commands. +vulnerabilities. On :ref:`some platforms <shlex-quote-warning>`, it is possible +to use :func:`shlex.quote` for this escaping. Popen Objects @@ -1190,8 +1187,9 @@ calls these functions. The arguments shown above are merely some common ones. The full function signature is largely the same as that of :func:`run` - most arguments are passed directly through to that interface. - However, explicitly passing ``input=None`` to inherit the parent's - standard input file handle is not supported. + One API deviation from :func:`run` behavior exists: passing ``input=None`` + will behave the same as ``input=b''`` (or ``input=''``, depending on other + arguments) rather than using the parent's standard input file handle. By default, this function will return the data as encoded bytes. The actual encoding of the output data may depend on the command being invoked, so the @@ -1284,7 +1282,7 @@ be used directly: becomes:: - output=check_output("dmesg | grep hda", shell=True) + output = check_output("dmesg | grep hda", shell=True) Replacing :func:`os.system` diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index c9521d649b..40c1795b08 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -156,7 +156,7 @@ Examining Symbol Tables Return ``True`` if the symbol is local to its block. - .. method:: is_annotated() + .. method:: is_annotated() Return ``True`` if the symbol is annotated. diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst index 78a1dfce9a..c9306e9bf9 100644 --- a/Doc/library/sysconfig.rst +++ b/Doc/library/sysconfig.rst @@ -32,7 +32,7 @@ can be accessed using :func:`get_config_vars` or :func:`get_config_var`. Notice that on Windows, it's a much smaller set. -.. function:: get_config_vars(\*args) +.. function:: get_config_vars(*args) With no arguments, return a dictionary of all configuration variables relevant for the current platform. diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 7a114fdf5d..13088a10d7 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -37,7 +37,7 @@ Some facts and figures: Added support for :mod:`lzma` compression. -.. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, \*\*kwargs) +.. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, **kwargs) Return a :class:`TarFile` object for the pathname *name*. For detailed information on :class:`TarFile` objects and the keyword arguments that are diff --git a/Doc/library/test.rst b/Doc/library/test.rst index ce6b868458..e4f779bd83 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -453,7 +453,7 @@ The :mod:`test.support` module defines the following functions: Define match test with regular expression *patterns*. -.. function:: run_unittest(\*classes) +.. function:: run_unittest(*classes) Execute :class:`unittest.TestCase` subclasses passed to the function. The function scans the classes for methods starting with the prefix ``test_`` @@ -1599,7 +1599,7 @@ The :mod:`test.support.warnings_helper` module provides support for warnings tes .. versionadded:: 3.8 -.. function:: check_warnings(\*filters, quiet=True) +.. function:: check_warnings(*filters, quiet=True) A convenience wrapper for :func:`warnings.catch_warnings()` that makes it easier to test that a warning was correctly raised. It is approximately diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index 7eb12fe116..690735469d 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -71,6 +71,13 @@ This module defines the following functions: .. versionadded:: 3.8 +.. data:: __excepthook__ + + Holds the original value of :func:`threading.excepthook`. It is saved so that the + original value can be restored in case they happen to get replaced with + broken or alternative objects. + + .. versionadded:: 3.10 .. function:: get_ident() @@ -121,6 +128,17 @@ This module defines the following functions: :meth:`~Thread.run` method is called. +.. function:: gettrace() + + .. index:: + single: trace function + single: debugger + + Get the trace function as set by :func:`settrace`. + + .. versionadded:: 3.10 + + .. function:: setprofile(func) .. index:: single: profile function @@ -130,6 +148,15 @@ This module defines the following functions: :meth:`~Thread.run` method is called. +.. function:: getprofile() + + .. index:: single: profile function + + Get the profiler function as set by :func:`setprofile`. + + .. versionadded:: 3.10 + + .. function:: stack_size([size]) Return the thread stack size used when creating new threads. The optional diff --git a/Doc/library/time.rst b/Doc/library/time.rst index cff6320b52..143f84b565 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -166,6 +166,9 @@ Functions Return the time of the specified clock *clk_id*. Refer to :ref:`time-clock-id-constants` for a list of accepted values for *clk_id*. + Use :func:`clock_gettime_ns` to avoid the precision loss caused by the + :class:`float` type. + .. availability:: Unix. .. versionadded:: 3.3 @@ -185,6 +188,9 @@ Functions Set the time of the specified clock *clk_id*. Currently, :data:`CLOCK_REALTIME` is the only accepted value for *clk_id*. + Use :func:`clock_settime_ns` to avoid the precision loss caused by the + :class:`float` type. + .. availability:: Unix. .. versionadded:: 3.3 @@ -273,10 +279,17 @@ Functions The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid. + Use :func:`monotonic_ns` to avoid the precision loss caused by the + :class:`float` type. + .. versionadded:: 3.3 + .. versionchanged:: 3.5 The function is now always available and always system-wide. + .. versionchanged:: 3.10 + On macOS, the function is now system-wide. + .. function:: monotonic_ns() -> int @@ -295,8 +308,14 @@ Functions point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid. + Use :func:`perf_counter_ns` to avoid the precision loss caused by the + :class:`float` type. + .. versionadded:: 3.3 + .. versionchanged:: 3.10 + On Windows, the function is now system-wide. + .. function:: perf_counter_ns() -> int Similar to :func:`perf_counter`, but return time as nanoseconds. @@ -317,6 +336,9 @@ Functions returned value is undefined, so that only the difference between the results of consecutive calls is valid. + Use :func:`process_time_ns` to avoid the precision loss caused by the + :class:`float` type. + .. versionadded:: 3.3 .. function:: process_time_ns() -> int @@ -581,6 +603,17 @@ Functions :class:`struct_time` object is returned, from which the components of the calendar date may be accessed as attributes. + Use :func:`time_ns` to avoid the precision loss caused by the :class:`float` + type. + + +.. function:: time_ns() -> int + + Similar to :func:`~time.time` but returns time as an integer number of nanoseconds + since the epoch_. + + .. versionadded:: 3.7 + .. function:: thread_time() -> float @@ -595,6 +628,9 @@ Functions returned value is undefined, so that only the difference between the results of consecutive calls in the same thread is valid. + Use :func:`thread_time_ns` to avoid the precision loss caused by the + :class:`float` type. + .. availability:: Windows, Linux, Unix systems supporting ``CLOCK_THREAD_CPUTIME_ID``. @@ -608,13 +644,6 @@ Functions .. versionadded:: 3.7 -.. function:: time_ns() -> int - - Similar to :func:`~time.time` but returns time as an integer number of nanoseconds - since the epoch_. - - .. versionadded:: 3.7 - .. function:: tzset() Reset the time conversion rules used by the library routines. The environment diff --git a/Doc/library/tkinter.font.rst b/Doc/library/tkinter.font.rst index b0f4505e9e..c7c2b7b566 100644 --- a/Doc/library/tkinter.font.rst +++ b/Doc/library/tkinter.font.rst @@ -91,6 +91,9 @@ The different font weights and slants are: Return the names of defined fonts. -.. function:: nametofont(name) +.. function:: nametofont(name, root=None) - Return a :class:`Font` representation of a tk named font.
\ No newline at end of file + Return a :class:`Font` representation of a tk named font. + + .. versionchanged:: 3.10 + The *root* parameter was added. diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst index c2732d900b..40cf198f12 100644 --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -153,47 +153,47 @@ Programmatic Interface count information. *timing* enables a timestamp relative to when tracing was started to be displayed. - .. method:: run(cmd) + .. method:: run(cmd) - Execute the command and gather statistics from the execution with - the current tracing parameters. *cmd* must be a string or code object, - suitable for passing into :func:`exec`. + Execute the command and gather statistics from the execution with + the current tracing parameters. *cmd* must be a string or code object, + suitable for passing into :func:`exec`. - .. method:: runctx(cmd, globals=None, locals=None) + .. method:: runctx(cmd, globals=None, locals=None) - Execute the command and gather statistics from the execution with the - current tracing parameters, in the defined global and local - environments. If not defined, *globals* and *locals* default to empty - dictionaries. + Execute the command and gather statistics from the execution with the + current tracing parameters, in the defined global and local + environments. If not defined, *globals* and *locals* default to empty + dictionaries. - .. method:: runfunc(func, /, *args, **kwds) + .. method:: runfunc(func, /, *args, **kwds) - Call *func* with the given arguments under control of the :class:`Trace` - object with the current tracing parameters. + Call *func* with the given arguments under control of the :class:`Trace` + object with the current tracing parameters. - .. method:: results() + .. method:: results() - Return a :class:`CoverageResults` object that contains the cumulative - results of all previous calls to ``run``, ``runctx`` and ``runfunc`` - for the given :class:`Trace` instance. Does not reset the accumulated - trace results. + Return a :class:`CoverageResults` object that contains the cumulative + results of all previous calls to ``run``, ``runctx`` and ``runfunc`` + for the given :class:`Trace` instance. Does not reset the accumulated + trace results. .. class:: CoverageResults A container for coverage results, created by :meth:`Trace.results`. Should not be created directly by the user. - .. method:: update(other) + .. method:: update(other) - Merge in data from another :class:`CoverageResults` object. + Merge in data from another :class:`CoverageResults` object. - .. method:: write_results(show_missing=True, summary=False, coverdir=None) + .. method:: write_results(show_missing=True, summary=False, coverdir=None) - Write coverage results. Set *show_missing* to show lines that had no - hits. Set *summary* to include in the output the coverage summary per - module. *coverdir* specifies the directory into which the coverage - result files will be output. If ``None``, the results for each source - file are placed in its directory. + Write coverage results. Set *show_missing* to show lines that had no + hits. Set *summary* to include in the output the coverage summary per + module. *coverdir* specifies the directory into which the coverage + result files will be output. If ``None``, the results for each source + file are placed in its directory. A simple example demonstrating the use of the programmatic interface:: diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 462a6a5566..c233f18d30 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -36,7 +36,8 @@ The module defines the following functions: Added negative *limit* support. -.. function:: print_exception(etype, value, tb, limit=None, file=None, chain=True) +.. function:: print_exception(exc, /[, value, tb], limit=None, \ + file=None, chain=True) Print exception information and stack trace entries from traceback object *tb* to *file*. This differs from :func:`print_tb` in the following @@ -45,7 +46,7 @@ The module defines the following functions: * if *tb* is not ``None``, it prints a header ``Traceback (most recent call last):`` - * it prints the exception *etype* and *value* after the stack trace + * it prints the exception type and *value* after the stack trace .. index:: single: ^ (caret); marker @@ -53,6 +54,10 @@ The module defines the following functions: format, it prints the line where the syntax error occurred with a caret indicating the approximate position of the error. + Since Python 3.10, instead of passing *value* and *tb*, an exception object + can be passed as the first argument. If *value* and *tb* are provided, the + first argument is ignored in order to provide backwards compatibility. + The optional *limit* argument has the same meaning as for :func:`print_tb`. If *chain* is true (the default), then chained exceptions (the :attr:`__cause__` or :attr:`__context__` attributes of the exception) will be @@ -62,6 +67,10 @@ The module defines the following functions: .. versionchanged:: 3.5 The *etype* argument is ignored and inferred from the type of *value*. + .. versionchanged:: 3.10 + The *etype* parameter has been renamed to *exc* and is now + positional-only. + .. function:: print_exc(limit=None, file=None, chain=True) @@ -121,18 +130,26 @@ The module defines the following functions: text line is not ``None``. -.. function:: format_exception_only(etype, value) +.. function:: format_exception_only(exc, /[, value]) + + Format the exception part of a traceback using an exception value such as + given by ``sys.last_value``. The return value is a list of strings, each + ending in a newline. Normally, the list contains a single string; however, + for :exc:`SyntaxError` exceptions, it contains several lines that (when + printed) display detailed information about where the syntax error occurred. + The message indicating which exception occurred is the always last string in + the list. - Format the exception part of a traceback. The arguments are the exception - type and value such as given by ``sys.last_type`` and ``sys.last_value``. - The return value is a list of strings, each ending in a newline. Normally, - the list contains a single string; however, for :exc:`SyntaxError` - exceptions, it contains several lines that (when printed) display detailed - information about where the syntax error occurred. The message indicating - which exception occurred is the always last string in the list. + Since Python 3.10, instead of passing *value*, an exception object + can be passed as the first argument. If *value* is provided, the first + argument is ignored in order to provide backwards compatibility. + .. versionchanged:: 3.10 + The *etype* parameter has been renamed to *exc* and is now + positional-only. -.. function:: format_exception(etype, value, tb, limit=None, chain=True) + +.. function:: format_exception(exc, /[, value, tb], limit=None, chain=True) Format a stack trace and the exception information. The arguments have the same meaning as the corresponding arguments to :func:`print_exception`. The @@ -143,6 +160,10 @@ The module defines the following functions: .. versionchanged:: 3.5 The *etype* argument is ignored and inferred from the type of *value*. + .. versionchanged:: 3.10 + This function's behavior and signature were modified to match + :func:`print_exception`. + .. function:: format_exc(limit=None, chain=True) diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index d3487537df..2084d75b3a 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -913,8 +913,8 @@ Color control Set pencolor to the RGB color represented by *r*, *g*, and *b*. Each of *r*, *g*, and *b* must be in the range 0..colormode. - If turtleshape is a polygon, the outline of that polygon is drawn with the - newly set pencolor. + If turtleshape is a polygon, the outline of that polygon is drawn with the + newly set pencolor. .. doctest:: :skipif: _tkinter is None @@ -962,8 +962,8 @@ Color control Set fillcolor to the RGB color represented by *r*, *g*, and *b*. Each of *r*, *g*, and *b* must be in the range 0..colormode. - If turtleshape is a polygon, the interior of that polygon is drawn - with the newly set fillcolor. + If turtleshape is a polygon, the interior of that polygon is drawn + with the newly set fillcolor. .. doctest:: :skipif: _tkinter is None @@ -1001,8 +1001,8 @@ Color control Equivalent to ``pencolor(colorstring1)`` and ``fillcolor(colorstring2)`` and analogously if the other input format is used. - If turtleshape is a polygon, outline and interior of that polygon is drawn - with the newly set colors. + If turtleshape is a polygon, outline and interior of that polygon is drawn + with the newly set colors. .. doctest:: :skipif: _tkinter is None diff --git a/Doc/library/types.rst b/Doc/library/types.rst index ad40a9fbf3..8e05f8408e 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -280,6 +280,10 @@ Standard names are defined for the following types: .. versionadded:: 3.9 + .. versionchanged:: 3.9.2 + This type can now be subclassed. + + .. data:: Union The type of :ref:`union type expressions<types-union>`. @@ -409,7 +413,9 @@ Additional Utility Classes and Functions return "{}({})".format(type(self).__name__, ", ".join(items)) def __eq__(self, other): - return self.__dict__ == other.__dict__ + if isinstance(self, SimpleNamespace) and isinstance(other, SimpleNamespace): + return self.__dict__ == other.__dict__ + return NotImplemented ``SimpleNamespace`` may be useful as a replacement for ``class NS: pass``. However, for a structured record type use :func:`~collections.namedtuple` diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 9a993c0732..b07bb8943d 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -641,7 +641,7 @@ These can be used as types in annotations using ``[]``, each having a unique syn :ref:`type variables <generics>`, and unions of any of these types. For example:: - def new_non_team_user(user_class: Type[Union[BaseUser, ProUser]]): ... + def new_non_team_user(user_class: Type[Union[BasicUser, ProUser]]): ... ``Type[Any]`` is equivalent to ``Type`` which in turn is equivalent to ``type``, which is the root of Python's metaclass hierarchy. @@ -674,6 +674,12 @@ These can be used as types in annotations using ``[]``, each having a unique syn .. versionadded:: 3.8 + .. versionchanged:: 3.9.1 + ``Literal`` now de-duplicates parameters. Equality comparison of + ``Literal`` objects are no longer order dependent. ``Literal`` objects + will now raise a :exc:`TypeError` exception during equality comparisons + if one of their parameters are not :term:`immutable`. + .. data:: ClassVar Special type construct to mark class variables. @@ -1706,6 +1712,9 @@ Introspection helpers For a typing object of the form ``X[Y, Z, ...]`` these functions return ``X`` and ``(Y, Z, ...)``. If ``X`` is a generic alias for a builtin or :mod:`collections` class, it gets normalized to the original class. + If ``X`` is a :class:`Union` or :class:`Literal` contained in another + generic type, the order of ``(Y, Z, ...)`` may be different from the order + of the original arguments ``[Y, Z, ...]`` due to type caching. For unsupported objects return ``None`` and ``()`` correspondingly. Examples:: diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index c5360f91f5..f795a2e8c1 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -262,9 +262,10 @@ the *new_callable* argument to :func:`patch`. this is a new Mock (created on first access). See the :attr:`return_value` attribute. - * *unsafe*: By default if any attribute starts with *assert* or - *assret* will raise an :exc:`AttributeError`. Passing ``unsafe=True`` - will allow access to these attributes. + * *unsafe*: By default, accessing any attribute with name starting with + *assert*, *assret*, *asert*, *aseert* or *assrt* will raise an + :exc:`AttributeError`. Passing ``unsafe=True`` will allow access to + these attributes. .. versionadded:: 3.5 diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 51e10119d3..0a0993518e 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -897,8 +897,7 @@ Test cases .. method:: assertIs(first, second, msg=None) assertIsNot(first, second, msg=None) - Test that *first* and *second* evaluate (or don't evaluate) to the - same object. + Test that *first* and *second* are (or are not) the same object. .. versionadded:: 3.1 diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index b37f230feb..b4435a62ad 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -109,6 +109,11 @@ The :mod:`urllib.request` module defines the following functions: .. versionchanged:: 3.4.3 *context* was added. + .. versionchanged:: 3.10 + HTTPS connection now send an ALPN extension with protocol indicator + ``http/1.1`` when no *context* is given. Custom *context* should set + ALPN protocols with :meth:`~ssl.SSLContext.set_alpn_protocol`. + .. deprecated:: 3.6 *cafile*, *capath* and *cadefault* are deprecated in favor of *context*. diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index a481a3509d..9c1743cad2 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -491,7 +491,7 @@ Available Functions Available Context Managers -------------------------- -.. class:: catch_warnings(\*, record=False, module=None) +.. class:: catch_warnings(*, record=False, module=None) A context manager that copies and, upon exit, restores the warnings filter and the :func:`showwarning` function. diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index dccb7db27e..487856a3ac 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -791,7 +791,7 @@ integer handle, and also disconnect the Windows handle from the handle object. .. method:: PyHKEY.__enter__() - PyHKEY.__exit__(\*exc_info) + PyHKEY.__exit__(*exc_info) The HKEY object implements :meth:`~object.__enter__` and :meth:`~object.__exit__` and thus supports the context protocol for the diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index 2c78cd9392..e1cc967942 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -132,7 +132,7 @@ module documentation. This section lists the differences between the API and ... # Work with dom. -.. method:: Node.writexml(writer, indent="", addindent="", newl="", +.. method:: Node.writexml(writer, indent="", addindent="", newl="", \ encoding=None, standalone=None) Write XML to the writer object. The writer receives texts but not bytes as input, @@ -174,7 +174,7 @@ module documentation. This section lists the differences between the API and The :meth:`toxml` method now preserves the attribute order specified by the user. -.. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None, +.. method:: Node.toprettyxml(indent="\t", newl="\n", encoding=None, \ standalone=None) Return a pretty-printed version of the document. *indent* specifies the diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index f4bccf6609..87f4ee347d 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -455,6 +455,12 @@ Supported XPath syntax | | has the given value. The value cannot contain | | | quotes. | +-----------------------+------------------------------------------------------+ +| ``[@attrib!='value']``| Selects all elements for which the given attribute | +| | does not have the given value. The value cannot | +| | contain quotes. | +| | | +| | .. versionadded:: 3.10 | ++-----------------------+------------------------------------------------------+ | ``[tag]`` | Selects all elements that have a child named | | | ``tag``. Only immediate children are supported. | +-----------------------+------------------------------------------------------+ @@ -463,10 +469,22 @@ Supported XPath syntax | | | | | .. versionadded:: 3.7 | +-----------------------+------------------------------------------------------+ +| ``[.!='text']`` | Selects all elements whose complete text content, | +| | including descendants, does not equal the given | +| | ``text``. | +| | | +| | .. versionadded:: 3.10 | ++-----------------------+------------------------------------------------------+ | ``[tag='text']`` | Selects all elements that have a child named | | | ``tag`` whose complete text content, including | | | descendants, equals the given ``text``. | +-----------------------+------------------------------------------------------+ +| ``[tag!='text']`` | Selects all elements that have a child named | +| | ``tag`` whose complete text content, including | +| | descendants, does not equal the given ``text``. | +| | | +| | .. versionadded:: 3.10 | ++-----------------------+------------------------------------------------------+ | ``[position]`` | Selects all elements that are located at the given | | | position. The position can be either an integer | | | (1 is the first position), the expression ``last()`` | diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 7126d8bd70..3db55e646c 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -483,7 +483,7 @@ Path Objects Path objects expose the following features of :mod:`pathlib.Path` objects: -Path objects are traversable using the ``/`` operator. +Path objects are traversable using the ``/`` operator or ``joinpath``. .. attribute:: Path.name @@ -532,6 +532,19 @@ Path objects are traversable using the ``/`` operator. Read the current file as bytes. +.. method:: Path.joinpath(*other) + + Return a new Path object with each of the *other* arguments + joined. The following are equivalent:: + + >>> Path(...).joinpath('child').joinpath('grandchild') + >>> Path(...).joinpath('child', 'grandchild') + >>> Path(...) / 'child' / 'grandchild' + + .. versionchanged:: 3.10 + Prior to 3.10, ``joinpath`` was undocumented and accepted + exactly one parameter. + .. _pyzipfile-objects: diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 8ac3fb16bd..62e1e47e98 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -44,8 +44,9 @@ doesn't contain :file:`.pyc` files, importing may be rather slow. follows the specification in :pep:`273`, but uses an implementation written by Just van Rossum that uses the import hooks described in :pep:`302`. - :pep:`302` - New Import Hooks - The PEP to add the import hooks that help this module work. + :mod:`importlib` - The implementation of the import machinery + Package providing the relevant protocols for all importers to + implement. This module defines an exception: @@ -73,7 +74,31 @@ zipimporter Objects :exc:`ZipImportError` is raised if *archivepath* doesn't point to a valid ZIP archive. - .. method:: find_module(fullname[, path]) + .. method:: create_module(spec) + + Implementation of :meth:`importlib.abc.Loader.create_module` that returns + :const:`None` to explicitly request the default semantics. + + .. versionadded:: 3.10 + + + .. method:: exec_module(module) + + Implementation of :meth:`importlib.abc.Loader.exec_module`. + + .. versionadded:: 3.10 + + + .. method:: find_loader(fullname, path=None) + + An implementation of :meth:`importlib.abc.PathEntryFinder.find_loader`. + + .. deprecated:: 3.10 + + Use :meth:`find_spec` instead. + + + .. method:: find_module(fullname, path=None) Search for a module specified by *fullname*. *fullname* must be the fully qualified (dotted) module name. It returns the zipimporter instance itself @@ -81,11 +106,22 @@ zipimporter Objects *path* argument is ignored---it's there for compatibility with the importer protocol. + .. deprecated:: 3.10 + + Use :meth:`find_spec` instead. + + + .. method:: find_spec(fullname, target=None) + + An implementation of :meth:`importlib.abc.PathEntryFinder.find_spec`. + + .. versionadded:: 3.10 + .. method:: get_code(fullname) Return the code object for the specified module. Raise - :exc:`ZipImportError` if the module couldn't be found. + :exc:`ZipImportError` if the module couldn't be imported. .. method:: get_data(pathname) @@ -101,7 +137,7 @@ zipimporter Objects Return the value ``__file__`` would be set to if the specified module was imported. Raise :exc:`ZipImportError` if the module couldn't be - found. + imported. .. versionadded:: 3.1 @@ -123,9 +159,12 @@ zipimporter Objects .. method:: load_module(fullname) Load the module specified by *fullname*. *fullname* must be the fully - qualified (dotted) module name. It returns the imported module, or raises - :exc:`ZipImportError` if it wasn't found. + qualified (dotted) module name. Returns the imported module on success, + raises :exc:`ZipImportError` on failure. + + .. deprecated:: 3.10 + Use :meth:`exec_module` instead. .. attribute:: archive diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 62986cb151..5bba3eea6f 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -254,7 +254,8 @@ present, must be last; it matches any exception. For an except clause with an expression, that expression is evaluated, and the clause matches the exception if the resulting object is "compatible" with the exception. An object is compatible with an exception if it is the class or a base class of the exception -object or a tuple containing an item compatible with the exception. +object, or a tuple containing an item that is the class or a base class of +the exception object. If no except clause matches the exception, the search for an exception handler continues in the surrounding code and on the invocation stack. [#]_ @@ -301,9 +302,27 @@ Before an except clause's suite is executed, details about the exception are stored in the :mod:`sys` module and can be accessed via :func:`sys.exc_info`. :func:`sys.exc_info` returns a 3-tuple consisting of the exception class, the exception instance and a traceback object (see section :ref:`types`) identifying -the point in the program where the exception occurred. :func:`sys.exc_info` -values are restored to their previous values (before the call) when returning -from a function that handled an exception. +the point in the program where the exception occurred. The details about the +exception accessed via :func:`sys.exc_info` are restored to their previous values +when leaving an exception handler:: + + >>> print(sys.exc_info()) + (None, None, None) + >>> try: + ... raise TypeError + ... except: + ... print(sys.exc_info()) + ... try: + ... raise ValueError + ... except: + ... print(sys.exc_info()) + ... print(sys.exc_info()) + ... + (<class 'TypeError'>, TypeError(), <traceback object at 0x10efad080>) + (<class 'ValueError'>, ValueError(), <traceback object at 0x10efad040>) + (<class 'TypeError'>, TypeError(), <traceback object at 0x10efad080>) + >>> print(sys.exc_info()) + (None, None, None) .. index:: keyword: else @@ -796,12 +815,12 @@ The :keyword:`!async for` statement .. productionlist:: python-grammar async_for_stmt: "async" `for_stmt` -An :term:`asynchronous iterable` is able to call asynchronous code in its -*iter* implementation, and :term:`asynchronous iterator` can call asynchronous -code in its *next* method. +An :term:`asynchronous iterable` provides an ``__aiter__`` method that directly +returns an :term:`asynchronous iterator`, which can call asynchronous code in +its ``__anext__`` method. The ``async for`` statement allows convenient iteration over asynchronous -iterators. +iterables. The following code:: diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index c882301ad3..62f8529536 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -217,7 +217,6 @@ Ellipsis There are two types of integers: Integers (:class:`int`) - These represent numbers in an unlimited range, subject to available (virtual) memory only. For the purpose of shift and mask operations, a binary representation is assumed, and negative numbers are represented in a variant of diff --git a/Doc/reference/grammar.rst b/Doc/reference/grammar.rst index acf83765b5..59b4500583 100644 --- a/Doc/reference/grammar.rst +++ b/Doc/reference/grammar.rst @@ -13,7 +13,8 @@ In particular, ``&`` followed by a symbol, token or parenthesized group indicates a positive lookahead (i.e., is required to match but not consumed), while ``!`` indicates a negative lookahead (i.e., is required _not_ to match). We use the ``|`` separator to mean PEG's -"ordered choice" (written as ``/`` in traditional PEG grammars). +"ordered choice" (written as ``/`` in traditional PEG grammars). See +:pep:`617` for more details on the grammar's syntax. .. literalinclude:: ../../Grammar/python.gram :language: peg diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index f8ab2e918c..2c6c901402 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -950,7 +950,7 @@ Names listed in a :keyword:`global` statement must not be used in the same code block textually preceding that :keyword:`!global` statement. Names listed in a :keyword:`global` statement must not be defined as formal -parameters or in a :keyword:`for` loop control target, :keyword:`class` +parameters, or as targets in :keyword:`with` statements or :keyword:`except` clauses, or in a :keyword:`for` target list, :keyword:`class` definition, function definition, :keyword:`import` statement, or variable annotation. diff --git a/Doc/tools/extensions/c_annotations.py b/Doc/tools/extensions/c_annotations.py index fa8244a8fd..76c9d920cb 100644 --- a/Doc/tools/extensions/c_annotations.py +++ b/Doc/tools/extensions/c_annotations.py @@ -79,9 +79,9 @@ class Annotations(dict): classes=['stableabi'])) if par['objtype'] != 'function': continue - if not par[0].has_key('names') or not par[0]['names']: + if not par[0].has_key('ids') or not par[0]['ids']: continue - name = par[0]['names'][0] + name = par[0]['ids'][0] if name.startswith("c."): name = name[2:] entry = self.get(name) diff --git a/Doc/tools/extensions/glossary_search.py b/Doc/tools/extensions/glossary_search.py new file mode 100644 index 0000000000..34d227d670 --- /dev/null +++ b/Doc/tools/extensions/glossary_search.py @@ -0,0 +1,57 @@ +# -*- coding: utf-8 -*- +""" + glossary_search.py + ~~~~~~~~~~~~~~~~ + + Feature search results for glossary items prominently. + + :license: Python license. +""" +from os import path +from sphinx.addnodes import glossary +from sphinx.util import logging +from docutils.nodes import definition_list_item +import json + + +logger = logging.getLogger(__name__) + + +def process_glossary_nodes(app, doctree, fromdocname): + if app.builder.format != 'html': + return + + terms = {} + + for node in doctree.traverse(glossary): + for glossary_item in node.traverse(definition_list_item): + term = glossary_item[0].astext().lower() + definition = glossary_item[1] + + rendered = app.builder.render_partial(definition) + terms[term] = { + 'title': glossary_item[0].astext(), + 'body': rendered['html_body'] + } + + if hasattr(app.env, 'glossary_terms'): + app.env.glossary_terms.update(terms) + else: + app.env.glossary_terms = terms + +def on_build_finish(app, exc): + if not hasattr(app.env, 'glossary_terms'): + return + if not app.env.glossary_terms: + return + + logger.info('Writing glossary.json', color='green') + with open(path.join(app.outdir, '_static', 'glossary.json'), 'w') as f: + json.dump(app.env.glossary_terms, f) + + +def setup(app): + app.connect('doctree-resolved', process_glossary_nodes) + app.connect('build-finished', on_build_finish) + + return {'version': '0.1', 'parallel_read_safe': True} diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 80fbd96d56..28994399e2 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -394,7 +394,12 @@ class DeprecatedRemoved(Directive): translatable=False) node.append(para) env = self.state.document.settings.env - env.get_domain('changeset').note_changeset(node) + # deprecated pre-Sphinx-2 method + if hasattr(env, 'note_versionchange'): + env.note_versionchange('deprecated', version[0], node, self.lineno) + # new method + else: + env.get_domain('changeset').note_changeset(node) return [node] + messages diff --git a/Doc/tools/rstlint.py b/Doc/tools/rstlint.py index a3024d6734..cbcb8eb801 100755 --- a/Doc/tools/rstlint.py +++ b/Doc/tools/rstlint.py @@ -13,6 +13,7 @@ import os import re import sys import getopt +from string import ascii_letters from os.path import join, splitext, abspath, exists from collections import defaultdict @@ -128,6 +129,81 @@ def check_leaked_markup(fn, lines): yield lno+1, 'possibly leaked markup: %r' % line +def hide_literal_blocks(lines): + """Tool to remove literal blocks from given lines. + + It yields empty lines in place of blocks, so line numbers are + still meaningful. + """ + in_block = False + for line in lines: + if line.endswith("::\n"): + in_block = True + elif in_block: + if line == "\n" or line.startswith(" "): + line = "\n" + else: + in_block = False + yield line + + +def type_of_explicit_markup(line): + if re.match(fr'\.\. {all_directives}::', line): + return 'directive' + if re.match(r'\.\. \[[0-9]+\] ', line): + return 'footnote' + if re.match(r'\.\. \[[^\]]+\] ', line): + return 'citation' + if re.match(r'\.\. _.*[^_]: ', line): + return 'target' + if re.match(r'\.\. \|[^\|]*\| ', line): + return 'substitution_definition' + return 'comment' + + +def hide_comments(lines): + """Tool to remove comments from given lines. + + It yields empty lines in place of comments, so line numbers are + still meaningfull. + """ + in_multiline_comment = False + for line in lines: + if line == "..\n": + in_multiline_comment = True + elif in_multiline_comment: + if line == "\n" or line.startswith(" "): + line = "\n" + else: + in_multiline_comment = False + if line.startswith(".. ") and type_of_explicit_markup(line) == 'comment': + line = "\n" + yield line + + + +@checker(".rst", severity=2) +def check_missing_surrogate_space_on_plural(fn, lines): + r"""Check for missing 'backslash-space' between a code sample a letter. + + Good: ``Point``\ s + Bad: ``Point``s + """ + in_code_sample = False + check_next_one = False + for lno, line in enumerate(hide_comments(hide_literal_blocks(lines))): + tokens = line.split("``") + for token_no, token in enumerate(tokens): + if check_next_one: + if token[0] in ascii_letters: + yield lno + 1, f"Missing backslash-space between code sample and {token!r}." + check_next_one = False + if token_no == len(tokens) - 1: + continue + if in_code_sample: + check_next_one = True + in_code_sample = not in_code_sample + def main(argv): usage = '''\ Usage: %s [-v] [-f] [-s sev] [-i path]* [path] diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js deleted file mode 100644 index 1a1c7d0fa5..0000000000 --- a/Doc/tools/static/switchers.js +++ /dev/null @@ -1,156 +0,0 @@ -(function() { - 'use strict'; - - // Parses versions in URL segments like: - // "3", "dev", "release/2.7" or "3.6rc2" - var version_regexs = [ - '(?:\\d)', - '(?:\\d\\.\\d[\\w\\d\\.]*)', - '(?:dev)', - '(?:release/\\d.\\d[\\x\\d\\.]*)']; - - var all_versions = { - '3.10': 'dev (3.10)', - '3.9': 'pre (3.9)', - '3.8': '3.8', - '3.7': '3.7', - '3.6': '3.6', - '2.7': '2.7', - }; - - var all_languages = { - 'en': 'English', - 'fr': 'French', - 'ja': 'Japanese', - 'ko': 'Korean', - 'pt-br': 'Brazilian Portuguese', - 'zh-cn': 'Simplified Chinese', - }; - - function build_version_select(current_version, current_release) { - var buf = ['<select>']; - - $.each(all_versions, function(version, title) { - buf.push('<option value="' + version + '"'); - if (version == current_version) - buf.push(' selected="selected">' + current_release + '</option>'); - else - buf.push('>' + title + '</option>'); - }); - - buf.push('</select>'); - return buf.join(''); - } - - function build_language_select(current_language) { - var buf = ['<select>']; - - $.each(all_languages, function(language, title) { - if (language == current_language) - buf.push('<option value="' + language + '" selected="selected">' + - all_languages[current_language] + '</option>'); - else - buf.push('<option value="' + language + '">' + title + '</option>'); - }); - if (!(current_language in all_languages)) { - // In case we're browsing a language that is not yet in all_languages. - buf.push('<option value="' + current_language + '" selected="selected">' + - current_language + '</option>'); - all_languages[current_language] = current_language; - } - buf.push('</select>'); - return buf.join(''); - } - - function navigate_to_first_existing(urls) { - // Navigate to the first existing URL in urls. - var url = urls.shift(); - if (urls.length == 0) { - window.location.href = url; - return; - } - $.ajax({ - url: url, - success: function() { - window.location.href = url; - }, - error: function() { - navigate_to_first_existing(urls); - } - }); - } - - function on_version_switch() { - var selected_version = $(this).children('option:selected').attr('value') + '/'; - var url = window.location.href; - var current_language = language_segment_from_url(url); - var current_version = version_segment_in_url(url); - var new_url = url.replace('.org/' + current_language + current_version, - '.org/' + current_language + selected_version); - if (new_url != url) { - navigate_to_first_existing([ - new_url, - url.replace('.org/' + current_language + current_version, - '.org/' + selected_version), - 'https://docs.python.org/' + current_language + selected_version, - 'https://docs.python.org/' + selected_version, - 'https://docs.python.org/' - ]); - } - } - - function on_language_switch() { - var selected_language = $(this).children('option:selected').attr('value') + '/'; - var url = window.location.href; - var current_language = language_segment_from_url(url); - var current_version = version_segment_in_url(url); - if (selected_language == 'en/') // Special 'default' case for english. - selected_language = ''; - var new_url = url.replace('.org/' + current_language + current_version, - '.org/' + selected_language + current_version); - if (new_url != url) { - navigate_to_first_existing([ - new_url, - 'https://docs.python.org/' - ]); - } - } - - // Returns the path segment of the language as a string, like 'fr/' - // or '' if not found. - function language_segment_from_url(url) { - var language_regexp = '\.org/([a-z]{2}(?:-[a-z]{2})?/)'; - var match = url.match(language_regexp); - if (match !== null) - return match[1]; - return ''; - } - - // Returns the path segment of the version as a string, like '3.6/' - // or '' if not found. - function version_segment_in_url(url) { - var language_segment = '(?:[a-z]{2}(?:-[a-z]{2})?/)'; - var version_segment = '(?:(?:' + version_regexs.join('|') + ')/)'; - var version_regexp = '\\.org/' + language_segment + '?(' + version_segment + ')'; - var match = url.match(version_regexp); - if (match !== null) - return match[1]; - return '' - } - - $(document).ready(function() { - var release = DOCUMENTATION_OPTIONS.VERSION; - var language_segment = language_segment_from_url(window.location.href); - var current_language = language_segment.replace(/\/+$/g, '') || 'en'; - var version = release.substr(0, 3); - var version_select = build_version_select(version, release); - - $('.version_switcher_placeholder').html(version_select); - $('.version_switcher_placeholder select').bind('change', on_version_switch); - - var language_select = build_language_select(current_language); - - $('.language_switcher_placeholder').html(language_select); - $('.language_switcher_placeholder select').bind('change', on_language_switch); - }); -})(); diff --git a/Doc/tools/templates/dummy.html b/Doc/tools/templates/dummy.html index 68ae3ad148..3438b44377 100644 --- a/Doc/tools/templates/dummy.html +++ b/Doc/tools/templates/dummy.html @@ -6,3 +6,12 @@ In extensions/pyspecific.py: {% trans %}CPython implementation detail:{% endtrans %} {% trans %}Deprecated since version {deprecated}, will be removed in version {removed}{% endtrans %} {% trans %}Deprecated since version {deprecated}, removed in version {removed}{% endtrans %} + + +In docsbuild-scripts, when rewriting indexsidebar.html with actual versions: + +{% trans %}in development{% endtrans %} +{% trans %}pre-release{% endtrans %} +{% trans %}stable{% endtrans %} +{% trans %}security-fixes{% endtrans %} +{% trans %}EOL{% endtrans %} diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html index 1c1cb5484a..f7bf6d8e49 100644 --- a/Doc/tools/templates/indexsidebar.html +++ b/Doc/tools/templates/indexsidebar.html @@ -2,12 +2,8 @@ <p><a href="{{ pathto('download') }}">{% trans %}Download these documents{% endtrans %}</a></p> <h3>{% trans %}Docs by version{% endtrans %}</h3> <ul> - <li><a href="https://docs.python.org/3.10/">{% trans %}Python 3.10 (in development){% endtrans %}</a></li> - <li><a href="https://docs.python.org/3.9/">{% trans %}Python 3.9 (pre-release){% endtrans %}</a></li> - <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (stable){% endtrans %}</a></li> - <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (stable){% endtrans %}</a></li> - <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (security-fixes){% endtrans %}</a></li> - <li><a href="https://docs.python.org/2.7/">{% trans %}Python 2.7 (EOL){% endtrans %}</a></li> + <li><a href="https://docs.python.org/">{% trans %}Stable{% endtrans %}</a></li> + <li><a href="https://docs.python.org/dev/">{% trans %}In development{% endtrans %}</a></li> <li><a href="https://www.python.org/doc/versions/">{% trans %}All versions{% endtrans %}</a></li> </ul> diff --git a/Doc/tools/templates/layout.html b/Doc/tools/templates/layout.html index 17592d74a4..98ccf42248 100644 --- a/Doc/tools/templates/layout.html +++ b/Doc/tools/templates/layout.html @@ -12,22 +12,14 @@ {% block rootrellink %} {{ super() }} - <li> - {%- if switchers is defined %} - <span class="language_switcher_placeholder">{{ language or 'en' }}</span> - <span class="version_switcher_placeholder">{{ release }}</span> - <a href="{{ pathto('index') }}">{% trans %}Documentation {% endtrans %}</a>{{ reldelim1 }} - {%- else %} + <li id="cpython-language-and-version"> <a href="{{ pathto('index') }}">{{ shorttitle }}</a>{{ reldelim1 }} - {%- endif %} </li> {% endblock %} {% block extrahead %} <link rel="canonical" href="https://docs.python.org/3/{{pagename}}.html" /> {% if builder != "htmlhelp" %} - {% if switchers is defined and not embedded %} - <script type="text/javascript" src="{{ pathto('_static/switchers.js', 1) }}"></script>{% endif %} {% if pagename == 'whatsnew/changelog' and not embedded %} <script type="text/javascript" src="{{ pathto('_static/changelog_search.js', 1) }}"></script>{% endif %} {% endif %} diff --git a/Doc/tools/templates/search.html b/Doc/tools/templates/search.html new file mode 100644 index 0000000000..cf20c2e1d4 --- /dev/null +++ b/Doc/tools/templates/search.html @@ -0,0 +1,48 @@ +{% extends "!search.html" %} +{% block extrahead %} + {{ super() }} + <script type="text/javascript"> + var GLOSSARY_PAGE = 'glossary.html'; + + jQuery(function() { + $.getJSON("_static/glossary.json", function(glossary) { + var RESULT_TEMPLATE = '<div style="display: none" class="admonition seealso" id="glossary-result">' + + ' <p class="topic-title">' + + ' <a class="glossary-title" href="#"></a>' + + ' </p>' + + ' <div class="glossary-body"></div>' + + '</div>'; + $("#search-results").prepend(RESULT_TEMPLATE); + + var params = $.getQueryParameters(); + if (params.q) { + var search_param = params.q[0].toLowerCase(); + var glossary_item = glossary[search_param]; + if (glossary_item) { + var resultDiv = $("#glossary-result"); + + // set up the title text with a link to the glossary page + resultDiv.find(".glossary-title").text('Glossary: ' + glossary_item.title); + var link_target = search_param.replace(/ /g, '-'); + resultDiv.find(".glossary-title").attr( + 'href', GLOSSARY_PAGE + '#term-' + link_target + ); + + // rewrite any anchor links (to other glossary terms) + // to have a full reference to the glossary page + var body = $(glossary_item.body).children(); + body.find("a[href^='#']").each(function() { + var current_url = $(this).attr('href'); + $(this).attr('href', GLOSSARY_PAGE + current_url); + }); + resultDiv.find(".glossary-body").html(body); + + resultDiv.show(); + } else { + $("#glossary-result").hide(''); + } + } + }); + }); + </script> +{% endblock %}
\ No newline at end of file diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index ff4c797f66..5c6b65f05e 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -78,7 +78,7 @@ objects: Return the number of times *x* appears in the list. -.. method:: list.sort(key=None, reverse=False) +.. method:: list.sort(*, key=None, reverse=False) :noindex: Sort the items of the list in place (the arguments can be used for sort diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst index 0ce96466e8..4a25861a05 100644 --- a/Doc/tutorial/errors.rst +++ b/Doc/tutorial/errors.rst @@ -273,36 +273,35 @@ Exception Chaining ================== The :keyword:`raise` statement allows an optional :keyword:`from` which enables -chaining exceptions by setting the ``__cause__`` attribute of the raised -exception. For example:: +chaining exceptions. For example:: - raise RuntimeError from OSError + # exc must be exception instance or None. + raise RuntimeError from exc This can be useful when you are transforming exceptions. For example:: >>> def func(): - ... raise IOError + ... raise ConnectionError ... >>> try: ... func() - ... except IOError as exc: + ... except ConnectionError as exc: ... raise RuntimeError('Failed to open database') from exc ... Traceback (most recent call last): File "<stdin>", line 2, in <module> File "<stdin>", line 2, in func - OSError + ConnectionError <BLANKLINE> The above exception was the direct cause of the following exception: <BLANKLINE> Traceback (most recent call last): File "<stdin>", line 4, in <module> - RuntimeError + RuntimeError: Failed to open database -The expression following the :keyword:`from` must be either an exception or -``None``. Exception chaining happens automatically when an exception is raised -inside an exception handler or :keyword:`finally` section. Exception chaining -can be disabled by using ``from None`` idiom: +Exception chaining happens automatically when an exception is raised inside an +:keyword:`except` or :keyword:`finally` section. This can be +disabled by using ``from None`` idiom: >>> try: ... open('database.sqlite') @@ -313,6 +312,8 @@ can be disabled by using ``from None`` idiom: File "<stdin>", line 4, in <module> RuntimeError +For more information about chaining mechanics, see :ref:`bltin-exceptions`. + .. _tut-userexceptions: diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index 366a532e81..4e27cff83c 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -329,11 +329,16 @@ equivalent :keyword:`try`\ -\ :keyword:`finally` blocks:: If you're not using the :keyword:`with` keyword, then you should call ``f.close()`` to close the file and immediately free up any system -resources used by it. If you don't explicitly close a file, Python's -garbage collector will eventually destroy the object and close the -open file for you, but the file may stay open for a while. Another -risk is that different Python implementations will do this clean-up at -different times. +resources used by it. + +.. warning:: + Calling ``f.write()`` without using the :keyword:`!with` keyword or calling + ``f.close()`` **might** result in the arguments + of ``f.write()`` not being completely written to the disk, even if the + program exits successfully. + +.. + See also https://bugs.python.org/issue17852 After a file object is closed, either by a :keyword:`with` statement or by calling ``f.close()``, attempts to use the file object will diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 78c1e03f74..265c07c709 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -23,8 +23,8 @@ available for application-local distributions. As specified in :pep:`11`, a Python release only supports a Windows platform while Microsoft considers the platform under extended support. This means that -Python |version| supports Windows Vista and newer. If you require Windows XP -support then please install Python 3.4. +Python |version| supports Windows 8.1 and newer. If you require Windows 7 +support, please install Python 3.8. There are a number of different installers available for Windows, each with certain benefits and downsides. diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index 60dee0c6bd..b5fb1e9a62 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -120,8 +120,8 @@ See :pep:`613` for more details. (Contributed by Mikhail Golubev in :issue:`41923`.) -PEP604: New Type Union Operator -------------------------------- +PEP 604: New Type Union Operator +-------------------------------- A new type union operator was introduced which enables the syntax ``X | Y``. This provides a cleaner way of expressing 'either type X or type Y' instead of @@ -154,6 +154,9 @@ Other Language Changes :meth:`~object.__index__` method). (Contributed by Serhiy Storchaka in :issue:`37999`.) +* Assignment expressions can now be used unparenthesized within set literals + and set comprehensions, as well as in sequence indexes (but not slices). + New Modules =========== @@ -176,6 +179,31 @@ codecs Add a :func:`codecs.unregister` function to unregister a codec search function. (Contributed by Hai Shi in :issue:`41842`.) +collections.abc +--------------- + +The ``__args__`` of the :ref:`parameterized generic <types-genericalias>` for +:class:`collections.abc.Callable` are now consistent with :data:`typing.Callable`. +:class:`collections.abc.Callable` generic now flattens type parameters, similar +to what :data:`typing.Callable` currently does. This means that +``collections.abc.Callable[[int, str], str]`` will have ``__args__`` of +``(int, str, str)``; previously this was ``([int, str], str)``. To allow this +change, :class:`types.GenericAlias` can now be subclassed, and a subclass will +be returned when subscripting the :class:`collections.abc.Callable` type. Note +that a :exc:`TypeError` may be raised for invalid forms of parameterizing +:class:`collections.abc.Callable` which may have passed silently in Python 3.9. +(Contributed by Ken Jin in :issue:`42195`.) + +contextlib +---------- + +Add a :func:`contextlib.aclosing` context manager to safely close async generators +and objects representing asynchronously released resources. +(Contributed by Joongi Kim and John Belmonte in :issue:`41229`.) + +Add asynchronous context manager support to :func:`contextlib.nullcontext`. +(Contributed by Tom Gringauz in :issue:`41543`.) + curses ------ @@ -186,6 +214,12 @@ by :func:`curses.color_content`, :func:`curses.init_color`, support is provided by the underlying ncurses library. (Contributed by Jeffrey Kintscher and Hans Petter Jansson in :issue:`36982`.) +doctest +------- + +When a module does not define ``__loader__``, fall back to ``__spec__.loader``. +(Contributed by Brett Cannon in :issue:`42133`.) + encodings --------- :func:`encodings.normalize_encoding` now ignores non-ASCII characters. @@ -198,12 +232,56 @@ Added the *root_dir* and *dir_fd* parameters in :func:`~glob.glob` and :func:`~glob.iglob` which allow to specify the root directory for searching. (Contributed by Serhiy Storchaka in :issue:`38144`.) +inspect +------- + +When a module does not define ``__loader__``, fall back to ``__spec__.loader``. +(Contributed by Brett Cannon in :issue:`42133`.) + +Added *globalns* and *localns* parameters in :func:`~inspect.signature` and +:meth:`inspect.Signature.from_callable` to retrieve the annotations in given +local and global namespaces. +(Contributed by Batuhan Taskaya in :issue:`41960`.) + +linecache +--------- + +When a module does not define ``__loader__``, fall back to ``__spec__.loader``. +(Contributed by Brett Cannon in :issue:`42133`.) + os -- Added :func:`os.cpu_count()` support for VxWorks RTOS. (Contributed by Peixing Xin in :issue:`41440`.) +Added a new function :func:`os.eventfd` and related helpers to wrap the +``eventfd2`` syscall on Linux. +(Contributed by Christian Heimes in :issue:`41001`.) + +Added :func:`os.splice()` that allows to move data between two file +descriptors without copying between kernel address space and user +address space, where one of the file descriptors must refer to a +pipe. (Contributed by Pablo Galindo in :issue:`41625`.) + +pathlib +------- + +Added slice support to :attr:`PurePath.parents <pathlib.PurePath.parents>`. +(Contributed by Joshua Cannon in :issue:`35498`) + +Added negative indexing support to :attr:`PurePath.parents +<pathlib.PurePath.parents>`. +(Contributed by Yaroslav Pankovych in :issue:`21041`) + +platform +-------- + +Added :func:`platform.freedesktop_os_release()` to retrieve operation system +identification from `freedesktop.org os-release +<https://www.freedesktop.org/software/systemd/man/os-release.html>`_ standard file. +(Contributed by Christian Heimes in :issue:`28468`) + py_compile ---------- @@ -217,6 +295,18 @@ The :mod:`shelve` module now uses :data:`pickle.DEFAULT_PROTOCOL` by default instead of :mod:`pickle` protocol ``3`` when creating shelves. (Contributed by Zackery Spytz in :issue:`34204`.) +site +---- + +When a module does not define ``__loader__``, fall back to ``__spec__.loader``. +(Contributed by Brett Cannon in :issue:`42133`.) + +socket +------ + +The exception :exc:`socket.timeout` is now an alias of :exc:`TimeoutError`. +(Contributed by Christian Heimes in :issue:`42413`.) + sys --- @@ -224,6 +314,28 @@ Add :data:`sys.orig_argv` attribute: the list of the original command line arguments passed to the Python executable. (Contributed by Victor Stinner in :issue:`23427`.) +threading +--------- + +Added :func:`threading.gettrace` and :func:`threading.getprofile` to +retrieve the functions set by :func:`threading.settrace` and +:func:`threading.setprofile` respectively. +(Contributed by Mario Corchero in :issue:`42251`.) + +Add :data:`threading.__excepthook__` to allow retrieving the original value +of :func:`threading.excepthook` in case it is set to a broken or a different +value. +(Contributed by Mario Corchero in :issue:`42308`.) + +traceback +--------- + +The :func:`~traceback.format_exception`, +:func:`~traceback.format_exception_only`, and +:func:`~traceback.print_exception` functions can now take an exception object +as a positional-only argument. +(Contributed by Zackery Spytz and Matthias Bussonnier in :issue:`26389`.) + types ----- @@ -232,6 +344,32 @@ and :data:`types.NotImplementedType` classes, providing a new set of types readily interpretable by type checkers. (Contributed by Bas van Beek in :issue:`41810`.) +typing +------ + +The behavior of :class:`typing.Literal` was changed to conform with :pep:`586` +and to match the behavior of static type checkers specified in the PEP. + +1. ``Literal`` now de-duplicates parameters. +2. Equality comparisons between ``Literal`` objects are now order independent. +3. ``Literal`` comparisons now respects types. For example, + ``Literal[0] == Literal[False]`` previously evaluated to ``True``. It is + now ``False``. To support this change, the internally used type cache now + supports differentiating types. +4. ``Literal`` objects will now raise a :exc:`TypeError` exception during + equality comparisons if one of their parameters are not :term:`immutable`. + Note that declaring ``Literal`` with mutable parameters will not throw + an error:: + + >>> from typing import Literal + >>> Literal[{0}] + >>> Literal[{0}] == Literal[{False}] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: unhashable type: 'set' + +(Contributed by Yurii Karabas in :issue:`42345`.) + unittest -------- @@ -246,6 +384,13 @@ Add a :class:`~xml.sax.handler.LexicalHandler` class to the :mod:`xml.sax.handler` module. (Contributed by Jonathan Gossage and Zackery Spytz in :issue:`35018`.) +zipimport +--------- +Add methods related to :pep:`451`: :meth:`~zipimport.zipimporter.find_spec`, +:meth:`zipimport.zipimporter.create_module`, and +:meth:`zipimport.zipimporter.exec_module`. +(Contributed by Brett Cannon in :issue:`42131`. + Optimizations ============= @@ -269,7 +414,13 @@ Optimizations with ``gcc`` by up to 30%. See `this article <https://developers.redhat.com/blog/2020/06/25/red-hat-enterprise-linux-8-2-brings-faster-python-3-8-run-speeds/>`_ for more details. (Contributed by Victor Stinner and Pablo Galindo in - :issue:`38980`) + :issue:`38980`.) + + +* Function parameters and their annotations are no longer computed at runtime, + but rather at compilation time. They are stored as a tuple of strings at the + bytecode level. It is now around 100% faster to create a function with parameter + annotations. (Contributed by Yurii Karabas and Inada Naoki in :issue:`42202`) Deprecated ========== @@ -291,6 +442,26 @@ Deprecated as appropriate to help identify code which needs updating during this transition. +* The various ``load_module()`` methods of :mod:`importlib` have been + documented as deprecated since Python 3.6, but will now also trigger + a :exc:`DeprecationWarning`. Use + :meth:`~importlib.abc.Loader.exec_module` instead. + (Contributed by Brett Cannon in :issue:`26131`.) + +* :meth:`zimport.zipimporter.load_module` has been deprecated in + preference for :meth:`~zipimport.zipimporter.exec_module`. + (Contributed by Brett Cannon in :issue:`26131`.) + +* The use of :meth:`~importlib.abc.Loader.load_module` by the import + system now triggers an :exc:`ImportWarning` as + :meth:`~importlib.abc.Loader.exec_module` is preferred. + (Contributed by Brett Cannon in :issue:`26131`.) + +* ``sqlite3.OptimizedUnicode`` has been undocumented and obsolete since Python + 3.3, when it was made an alias to :class:`str`. It is now deprecated, + scheduled for removal in Python 3.12. + (Contributed by Erlend E. Aasland in :issue:`42264`.) + Removed ======= @@ -312,6 +483,27 @@ Removed moved to the internal C API. (Contributed by Victor Stinner in :issue:`42157`.) +* Removed the ``parser`` module, which was deprecated in 3.9 due to the + switch to the new PEG parser, as well as all the C source and header files + that were only being used by the old parser, including ``node.h``, ``parser.h``, + ``graminit.h`` and ``grammar.h``. + +* Removed the Public C API functions :c:func:`PyParser_SimpleParseStringFlags`, + :c:func:`PyParser_SimpleParseStringFlagsFilename`, + :c:func:`PyParser_SimpleParseFileFlags` and :c:func:`PyNode_Compile` + that were deprecated in 3.9 due to the switch to the new PEG parser. + +* Removed the ``formatter`` module, which was deprecated in Python 3.4. + It is somewhat obsolete, little used, and not tested. It was originally + scheduled to be removed in Python 3.6, but such removals were delayed until + after Python 2.7 EOL. Existing users should copy whatever classes they use + into their code. + (Contributed by Dong-hee Na and Terry J. Reedy in :issue:`42299`.) + +* Removed the :c:func:`PyModule_GetWarningsModule` function that was useless + now due to the _warnings module was converted to a builtin module in 2.6. + (Contributed by Hai Shi in :issue:`42599`.) + Porting to Python 3.10 ====================== @@ -320,6 +512,37 @@ This section lists previously described changes and other bugfixes that may require changes to your code. +Changes in the Python API +------------------------- + +* The *etype* parameters of the :func:`~traceback.format_exception`, + :func:`~traceback.format_exception_only`, and + :func:`~traceback.print_exception` functions in the :mod:`traceback` module + have been renamed to *exc*. + (Contributed by Zackery Spytz and Matthias Bussonnier in :issue:`26389`.) + +* :mod:`atexit`: At Python exit, if a callback registered with + :func:`atexit.register` fails, its exception is now logged. Previously, only + some exceptions were logged, and the last exception was always silently + ignored. + (Contributed by Victor Stinner in :issue:`42639`.) + +* :class:`collections.abc.Callable` generic now flattens type parameters, similar + to what :data:`typing.Callable` currently does. This means that + ``collections.abc.Callable[[int, str], str]`` will have ``__args__`` of + ``(int, str, str)``; previously this was ``([int, str], str)``. Code which + accesses the arguments via :func:`typing.get_args` or ``__args__`` need to account + for this change. Furthermore, :exc:`TypeError` may be raised for invalid forms + of parameterizing :class:`collections.abc.Callable` which may have passed + silently in Python 3.9. + (Contributed by Ken Jin in :issue:`42195`.) + +CPython bytecode changes +======================== + +* The ``MAKE_FUNCTION`` instruction accepts tuple of strings as annotations + instead of dictionary. + (Contributed by Yurii Karabas and Inada Naoki in :issue:`42202`) Build Changes ============= @@ -332,6 +555,8 @@ Build Changes * :mod:`sqlite3` requires SQLite 3.7.3 or higher. (Contributed by Sergey Fedoseev and Erlend E. Aasland :issue:`40744`.) +* The :mod:`atexit` module must now always be built as a built-in module. + (Contributed by Victor Stinner in :issue:`42639`.) C API Changes @@ -366,6 +591,26 @@ New Features * Added :c:func:`PyUnicode_AsUTF8AndSize` to the limited C API. (Contributed by Alex Gaynor in :issue:`41784`.) +* Added :c:func:`PyModule_AddObjectRef` function: similar to + :c:func:`PyModule_AddObject` but don't steal a reference to the value on + success. + (Contributed by Victor Stinner in :issue:`1635741`.) + +* Added :c:func:`Py_NewRef` and :c:func:`Py_XNewRef` functions to increment the + reference count of an object and return the object. + (Contributed by Victor Stinner in :issue:`42262`.) + +* The :c:func:`PyType_FromSpecWithBases` and :c:func:`PyType_FromModuleAndSpec` + functions now accept a single class as the *bases* argument. + (Contributed by Serhiy Storchaka in :issue:`42423`.) + +* The :c:func:`PyType_FromModuleAndSpec` function now accepts NULL ``tp_doc`` + slot. + (Contributed by Hai Shi in :issue:`41832`.) + +* The :c:func:`PyType_GetSlot` function can accept static types. + (Contributed by Hai Shi and Petr Viktorin in :issue:`41073`.) + Porting to Python 3.10 ---------------------- @@ -377,17 +622,6 @@ Porting to Python 3.10 <arg-parsing>` and the :pep:`353`. (Contributed by Victor Stinner in :issue:`40943`.) -* Since :c:func:`Py_TYPE()` is changed to the inline static function, - ``Py_TYPE(obj) = new_type`` must be replaced with ``Py_SET_TYPE(obj, new_type)``: - see :c:func:`Py_SET_TYPE()` (available since Python 3.9). For backward - compatibility, this macro can be used:: - - #if PY_VERSION_HEX < 0x030900A4 - # define Py_SET_TYPE(obj, type) ((Py_TYPE(obj) = (type)), (void)0) - #endif - - (Contributed by Dong-hee Na in :issue:`39573`.) - * Since :c:func:`Py_REFCNT()` is changed to the inline static function, ``Py_REFCNT(obj) = new_refcnt`` must be replaced with ``Py_SET_REFCNT(obj, new_refcnt)``: see :c:func:`Py_SET_REFCNT()` (available since Python 3.9). For backward @@ -399,17 +633,6 @@ Porting to Python 3.10 (Contributed by Victor Stinner in :issue:`39573`.) -* Since :c:func:`Py_SIZE()` is changed to the inline static function, - ``Py_SIZE(obj) = new_size`` must be replaced with ``Py_SET_SIZE(obj, new_size)``: - see :c:func:`Py_SET_SIZE()` (available since Python 3.9). For backward - compatibility, this macro can be used:: - - #if PY_VERSION_HEX < 0x030900A4 - # define Py_SET_SIZE(obj, size) ((Py_SIZE(obj) = (size)), (void)0) - #endif - - (Contributed by Victor Stinner in :issue:`39573`.) - * Calling :c:func:`PyDict_GetItem` without :term:`GIL` held had been allowed for historical reason. It is no longer allowed. (Contributed by Victor Stinner in :issue:`40839`.) @@ -423,6 +646,21 @@ Porting to Python 3.10 ``unicodedata.ucnhash_CAPI`` has been moved to the internal C API. (Contributed by Victor Stinner in :issue:`42157`.) +* :c:func:`Py_GetPath`, :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, + :c:func:`Py_GetProgramFullPath`, :c:func:`Py_GetPythonHome` and + :c:func:`Py_GetProgramName` functions now return ``NULL`` if called before + :c:func:`Py_Initialize` (before Python is initialized). Use the new + :ref:`Python Initialization Configuration API <init-config>` to get the + :ref:`Python Path Configuration. <init-path-config>`. + (Contributed by Victor Stinner in :issue:`42260`.) + +* :c:func:`PyList_SET_ITEM`, :c:func:`PyTuple_SET_ITEM` and + :c:func:`PyCell_SET` macros can no longer be used as l-value or r-value. + For example, ``x = PyList_SET_ITEM(a, b, c)`` and + ``PyList_SET_ITEM(a, b, c) = x`` now fail with a compiler error. It prevents + bugs like ``if (PyList_SET_ITEM (a, b, c) < 0) ...`` test. + (Contributed by Zackery Spytz and Victor Stinner in :issue:`30459`.) + Deprecated ---------- @@ -471,3 +709,8 @@ Removed ``Py_END_ALLOW_RECURSION`` and the ``recursion_critical`` field of the :c:type:`PyInterpreterState` structure. (Contributed by Serhiy Storchaka in :issue:`41936`.) + +* Removed the undocumented ``PyOS_InitInterrupts()`` function. Initializing + Python already implicitly installs signal handlers: see + :c:member:`PyConfig.install_signal_handlers`. + (Contributed by Victor Stinner in :issue:`41713`.) diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index 6a9fa34156..0b4820f333 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -2113,9 +2113,6 @@ Changes in the C API (Contributed by Antoine Pitrou in :issue:`32388`.) -* The :c:func:`PyCode_New` has a new parameter in the second position (*posonlyargcount*) - to support :pep:`570`, indicating the number of positional-only arguments. - * The functions :c:func:`PyNode_AddChild` and :c:func:`PyParser_AddToken` now accept two additional ``int`` arguments *end_lineno* and *end_col_offset*. diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst index a601b16f1c..b94f1bfadd 100644 --- a/Doc/whatsnew/3.9.rst +++ b/Doc/whatsnew/3.9.rst @@ -135,8 +135,8 @@ More generally, try to run your tests in the :ref:`Python Development Mode <devmode>` which helps to prepare your code to make it compatible with the next Python version. -Note: a number of pre-existing deprecatations were removed in this version -of Python as well. Consult the :ref:`removed-in-python-39` section. +Note: a number of pre-existing deprecations were removed in this version of +Python as well. Consult the :ref:`removed-in-python-39` section. New Features @@ -1454,3 +1454,65 @@ Removed ``PyNullImporter_Type``, ``PyCmpWrapper_Type``, ``PySortWrapper_Type``, ``PyNoArgsFunction``. (Contributed by Pablo Galindo Salgado in :issue:`39372`.) + +Notable changes in Python 3.9.1 +=============================== + +typing +------ + +The behavior of :class:`typing.Literal` was changed to conform with :pep:`586` +and to match the behavior of static type checkers specified in the PEP. + +1. ``Literal`` now de-duplicates parameters. +2. Equality comparisons between ``Literal`` objects are now order independent. +3. ``Literal`` comparisons now respect types. For example, + ``Literal[0] == Literal[False]`` previously evaluated to ``True``. It is + now ``False``. To support this change, the internally used type cache now + supports differentiating types. +4. ``Literal`` objects will now raise a :exc:`TypeError` exception during + equality comparisons if one of their parameters are not :term:`immutable`. + Note that declaring ``Literal`` with mutable parameters will not throw + an error:: + + >>> from typing import Literal + >>> Literal[{0}] + >>> Literal[{0}] == Literal[{False}] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: unhashable type: 'set' + +(Contributed by Yurii Karabas in :issue:`42345`.) + +macOS 11.0 (Big Sur) and Apple Silicon Mac support +-------------------------------------------------- + +As of 3.9.1, Python now fully supports building and running on macOS 11.0 +(Big Sur) and on Apple Silicon Macs (based on the ``ARM64`` architecture). +A new universal build variant, ``universal2``, is now available to natively +support both ``ARM64`` and ``Intel 64`` in one set of executables. Binaries +can also now be built on current versions of macOS to be deployed on a range +of older macOS versions (tested to 10.9) while making some newer OS +functions and options conditionally available based on the operating system +version in use at runtime ("weaklinking"). + +(Contributed by Ronald Oussoren and Lawrence D'Anna in :issue:`41100`.) + +Notable changes in Python 3.9.2 +=============================== + +collections.abc +--------------- + +:class:`collections.abc.Callable` generic now flattens type parameters, similar +to what :data:`typing.Callable` currently does. This means that +``collections.abc.Callable[[int, str], str]`` will have ``__args__`` of +``(int, str, str)``; previously this was ``([int, str], str)``. To allow this +change, :class:`types.GenericAlias` can now be subclassed, and a subclass will +be returned when subscripting the :class:`collections.abc.Callable` type. +Code which accesses the arguments via :func:`typing.get_args` or ``__args__`` +need to account for this change. A :exc:`DeprecationWarning` may be emitted for +invalid forms of parameterizing :class:`collections.abc.Callable` which may have +passed silently in Python 3.9.1. This :exc:`DeprecationWarning` will +become a :exc:`TypeError` in Python 3.10. +(Contributed by Ken Jin in :issue:`42195`.)
\ No newline at end of file |