diff options
| author | Martin Panter <vadmium+py@gmail.com> | 2016-10-22 03:21:36 +0000 |
|---|---|---|
| committer | Martin Panter <vadmium+py@gmail.com> | 2016-10-22 03:21:36 +0000 |
| commit | 73a0af3545f0e9b62ac149367c0cfbdbfe40afe4 (patch) | |
| tree | 9db0bac0a71ca7296ff714745c0f4eee0b07f6a5 /Doc | |
| parent | 6f5021addeb1d9ddd3610409d2461ec79870f9a5 (diff) | |
| parent | 6fec8a2a83b0f93a80bb83e76dceeda1cf9d00dc (diff) | |
| download | cpython-73a0af3545f0e9b62ac149367c0cfbdbfe40afe4.tar.gz | |
Issue #28435: Merge urllib test fixes from 3.5 into 3.6
Diffstat (limited to 'Doc')
176 files changed, 9183 insertions, 3125 deletions
diff --git a/Doc/Makefile b/Doc/Makefile index 10c3288c59..91f937f985 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -153,7 +153,7 @@ dist: cp -pPR build/epub/Python.epub dist/python-$(DISTVERSION)-docs.epub check: - $(PYTHON) tools/rstlint.py -i tools + $(PYTHON) tools/rstlint.py -i tools -i venv serve: ../Tools/scripts/serve.py build/html diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index d5e4703ff5..569903f000 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -223,8 +223,7 @@ which disallows mutable objects such as :class:`bytearray`. :c:func:`PyArg_ParseTuple` will use this location as the buffer and interpret the initial value of *\*buffer_length* as the buffer size. It will then copy the encoded data into the buffer and NUL-terminate it. If the buffer is not large - enough, a :exc:`TypeError` will be set. - Note: starting from Python 3.6 a :exc:`ValueError` will be set. + enough, a :exc:`ValueError` will be set. In both cases, *\*buffer_length* is set to the length of the encoded data without the trailing NUL byte. @@ -266,15 +265,12 @@ Numbers Convert a Python integer to a C :c:type:`unsigned long` without overflow checking. -``L`` (:class:`int`) [PY_LONG_LONG] - Convert a Python integer to a C :c:type:`long long`. This format is only - available on platforms that support :c:type:`long long` (or :c:type:`_int64` on - Windows). +``L`` (:class:`int`) [long long] + Convert a Python integer to a C :c:type:`long long`. -``K`` (:class:`int`) [unsigned PY_LONG_LONG] +``K`` (:class:`int`) [unsigned long long] Convert a Python integer to a C :c:type:`unsigned long long` - without overflow checking. This format is only available on platforms that - support :c:type:`unsigned long long` (or :c:type:`unsigned _int64` on Windows). + without overflow checking. ``n`` (:class:`int`) [Py_ssize_t] Convert a Python integer to a C :c:type:`Py_ssize_t`. @@ -424,8 +420,15 @@ API Functions .. c:function:: int PyArg_ParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], ...) Parse the parameters of a function that takes both positional and keyword - parameters into local variables. Returns true on success; on failure, it - returns false and raises the appropriate exception. + parameters into local variables. The *keywords* argument is a + *NULL*-terminated array of keyword parameter names. Empty names denote + :ref:`positional-only parameters <positional-only_parameter>`. + Returns true on success; on failure, it returns false and raises the + appropriate exception. + + .. versionchanged:: 3.6 + Added support for :ref:`positional-only parameters + <positional-only_parameter>`. .. c:function:: int PyArg_VaParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], va_list vargs) @@ -588,15 +591,11 @@ Building values ``k`` (:class:`int`) [unsigned long] Convert a C :c:type:`unsigned long` to a Python integer object. - ``L`` (:class:`int`) [PY_LONG_LONG] - Convert a C :c:type:`long long` to a Python integer object. Only available - on platforms that support :c:type:`long long` (or :c:type:`_int64` on - Windows). + ``L`` (:class:`int`) [long long] + Convert a C :c:type:`long long` to a Python integer object. - ``K`` (:class:`int`) [unsigned PY_LONG_LONG] - Convert a C :c:type:`unsigned long long` to a Python integer object. Only - available on platforms that support :c:type:`unsigned long long` (or - :c:type:`unsigned _int64` on Windows). + ``K`` (:class:`int`) [unsigned long long] + Convert a C :c:type:`unsigned long long` to a Python integer object. ``n`` (:class:`int`) [Py_ssize_t] Convert a C :c:type:`Py_ssize_t` to a Python integer. diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index 19cbb3bcb8..25fb29c48c 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -306,6 +306,13 @@ an error value). :mod:`warnings` module and the :option:`-W` option in the command line documentation. There is no C API for warning control. +.. c:function:: PyObject* PyErr_SetImportErrorSubclass(PyObject *msg, PyObject *name, PyObject *path) + + Much like :c:func:`PyErr_SetImportError` but this function allows for + specifying a subclass of :exc:`ImportError` to raise. + + .. versionadded:: 3.4 + .. c:function:: int PyErr_WarnExplicitObject(PyObject *category, PyObject *message, PyObject *filename, int lineno, PyObject *module, PyObject *registry) @@ -334,6 +341,14 @@ an error value). .. versionadded:: 3.2 +.. c:function:: int PyErr_ResourceWarning(PyObject *source, Py_ssize_t stack_level, const char *format, ...) + + Function similar to :c:func:`PyErr_WarnFormat`, but *category* is + :exc:`ResourceWarning` and pass *source* to :func:`warnings.WarningMessage`. + + .. versionadded:: 3.6 + + Querying the error indicator ============================ @@ -774,6 +789,8 @@ the variables: +-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_ImportError` | :exc:`ImportError` | | +-----------------------------------------+---------------------------------+----------+ +| :c:data:`PyExc_ModuleNotFoundError` | :exc:`ModuleNotFoundError` | | ++-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_IndexError` | :exc:`IndexError` | | +-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_InterruptedError` | :exc:`InterruptedError` | | diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index e78ddae708..c272a1cdaf 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -25,7 +25,7 @@ Initializing and finalizing the interpreter triple: module; search; path single: PySys_SetArgv() single: PySys_SetArgvEx() - single: Py_Finalize() + single: Py_FinalizeEx() Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; with the @@ -34,7 +34,7 @@ Initializing and finalizing the interpreter modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. It also initializes the module search path (``sys.path``). It does not set ``sys.argv``; use :c:func:`PySys_SetArgvEx` for that. This is a no-op when called for a second time - (without calling :c:func:`Py_Finalize` first). There is no return value; it is a + (without calling :c:func:`Py_FinalizeEx` first). There is no return value; it is a fatal error if the initialization fails. .. note:: @@ -52,19 +52,20 @@ Initializing and finalizing the interpreter .. c:function:: int Py_IsInitialized() Return true (nonzero) when the Python interpreter has been initialized, false - (zero) if not. After :c:func:`Py_Finalize` is called, this returns false until + (zero) if not. After :c:func:`Py_FinalizeEx` is called, this returns false until :c:func:`Py_Initialize` is called again. -.. c:function:: void Py_Finalize() +.. c:function:: int Py_FinalizeEx() Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of Python/C API functions, and destroy all sub-interpreters (see :c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since the last call to :c:func:`Py_Initialize`. Ideally, this frees all memory allocated by the Python interpreter. This is a no-op when called for a second - time (without calling :c:func:`Py_Initialize` again first). There is no return - value; errors during finalization are ignored. + time (without calling :c:func:`Py_Initialize` again first). Normally the + return value is 0. If there were errors during finalization + (flushing buffered data), -1 is returned. This function is provided for a number of reasons. An embedding application might want to restart Python without having to restart the application itself. @@ -83,7 +84,15 @@ Initializing and finalizing the interpreter freed. Some memory allocated by extension modules may not be freed. Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls :c:func:`Py_Initialize` and - :c:func:`Py_Finalize` more than once. + :c:func:`Py_FinalizeEx` more than once. + + .. versionadded:: 3.6 + + +.. c:function:: void Py_Finalize() + + This is a backwards-compatible version of :c:func:`Py_FinalizeEx` that + disregards the return value. Process-wide parameters @@ -111,7 +120,7 @@ Process-wide parameters Note that :data:`sys.stderr` always uses the "backslashreplace" error handler, regardless of this (or any other) setting. - If :c:func:`Py_Finalize` is called, this function will need to be called + If :c:func:`Py_FinalizeEx` is called, this function will need to be called again in order to affect subsequent calls to :c:func:`Py_Initialize`. Returns 0 if successful, a nonzero value on error (e.g. calling after the @@ -922,7 +931,7 @@ using the following functions: entry.) .. index:: - single: Py_Finalize() + single: Py_FinalizeEx() single: Py_Initialize() Extension modules are shared between (sub-)interpreters as follows: the first @@ -932,7 +941,7 @@ using the following functions: and filled with the contents of this copy; the extension's ``init`` function is not called. Note that this is different from what happens when an extension is imported after the interpreter has been completely re-initialized by calling - :c:func:`Py_Finalize` and :c:func:`Py_Initialize`; in that case, the extension's + :c:func:`Py_FinalizeEx` and :c:func:`Py_Initialize`; in that case, the extension's ``initmodule`` function *is* called again. .. index:: single: close() (in module os) @@ -940,14 +949,14 @@ using the following functions: .. c:function:: void Py_EndInterpreter(PyThreadState *tstate) - .. index:: single: Py_Finalize() + .. index:: single: Py_FinalizeEx() Destroy the (sub-)interpreter represented by the given thread state. The given thread state must be the current thread state. See the discussion of thread states below. When the call returns, the current thread state is *NULL*. All thread states associated with this interpreter are destroyed. (The global interpreter lock must be held before calling this function and is still held - when it returns.) :c:func:`Py_Finalize` will destroy all sub-interpreters that + when it returns.) :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that haven't been explicitly destroyed at that point. diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst index bc3a752132..74681d2c85 100644 --- a/Doc/c-api/intro.rst +++ b/Doc/c-api/intro.rst @@ -64,9 +64,10 @@ The header files are typically installed with Python. On Unix, these are located in the directories :file:`{prefix}/include/pythonversion/` and :file:`{exec_prefix}/include/pythonversion/`, where :envvar:`prefix` and :envvar:`exec_prefix` are defined by the corresponding parameters to Python's -:program:`configure` script and *version* is ``sys.version[:3]``. On Windows, -the headers are installed in :file:`{prefix}/include`, where :envvar:`prefix` is -the installation directory specified to the installer. +:program:`configure` script and *version* is +``'%d.%d' % sys.version_info[:2]``. On Windows, the headers are installed +in :file:`{prefix}/include`, where :envvar:`prefix` is the installation +directory specified to the installer. To include the headers, place both directories (if different) on your compiler's search path for includes. Do *not* place the parent directories on the search @@ -578,9 +579,9 @@ Sometimes, it is desirable to "uninitialize" Python. For instance, the application may want to start over (make another call to :c:func:`Py_Initialize`) or the application is simply done with its use of Python and wants to free memory allocated by Python. This can be accomplished -by calling :c:func:`Py_Finalize`. The function :c:func:`Py_IsInitialized` returns +by calling :c:func:`Py_FinalizeEx`. The function :c:func:`Py_IsInitialized` returns true if Python is currently in the initialized state. More information about -these functions is given in a later chapter. Notice that :c:func:`Py_Finalize` +these functions is given in a later chapter. Notice that :c:func:`Py_FinalizeEx` does *not* free all memory allocated by the Python interpreter, e.g. memory allocated by extension modules currently cannot be released. diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst index b3480159c0..bae9703590 100644 --- a/Doc/c-api/long.rst +++ b/Doc/c-api/long.rst @@ -62,13 +62,13 @@ All integers are implemented as "long" integer objects of arbitrary size. *NULL* on failure. -.. c:function:: PyObject* PyLong_FromLongLong(PY_LONG_LONG v) +.. c:function:: PyObject* PyLong_FromLongLong(long long v) Return a new :c:type:`PyLongObject` object from a C :c:type:`long long`, or *NULL* on failure. -.. c:function:: PyObject* PyLong_FromUnsignedLongLong(unsigned PY_LONG_LONG v) +.. c:function:: PyObject* PyLong_FromUnsignedLongLong(unsigned long long v) Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long long`, or *NULL* on failure. @@ -148,7 +148,7 @@ All integers are implemented as "long" integer objects of arbitrary size. occurs set *\*overflow* to ``0`` and return ``-1`` as usual. -.. c:function:: PY_LONG_LONG PyLong_AsLongLong(PyObject *obj) +.. c:function:: long long PyLong_AsLongLong(PyObject *obj) .. index:: single: OverflowError (built-in exception) @@ -161,7 +161,7 @@ All integers are implemented as "long" integer objects of arbitrary size. :c:type:`long`. -.. c:function:: PY_LONG_LONG PyLong_AsLongLongAndOverflow(PyObject *obj, int *overflow) +.. c:function:: long long PyLong_AsLongLongAndOverflow(PyObject *obj, int *overflow) Return a C :c:type:`long long` representation of *obj*. If *obj* is not an instance of :c:type:`PyLongObject`, first call its :meth:`__int__` method @@ -210,16 +210,16 @@ All integers are implemented as "long" integer objects of arbitrary size. :c:type:`size_t`. -.. c:function:: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLong(PyObject *pylong) +.. c:function:: unsigned long long PyLong_AsUnsignedLongLong(PyObject *pylong) .. index:: single: OverflowError (built-in exception) - Return a C :c:type:`unsigned PY_LONG_LONG` representation of *pylong*. - *pylong* must be an instance of :c:type:`PyLongObject`. + Return a C :c:type:`unsigned long long` representation of *pylong*. *pylong* + must be an instance of :c:type:`PyLongObject`. Raise :exc:`OverflowError` if the value of *pylong* is out of range for an - :c:type:`unsigned PY_LONG_LONG`. + :c:type:`unsigned long long`. .. versionchanged:: 3.1 A negative *pylong* now raises :exc:`OverflowError`, not :exc:`TypeError`. @@ -235,7 +235,7 @@ All integers are implemented as "long" integer objects of arbitrary size. return the reduction of that value modulo :const:`ULONG_MAX + 1`. -.. c:function:: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLongMask(PyObject *obj) +.. c:function:: unsigned long long PyLong_AsUnsignedLongLongMask(PyObject *obj) Return a C :c:type:`unsigned long long` representation of *obj*. If *obj* is not an instance of :c:type:`PyLongObject`, first call its :meth:`__int__` diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 290ef09dce..3ff545275f 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -85,9 +85,12 @@ for the I/O buffer escapes completely the Python memory manager. .. seealso:: + The :envvar:`PYTHONMALLOC` environment variable can be used to configure + the memory allocators used by Python. + The :envvar:`PYTHONMALLOCSTATS` environment variable can be used to print - memory allocation statistics every time a new object arena is created, and - on shutdown. + statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a + new pymalloc object arena is created, and on shutdown. Raw Memory Interface @@ -162,15 +165,17 @@ The following function sets, modeled after the ANSI C standard, but specifying behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap. -The default memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call -``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. +By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`. .. warning:: The :term:`GIL <global interpreter lock>` must be held when using these functions. +.. versionchanged:: 3.6 + + The default allocator is now pymalloc instead of system :c:func:`malloc`. + .. c:function:: void* PyMem_Malloc(size_t n) Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the @@ -292,15 +297,32 @@ Customize Memory Allocators Enum used to identify an allocator domain. Domains: - * :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`, - :c:func:`PyMem_RawRealloc`, :c:func:`PyMem_RawCalloc` and - :c:func:`PyMem_RawFree` - * :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`, - :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc` and :c:func:`PyMem_Free` - * :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`, - :c:func:`PyObject_Realloc`, :c:func:`PyObject_Calloc` and - :c:func:`PyObject_Free` + .. c:var:: PYMEM_DOMAIN_RAW + + Functions: + + * :c:func:`PyMem_RawMalloc` + * :c:func:`PyMem_RawRealloc` + * :c:func:`PyMem_RawCalloc` + * :c:func:`PyMem_RawFree` + + .. c:var:: PYMEM_DOMAIN_MEM + + Functions: + + * :c:func:`PyMem_Malloc`, + * :c:func:`PyMem_Realloc` + * :c:func:`PyMem_Calloc` + * :c:func:`PyMem_Free` + .. c:var:: PYMEM_DOMAIN_OBJ + + Functions: + + * :c:func:`PyObject_Malloc` + * :c:func:`PyObject_Realloc` + * :c:func:`PyObject_Calloc` + * :c:func:`PyObject_Free` .. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) @@ -325,43 +347,62 @@ Customize Memory Allocators .. c:function:: void PyMem_SetupDebugHooks(void) - Setup hooks to detect bugs in the following Python memory allocator - functions: - - - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`, - :c:func:`PyMem_RawCalloc`, :c:func:`PyMem_RawFree` - - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc`, - :c:func:`PyMem_Free` - - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`, - :c:func:`PyObject_Calloc`, :c:func:`PyObject_Free` + Setup hooks to detect bugs in the Python memory allocator functions. Newly allocated memory is filled with the byte ``0xCB``, freed memory is - filled with the byte ``0xDB``. Additional checks: + filled with the byte ``0xDB``. - - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer + Runtime checks: + + - Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer allocated by :c:func:`PyMem_Malloc` - - detect write before the start of the buffer (buffer underflow) - - detect write after the end of the buffer (buffer overflow) + - Detect write before the start of the buffer (buffer underflow) + - Detect write after the end of the buffer (buffer overflow) + - Check that the :term:`GIL <global interpreter lock>` is held when + allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: + :c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex: + :c:func:`PyMem_Malloc`) domains are called + + On error, the debug hooks use the :mod:`tracemalloc` module to get the + traceback where a memory block was allocated. The traceback is only + displayed if :mod:`tracemalloc` is tracing Python memory allocations and the + memory block was traced. - The function does nothing if Python is not compiled is debug mode. + These hooks are installed by default if Python is compiled in debug + mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install + debug hooks on a Python compiled in release mode. + .. versionchanged:: 3.6 + This function now also works on Python compiled in release mode. + On error, the debug hooks now use :mod:`tracemalloc` to get the traceback + where a memory block was allocated. The debug hooks now also check + if the GIL is held when functions of :c:data:`PYMEM_DOMAIN_OBJ` and + :c:data:`PYMEM_DOMAIN_MEM` domains are called. -Customize PyObject Arena Allocator -================================== -Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This -allocator is optimized for small objects with a short lifetime. It uses memory -mappings called "arenas" with a fixed size of 256 KB. It falls back to -:c:func:`PyMem_RawMalloc` and :c:func:`PyMem_RawRealloc` for allocations larger -than 512 bytes. *pymalloc* is the default allocator used by -:c:func:`PyObject_Malloc`. +.. _pymalloc: -The default arena allocator uses the following functions: +The pymalloc allocator +====================== + +Python has a *pymalloc* allocator optimized for small objects (smaller or equal +to 512 bytes) with a short lifetime. It uses memory mappings called "arenas" +with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and +:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes. + +*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_MEM` (ex: +:c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_OBJ` (ex: +:c:func:`PyObject_Malloc`) domains. + +The arena allocator uses the following functions: * :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows, * :c:func:`mmap` and :c:func:`munmap` if available, * :c:func:`malloc` and :c:func:`free` otherwise. +Customize pymalloc Arena Allocator +---------------------------------- + .. versionadded:: 3.4 .. c:type:: PyObjectArenaAllocator diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index cc843146bd..e9e8add6c1 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -150,8 +150,9 @@ specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. The individual flags indicate either a calling convention or a binding convention. Of the calling convention flags, only :const:`METH_VARARGS` and -:const:`METH_KEYWORDS` can be combined. Any of the calling convention flags -can be combined with a binding flag. +:const:`METH_KEYWORDS` can be combined (but note that :const:`METH_KEYWORDS` +alone is equivalent to ``METH_VARARGS | METH_KEYWORDS``). Any of the calling +convention flags can be combined with a binding flag. .. data:: METH_VARARGS diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index 3d83b279c2..035cdc1682 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -5,6 +5,17 @@ Operating System Utilities ========================== +.. c:function:: PyObject* PyOS_FSPath(PyObject *path) + + Return the file system representation for *path*. If the object is a + :class:`str` or :class:`bytes` object, then its reference count is + incremented. If the object implements the :class:`os.PathLike` interface, + then :meth:`~os.PathLike.__fspath__` is returned as long as it is a + :class:`str` or :class:`bytes` object. Otherwise :exc:`TypeError` is raised + and ``NULL`` is returned. + + .. versionadded:: 3.6 + .. c:function:: int Py_FdIsInteractive(FILE *fp, const char *filename) @@ -212,20 +223,24 @@ Process Control .. c:function:: void Py_Exit(int status) .. index:: - single: Py_Finalize() + single: Py_FinalizeEx() single: exit() - Exit the current process. This calls :c:func:`Py_Finalize` and then calls the - standard C library function ``exit(status)``. + Exit the current process. This calls :c:func:`Py_FinalizeEx` and then calls the + standard C library function ``exit(status)``. If :c:func:`Py_FinalizeEx` + indicates an error, the exit status is set to 120. + + .. versionchanged:: 3.6 + Errors from finalization no longer ignored. .. c:function:: int Py_AtExit(void (*func) ()) .. index:: - single: Py_Finalize() + single: Py_FinalizeEx() single: cleanup functions - Register a cleanup function to be called by :c:func:`Py_Finalize`. The cleanup + Register a cleanup function to be called by :c:func:`Py_FinalizeEx`. The cleanup function will be called with no arguments and should return no value. At most 32 cleanup functions can be registered. When the registration is successful, :c:func:`Py_AtExit` returns ``0``; on failure, it returns ``-1``. The cleanup diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 26831c47a2..f02318b682 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -440,7 +440,6 @@ APIs: .. % because not all compilers support the %z width modifier -- we fake it .. % when necessary via interpolating PY_FORMAT_SIZE_T. .. % Similar comments apply to the %ll width modifier and - .. % PY_FORMAT_LONG_LONG. .. tabularcolumns:: |l|l|L| @@ -525,11 +524,6 @@ APIs: copied as-is to the result string, and any extra arguments discarded. .. note:: - - The `"%lld"` and `"%llu"` format specifiers are only available - when :const:`HAVE_LONG_LONG` is defined. - - .. note:: The width formatter unit is number of characters rather than bytes. The precision formatter unit is number of bytes for ``"%s"`` and ``"%V"`` (if the ``PyObject*`` argument is NULL), and a number of @@ -811,38 +805,47 @@ File System Encoding """""""""""""""""""" To encode and decode file names and other environment strings, -:c:data:`Py_FileSystemEncoding` should be used as the encoding, and -``"surrogateescape"`` should be used as the error handler (:pep:`383`). To -encode file names during argument parsing, the ``"O&"`` converter should be -used, passing :c:func:`PyUnicode_FSConverter` as the conversion function: +:c:data:`Py_FileSystemDefaultEncoding` should be used as the encoding, and +:c:data:`Py_FileSystemDefaultEncodeErrors` should be used as the error handler +(:pep:`383` and :pep:`529`). To encode file names to :class:`bytes` during +argument parsing, the ``"O&"`` converter should be used, passing +:c:func:`PyUnicode_FSConverter` as the conversion function: .. c:function:: int PyUnicode_FSConverter(PyObject* obj, void* result) - ParseTuple converter: encode :class:`str` objects to :class:`bytes` using + ParseTuple converter: encode :class:`str` objects -- obtained directly or + through the :class:`os.PathLike` interface -- to :class:`bytes` using :c:func:`PyUnicode_EncodeFSDefault`; :class:`bytes` objects are output as-is. *result* must be a :c:type:`PyBytesObject*` which must be released when it is no longer used. .. versionadded:: 3.1 + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. -To decode file names during argument parsing, the ``"O&"`` converter should be -used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: +To decode file names to :class:`str` during argument parsing, the ``"O&"`` +converter should be used, passing :c:func:`PyUnicode_FSDecoder` as the +conversion function: .. c:function:: int PyUnicode_FSDecoder(PyObject* obj, void* result) - ParseTuple converter: decode :class:`bytes` objects to :class:`str` using - :c:func:`PyUnicode_DecodeFSDefaultAndSize`; :class:`str` objects are output - as-is. *result* must be a :c:type:`PyUnicodeObject*` which must be released - when it is no longer used. + ParseTuple converter: decode :class:`bytes` objects -- obtained either + directly or indirectly through the :class:`os.PathLike` interface -- to + :class:`str` using :c:func:`PyUnicode_DecodeFSDefaultAndSize`; :class:`str` + objects are output as-is. *result* must be a :c:type:`PyUnicodeObject*` which + must be released when it is no longer used. .. versionadded:: 3.2 + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. c:function:: PyObject* PyUnicode_DecodeFSDefaultAndSize(const char *s, Py_ssize_t size) Decode a string using :c:data:`Py_FileSystemDefaultEncoding` and the - ``"surrogateescape"`` error handler, or ``"strict"`` on Windows. + :c:data:`Py_FileSystemDefaultEncodeErrors` error handler. If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. @@ -856,28 +859,28 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: The :c:func:`Py_DecodeLocale` function. - .. versionchanged:: 3.2 - Use ``"strict"`` error handler on Windows. + .. versionchanged:: 3.6 + Use :c:data:`Py_FileSystemDefaultEncodeErrors` error handler. .. c:function:: PyObject* PyUnicode_DecodeFSDefault(const char *s) Decode a null-terminated string using :c:data:`Py_FileSystemDefaultEncoding` - and the ``"surrogateescape"`` error handler, or ``"strict"`` on Windows. + and the :c:data:`Py_FileSystemDefaultEncodeErrors` error handler. If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` if you know the string length. - .. versionchanged:: 3.2 - Use ``"strict"`` error handler on Windows. + .. versionchanged:: 3.6 + Use :c:data:`Py_FileSystemDefaultEncodeErrors` error handler. .. c:function:: PyObject* PyUnicode_EncodeFSDefault(PyObject *unicode) Encode a Unicode object to :c:data:`Py_FileSystemDefaultEncoding` with the - ``"surrogateescape"`` error handler, or ``"strict"`` on Windows, and return + :c:data:`Py_FileSystemDefaultEncodeErrors` error handler, and return :class:`bytes`. Note that the resulting :class:`bytes` object may contain null bytes. @@ -894,6 +897,8 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: .. versionadded:: 3.2 + .. versionchanged:: 3.6 + Use :c:data:`Py_FileSystemDefaultEncodeErrors` error handler. wchar_t Support """"""""""""""" diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index e388195757..8b469f4aac 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -921,6 +921,9 @@ PyNumber_Xor:PyObject*:o2:0: PyObject_AsFileDescriptor:int::: PyObject_AsFileDescriptor:PyObject*:o:0: +PyOS_FSPath:PyObject*::+1: +PyOS_FSPath:PyObject*:path:0: + PyObject_Call:PyObject*::+1: PyObject_Call:PyObject*:callable_object:0: PyObject_Call:PyObject*:args:0: diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst index 64033dcd97..ab2f61614a 100644 --- a/Doc/extending/embedding.rst +++ b/Doc/extending/embedding.rst @@ -67,7 +67,9 @@ perform some operation on a file. :: Py_Initialize(); PyRun_SimpleString("from time import time,ctime\n" "print('Today is', ctime(time()))\n"); - Py_Finalize(); + if (Py_FinalizeEx() < 0) { + exit(120); + } PyMem_RawFree(program); return 0; } @@ -76,7 +78,7 @@ The :c:func:`Py_SetProgramName` function should be called before :c:func:`Py_Initialize` to inform the interpreter about paths to Python run-time libraries. Next, the Python interpreter is initialized with :c:func:`Py_Initialize`, followed by the execution of a hard-coded Python script -that prints the date and time. Afterwards, the :c:func:`Py_Finalize` call shuts +that prints the date and time. Afterwards, the :c:func:`Py_FinalizeEx` call shuts the interpreter down, followed by the end of the program. In a real program, you may want to get the Python script from another source, perhaps a text-editor routine, a file, or a database. Getting the Python code from a file can better diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 694753e5b9..9c5e20dcad 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -838,7 +838,8 @@ How do I convert a number to a string? To convert, e.g., the number 144 to the string '144', use the built-in type constructor :func:`str`. If you want a hexadecimal or octal representation, use the built-in functions :func:`hex` or :func:`oct`. For fancy formatting, see -the :ref:`formatstrings` section, e.g. ``"{:04d}".format(144)`` yields +the :ref:`f-strings` and :ref:`formatstrings` sections, +e.g. ``"{:04d}".format(144)`` yields ``'0144'`` and ``"{:.3f}".format(1.0/3.0)`` yields ``'0.333'``. diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 45b794f48e..f80b6df212 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -718,6 +718,8 @@ Glossary def func(foo, bar=None): ... + .. _positional-only_parameter: + * :dfn:`positional-only`: specifies an argument that can be supplied only by position. Python has no syntax for defining positional-only parameters. However, some built-in functions have positional-only @@ -776,6 +778,16 @@ Glossary One of the default :term:`meta path finders <meta path finder>` which searches an :term:`import path` for modules. + path-like object + An object representing a file system path. A path-like object is either + a :class:`str` or :class:`bytes` object representing a path, or an object + implementing the :class:`os.PathLike` protocol. An object that supports + the :class:`os.PathLike` protocol can be converted to a :class:`str` or + :class:`bytes` file system path by calling the :func:`os.fspath` function; + :func:`os.fsdecode` and :func:`os.fsencode` can be used to guarantee a + :class:`str` or :class:`bytes` result instead, respectively. Introduced + by :pep:`519`. + portion A set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package, as defined in :pep:`420`. @@ -952,13 +964,25 @@ Glossary ``'\r'``. See :pep:`278` and :pep:`3116`, as well as :func:`bytes.splitlines` for an additional use. + variable annotation + A type metadata value associated with a module global variable or + a class attribute. Its syntax is explained in section :ref:`annassign`. + Annotations are stored in the :attr:`__annotations__` special + attribute of a class or module object and can be accessed using + :func:`typing.get_type_hints`. + + Python itself does not assign any particular meaning to variable + annotations. They are intended to be interpreted by third-party libraries + or type checking tools. See :pep:`526`, :pep:`484` which describe + some of their potential uses. + virtual environment A cooperatively isolated runtime environment that allows Python users and applications to install and upgrade Python distribution packages without interfering with the behaviour of other Python applications running on the same system. - See also :ref:`scripts-pyvenv`. + See also :mod:`venv`. virtual machine A computer defined entirely in software. Python's virtual machine diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst index 3b79b92082..7e161a59ad 100644 --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -547,7 +547,8 @@ And this is what it gives: Traceback (most recent call last): File "prog.py", line 11, in <module> if args.verbosity >= 2: - TypeError: unorderable types: NoneType() >= int() + TypeError: '>=' not supported between instances of 'NoneType' and 'int' + * First output went well, and fixes the bug we had before. That is, we want any value >= 2 to be as verbose as possible. diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index caa497521c..f435884082 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -827,9 +827,9 @@ on the right is the text you'd replace it with. ``'i'`` ``int`` ``'I'`` ``unsigned_int(bitwise=True)`` ``'k'`` ``unsigned_long(bitwise=True)`` -``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` +``'K'`` ``unsigned_long_long(bitwise=True)`` ``'l'`` ``long`` -``'L'`` ``PY_LONG_LONG`` +``'L'`` ``long long`` ``'n'`` ``Py_ssize_t`` ``'O'`` ``object`` ``'O!'`` ``object(subclass_of='&PySomething_Type')`` diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst index de659505f7..593341cc2b 100644 --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -28,4 +28,5 @@ Currently, the HOWTOs are: argparse.rst ipaddress.rst clinic.rst + instrumentation.rst diff --git a/Doc/howto/instrumentation.rst b/Doc/howto/instrumentation.rst new file mode 100644 index 0000000000..621a608c55 --- /dev/null +++ b/Doc/howto/instrumentation.rst @@ -0,0 +1,411 @@ +.. _instrumentation: + +=============================================== +Instrumenting CPython with DTrace and SystemTap +=============================================== + +:author: David Malcolm +:author: Łukasz Langa + +DTrace and SystemTap are monitoring tools, each providing a way to inspect +what the processes on a computer system are doing. They both use +domain-specific languages allowing a user to write scripts which: + + - filter which processes are to be observed + - gather data from the processes of interest + - generate reports on the data + +As of Python 3.6, CPython can be built with embedded "markers", also +known as "probes", that can be observed by a DTrace or SystemTap script, +making it easier to monitor what the CPython processes on a system are +doing. + +.. I'm using ".. code-block:: c" for SystemTap scripts, as "c" is syntactically + the closest match that Sphinx supports + +.. impl-detail:: + + DTrace markers are implementation details of the CPython interpreter. + No guarantees are made about probe compatibility between versions of + CPython. DTrace scripts can stop working or work incorrectly without + warning when changing CPython versions. + + +Enabling the static markers +--------------------------- + +macOS comes with built-in support for DTrace. On Linux, in order to +build CPython with the embedded markers for SystemTap, the SystemTap +development tools must be installed. + +On a Linux machine, this can be done via:: + + yum install systemtap-sdt-devel + +or:: + + sudo apt-get install systemtap-sdt-dev + + +CPython must then be configured `--with-dtrace`:: + + checking for --with-dtrace... yes + +On macOS, you can list available DTrace probes by running a Python +process in the background and listing all probes made available by the +Python provider:: + + $ python3.6 -q & + $ sudo dtrace -l -P python$! # or: dtrace -l -m python3.6 + + ID PROVIDER MODULE FUNCTION NAME + 29564 python18035 python3.6 _PyEval_EvalFrameDefault function-entry + 29565 python18035 python3.6 dtrace_function_entry function-entry + 29566 python18035 python3.6 _PyEval_EvalFrameDefault function-return + 29567 python18035 python3.6 dtrace_function_return function-return + 29568 python18035 python3.6 collect gc-done + 29569 python18035 python3.6 collect gc-start + 29570 python18035 python3.6 _PyEval_EvalFrameDefault line + 29571 python18035 python3.6 maybe_dtrace_line line + +On Linux, you can verify if the SystemTap static markers are present in +the built binary by seeing if it contains a ".note.stapsdt" section. + +.. code-block:: bash + + $ readelf -S ./python | grep .note.stapsdt + [30] .note.stapsdt NOTE 0000000000000000 00308d78 + +If you've built Python as a shared library (with --enable-shared), you +need to look instead within the shared library. For example: + +.. code-block:: bash + + $ readelf -S libpython3.3dm.so.1.0 | grep .note.stapsdt + [29] .note.stapsdt NOTE 0000000000000000 00365b68 + +Sufficiently modern readelf can print the metadata: + +.. code-block:: bash + + $ readelf -n ./python + + Displaying notes found at file offset 0x00000254 with length 0x00000020: + Owner Data size Description + GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag) + OS: Linux, ABI: 2.6.32 + + Displaying notes found at file offset 0x00000274 with length 0x00000024: + Owner Data size Description + GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) + Build ID: df924a2b08a7e89f6e11251d4602022977af2670 + + Displaying notes found at file offset 0x002d6c30 with length 0x00000144: + Owner Data size Description + stapsdt 0x00000031 NT_STAPSDT (SystemTap probe descriptors) + Provider: python + Name: gc__start + Location: 0x00000000004371c3, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf6 + Arguments: -4@%ebx + stapsdt 0x00000030 NT_STAPSDT (SystemTap probe descriptors) + Provider: python + Name: gc__done + Location: 0x00000000004374e1, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf8 + Arguments: -8@%rax + stapsdt 0x00000045 NT_STAPSDT (SystemTap probe descriptors) + Provider: python + Name: function__entry + Location: 0x000000000053db6c, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6be8 + Arguments: 8@%rbp 8@%r12 -4@%eax + stapsdt 0x00000046 NT_STAPSDT (SystemTap probe descriptors) + Provider: python + Name: function__return + Location: 0x000000000053dba8, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bea + Arguments: 8@%rbp 8@%r12 -4@%eax + +The above metadata contains information for SystemTap describing how it +can patch strategically-placed machine code instructions to enable the +tracing hooks used by a SystemTap script. + + +Static DTrace probes +-------------------- + +The following example DTrace script can be used to show the call/return +hierarchy of a Python script, only tracing within the invocation of +a function called "start". In other words, import-time function +invocations are not going to be listed: + +.. code-block:: c + + self int indent; + + python$target:::function-entry + /copyinstr(arg1) == "start"/ + { + self->trace = 1; + } + + python$target:::function-entry + /self->trace/ + { + printf("%d\t%*s:", timestamp, 15, probename); + printf("%*s", self->indent, ""); + printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2); + self->indent++; + } + + python$target:::function-return + /self->trace/ + { + self->indent--; + printf("%d\t%*s:", timestamp, 15, probename); + printf("%*s", self->indent, ""); + printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2); + } + + python$target:::function-return + /copyinstr(arg1) == "start"/ + { + self->trace = 0; + } + +It can be invoked like this: + +.. code-block:: bash + + $ sudo dtrace -q -s call_stack.d -c "python3.6 script.py" + +The output looks like this:: + + 156641360502280 function-entry:call_stack.py:start:23 + 156641360518804 function-entry: call_stack.py:function_1:1 + 156641360532797 function-entry: call_stack.py:function_3:9 + 156641360546807 function-return: call_stack.py:function_3:10 + 156641360563367 function-return: call_stack.py:function_1:2 + 156641360578365 function-entry: call_stack.py:function_2:5 + 156641360591757 function-entry: call_stack.py:function_1:1 + 156641360605556 function-entry: call_stack.py:function_3:9 + 156641360617482 function-return: call_stack.py:function_3:10 + 156641360629814 function-return: call_stack.py:function_1:2 + 156641360642285 function-return: call_stack.py:function_2:6 + 156641360656770 function-entry: call_stack.py:function_3:9 + 156641360669707 function-return: call_stack.py:function_3:10 + 156641360687853 function-entry: call_stack.py:function_4:13 + 156641360700719 function-return: call_stack.py:function_4:14 + 156641360719640 function-entry: call_stack.py:function_5:18 + 156641360732567 function-return: call_stack.py:function_5:21 + 156641360747370 function-return:call_stack.py:start:28 + + +Static SystemTap markers +------------------------ + +The low-level way to use the SystemTap integration is to use the static +markers directly. This requires you to explicitly state the binary file +containing them. + +For example, this SystemTap script can be used to show the call/return +hierarchy of a Python script: + +.. code-block:: c + + probe process("python").mark("function__entry") { + filename = user_string($arg1); + funcname = user_string($arg2); + lineno = $arg3; + + printf("%s => %s in %s:%d\\n", + thread_indent(1), funcname, filename, lineno); + } + + probe process("python").mark("function__return") { + filename = user_string($arg1); + funcname = user_string($arg2); + lineno = $arg3; + + printf("%s <= %s in %s:%d\\n", + thread_indent(-1), funcname, filename, lineno); + } + +It can be invoked like this: + +.. code-block:: bash + + $ stap \ + show-call-hierarchy.stp \ + -c "./python test.py" + +The output looks like this:: + + 11408 python(8274): => __contains__ in Lib/_abcoll.py:362 + 11414 python(8274): => __getitem__ in Lib/os.py:425 + 11418 python(8274): => encode in Lib/os.py:490 + 11424 python(8274): <= encode in Lib/os.py:493 + 11428 python(8274): <= __getitem__ in Lib/os.py:426 + 11433 python(8274): <= __contains__ in Lib/_abcoll.py:366 + +where the columns are: + + - time in microseconds since start of script + + - name of executable + + - PID of process + +and the remainder indicates the call/return hierarchy as the script executes. + +For a `--enable-shared` build of CPython, the markers are contained within the +libpython shared library, and the probe's dotted path needs to reflect this. For +example, this line from the above example:: + + probe process("python").mark("function__entry") { + +should instead read:: + + probe process("python").library("libpython3.6dm.so.1.0").mark("function__entry") { + +(assuming a debug build of CPython 3.6) + + +Available static markers +------------------------ + +.. I'm reusing the "c:function" type for markers + +.. c:function:: function__entry(str filename, str funcname, int lineno) + + This marker indicates that execution of a Python function has begun. + It is only triggered for pure-Python (bytecode) functions. + + The filename, function name, and line number are provided back to the + tracing script as positional arguments, which must be accessed using + ``$arg1``, ``$arg2``, ``$arg3``: + + * ``$arg1`` : ``(const char *)`` filename, accessible using ``user_string($arg1)`` + + * ``$arg2`` : ``(const char *)`` function name, accessible using + ``user_string($arg2)`` + + * ``$arg3`` : ``int`` line number + +.. c:function:: function__return(str filename, str funcname, int lineno) + + This marker is the converse of :c:func:`function__entry`, and indicates that + execution of a Python function has ended (either via ``return``, or via an + exception). It is only triggered for pure-Python (bytecode) functions. + + The arguments are the same as for :c:func:`function__entry` + +.. c:function:: line(str filename, str funcname, int lineno) + + This marker indicates a Python line is about to be executed. It is + the equivalent of line-by-line tracing with a Python profiler. It is + not triggered within C functions. + + The arguments are the same as for :c:func:`function__entry`. + +.. c:function:: gc__start(int generation) + + Fires when the Python interpreter starts a garbage collection cycle. + ``arg0`` is the generation to scan, like :func:`gc.collect()`. + +.. c:function:: gc__done(long collected) + + Fires when the Python interpreter finishes a garbage collection + cycle. ``arg0`` is the number of collected objects. + + +SystemTap Tapsets +----------------- + +The higher-level way to use the SystemTap integration is to use a "tapset": +SystemTap's equivalent of a library, which hides some of the lower-level +details of the static markers. + +Here is a tapset file, based on a non-shared build of CPython: + +.. code-block:: c + + /* + Provide a higher-level wrapping around the function__entry and + function__return markers: + \*/ + probe python.function.entry = process("python").mark("function__entry") + { + filename = user_string($arg1); + funcname = user_string($arg2); + lineno = $arg3; + frameptr = $arg4 + } + probe python.function.return = process("python").mark("function__return") + { + filename = user_string($arg1); + funcname = user_string($arg2); + lineno = $arg3; + frameptr = $arg4 + } + +If this file is installed in SystemTap's tapset directory (e.g. +``/usr/share/systemtap/tapset``), then these additional probepoints become +available: + +.. c:function:: python.function.entry(str filename, str funcname, int lineno, frameptr) + + This probe point indicates that execution of a Python function has begun. + It is only triggered for pure-python (bytecode) functions. + +.. c:function:: python.function.return(str filename, str funcname, int lineno, frameptr) + + This probe point is the converse of :c:func:`python.function.return`, and + indicates that execution of a Python function has ended (either via + ``return``, or via an exception). It is only triggered for pure-python + (bytecode) functions. + + +Examples +-------- +This SystemTap script uses the tapset above to more cleanly implement the +example given above of tracing the Python function-call hierarchy, without +needing to directly name the static markers: + +.. code-block:: c + + probe python.function.entry + { + printf("%s => %s in %s:%d\n", + thread_indent(1), funcname, filename, lineno); + } + + probe python.function.return + { + printf("%s <= %s in %s:%d\n", + thread_indent(-1), funcname, filename, lineno); + } + + +The following script uses the tapset above to provide a top-like view of all +running CPython code, showing the top 20 most frequently-entered bytecode +frames, each second, across the whole system: + +.. code-block:: c + + global fn_calls; + + probe python.function.entry + { + fn_calls[pid(), filename, funcname, lineno] += 1; + } + + probe timer.ms(1000) { + printf("\033[2J\033[1;1H") /* clear screen \*/ + printf("%6s %80s %6s %30s %6s\n", + "PID", "FILENAME", "LINE", "FUNCTION", "CALLS") + foreach ([pid, filename, funcname, lineno] in fn_calls- limit 20) { + printf("%6d %80s %6d %30s %6d\n", + pid, filename, lineno, funcname, + fn_calls[pid, filename, funcname, lineno]); + } + delete fn_calls; + } + diff --git a/Doc/includes/email-alternative-new-api.py b/Doc/includes/email-alternative-new-api.py deleted file mode 100644 index 321f727c31..0000000000 --- a/Doc/includes/email-alternative-new-api.py +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env python3 - -import smtplib - -from email.message import EmailMessage -from email.headerregistry import Address -from email.utils import make_msgid - -# Create the base text message. -msg = EmailMessage() -msg['Subject'] = "Ayons asperges pour le déjeuner" -msg['From'] = Address("Pepé Le Pew", "pepe", "example.com") -msg['To'] = (Address("Penelope Pussycat", "penelope", "example.com"), - Address("Fabrette Pussycat", "fabrette", "example.com")) -msg.set_content("""\ -Salut! - -Cela ressemble à un excellent recipie[1] déjeuner. - -[1] http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718 - ---Pepé -""") - -# Add the html version. This converts the message into a multipart/alternative -# container, with the original text message as the first part and the new html -# message as the second part. -asparagus_cid = make_msgid() -msg.add_alternative("""\ -<html> - <head></head> - <body> - <p>Salut!<\p> - <p>Cela ressemble à un excellent - <a href="http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718> - recipie - </a> déjeuner. - </p> - <img src="cid:{asparagus_cid}" \> - </body> -</html> -""".format(asparagus_cid=asparagus_cid[1:-1]), subtype='html') -# note that we needed to peel the <> off the msgid for use in the html. - -# Now add the related image to the html part. -with open("roasted-asparagus.jpg", 'rb') as img: - msg.get_payload()[1].add_related(img.read(), 'image', 'jpeg', - cid=asparagus_cid) - -# Make a local copy of what we are going to send. -with open('outgoing.msg', 'wb') as f: - f.write(bytes(msg)) - -# Send the message via local SMTP server. -with smtplib.SMTP('localhost') as s: - s.send_message(msg) diff --git a/Doc/includes/email-alternative.py b/Doc/includes/email-alternative.py index 85070f36ae..2e142b1e3b 100755..100644 --- a/Doc/includes/email-alternative.py +++ b/Doc/includes/email-alternative.py @@ -2,47 +2,55 @@ import smtplib -from email.mime.multipart import MIMEMultipart -from email.mime.text import MIMEText - -# me == my email address -# you == recipient's email address -me = "my@email.com" -you = "your@email.com" - -# Create message container - the correct MIME type is multipart/alternative. -msg = MIMEMultipart('alternative') -msg['Subject'] = "Link" -msg['From'] = me -msg['To'] = you - -# Create the body of the message (a plain-text and an HTML version). -text = "Hi!\nHow are you?\nHere is the link you wanted:\nhttps://www.python.org" -html = """\ +from email.message import EmailMessage +from email.headerregistry import Address +from email.utils import make_msgid + +# Create the base text message. +msg = EmailMessage() +msg['Subject'] = "Ayons asperges pour le déjeuner" +msg['From'] = Address("Pepé Le Pew", "pepe", "example.com") +msg['To'] = (Address("Penelope Pussycat", "penelope", "example.com"), + Address("Fabrette Pussycat", "fabrette", "example.com")) +msg.set_content("""\ +Salut! + +Cela ressemble à un excellent recipie[1] déjeuner. + +[1] http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718 + +--Pepé +""") + +# Add the html version. This converts the message into a multipart/alternative +# container, with the original text message as the first part and the new html +# message as the second part. +asparagus_cid = make_msgid() +msg.add_alternative("""\ <html> <head></head> <body> - <p>Hi!<br> - How are you?<br> - Here is the <a href="https://www.python.org">link</a> you wanted. + <p>Salut!</p> + <p>Cela ressemble à un excellent + <a href="http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718> + recipie + </a> déjeuner. </p> + <img src="cid:{asparagus_cid}" /> </body> </html> -""" +""".format(asparagus_cid=asparagus_cid[1:-1]), subtype='html') +# note that we needed to peel the <> off the msgid for use in the html. -# Record the MIME types of both parts - text/plain and text/html. -part1 = MIMEText(text, 'plain') -part2 = MIMEText(html, 'html') +# Now add the related image to the html part. +with open("roasted-asparagus.jpg", 'rb') as img: + msg.get_payload()[1].add_related(img.read(), 'image', 'jpeg', + cid=asparagus_cid) -# Attach parts into message container. -# According to RFC 2046, the last part of a multipart message, in this case -# the HTML message, is best and preferred. -msg.attach(part1) -msg.attach(part2) +# Make a local copy of what we are going to send. +with open('outgoing.msg', 'wb') as f: + f.write(bytes(msg)) # Send the message via local SMTP server. -s = smtplib.SMTP('localhost') -# sendmail function takes 3 arguments: sender's address, recipient's address -# and message to send - here it is sent as one string. -s.sendmail(me, you, msg.as_string()) -s.quit() +with smtplib.SMTP('localhost') as s: + s.send_message(msg) diff --git a/Doc/includes/email-dir.py b/Doc/includes/email-dir.py index 3c7c770dda..0dcfbfb402 100644 --- a/Doc/includes/email-dir.py +++ b/Doc/includes/email-dir.py @@ -3,22 +3,14 @@ """Send the contents of a directory as a MIME message.""" import os -import sys import smtplib # For guessing MIME type based on file name extension import mimetypes from argparse import ArgumentParser -from email import encoders -from email.message import Message -from email.mime.audio import MIMEAudio -from email.mime.base import MIMEBase -from email.mime.image import MIMEImage -from email.mime.multipart import MIMEMultipart -from email.mime.text import MIMEText - -COMMASPACE = ', ' +from email.message import EmailMessage +from email.policy import SMTP def main(): @@ -47,12 +39,12 @@ must be running an SMTP server. directory = args.directory if not directory: directory = '.' - # Create the enclosing (outer) message - outer = MIMEMultipart() - outer['Subject'] = 'Contents of directory %s' % os.path.abspath(directory) - outer['To'] = COMMASPACE.join(args.recipients) - outer['From'] = args.sender - outer.preamble = 'You will not see this in a MIME-aware mail reader.\n' + # Create the message + msg = EmailMessage() + msg['Subject'] = 'Contents of directory %s' % os.path.abspath(directory) + msg['To'] = ', '.join(args.recipients) + msg['From'] = args.sender + msg.preamble = 'You will not see this in a MIME-aware mail reader.\n' for filename in os.listdir(directory): path = os.path.join(directory, filename) @@ -67,33 +59,18 @@ must be running an SMTP server. # use a generic bag-of-bits type. ctype = 'application/octet-stream' maintype, subtype = ctype.split('/', 1) - if maintype == 'text': - with open(path) as fp: - # Note: we should handle calculating the charset - msg = MIMEText(fp.read(), _subtype=subtype) - elif maintype == 'image': - with open(path, 'rb') as fp: - msg = MIMEImage(fp.read(), _subtype=subtype) - elif maintype == 'audio': - with open(path, 'rb') as fp: - msg = MIMEAudio(fp.read(), _subtype=subtype) - else: - with open(path, 'rb') as fp: - msg = MIMEBase(maintype, subtype) - msg.set_payload(fp.read()) - # Encode the payload using Base64 - encoders.encode_base64(msg) - # Set the filename parameter - msg.add_header('Content-Disposition', 'attachment', filename=filename) - outer.attach(msg) + with open(path, 'rb') as fp: + msg.add_attachment(fp.read(), + maintype=maintype, + subtype=subtype, + filename=filename) # Now send or store the message - composed = outer.as_string() if args.output: - with open(args.output, 'w') as fp: - fp.write(composed) + with open(args.output, 'wb') as fp: + fp.write(msg.as_bytes(policy=SMTP)) else: with smtplib.SMTP('localhost') as s: - s.sendmail(args.sender, args.recipients, composed) + s.send_message(msg) if __name__ == '__main__': diff --git a/Doc/includes/email-headers.py b/Doc/includes/email-headers.py index 89c8f3af32..2c421451a8 100644 --- a/Doc/includes/email-headers.py +++ b/Doc/includes/email-headers.py @@ -1,18 +1,24 @@ # Import the email modules we'll need -from email.parser import Parser +from email.parser import BytesParser, Parser +from email.policy import default # If the e-mail headers are in a file, uncomment these two lines: -# with open(messagefile) as fp: -# headers = Parser().parse(fp) +# with open(messagefile, 'rb') as fp: +# headers = BytesParser(policy=default).parse(fp) -# Or for parsing headers in a string, use: -headers = Parser().parsestr('From: <user@example.com>\n' +# Or for parsing headers in a string (this is an uncommon operation), use: +headers = Parser(policy=default).parsestr( + 'From: Foo Bar <user@example.com>\n' 'To: <someone_else@example.com>\n' 'Subject: Test message\n' '\n' 'Body would go here\n') # Now the header items can be accessed as a dictionary: -print('To: %s' % headers['to']) -print('From: %s' % headers['from']) -print('Subject: %s' % headers['subject']) +print('To: {}'.format(headers['to'])) +print('From: {}'.format(headers['from'])) +print('Subject: {}'.format(headers['subject'])) + +# You can also access the parts of the addresses: +print('Recipient username: {}'.format(headers['to'].addresses[0].username)) +print('Sender name: {}'.format(headers['from'].addresses[0].display_name)) diff --git a/Doc/includes/email-mime.py b/Doc/includes/email-mime.py index 61d08302a2..c610242f11 100644 --- a/Doc/includes/email-mime.py +++ b/Doc/includes/email-mime.py @@ -1,30 +1,29 @@ # Import smtplib for the actual sending function import smtplib -# Here are the email package modules we'll need -from email.mime.image import MIMEImage -from email.mime.multipart import MIMEMultipart +# And imghdr to find the types of our images +import imghdr -COMMASPACE = ', ' +# Here are the email package modules we'll need +from email.message import EmailMessage -# Create the container (outer) email message. -msg = MIMEMultipart() +# Create the container email message. +msg = EmailMessage() msg['Subject'] = 'Our family reunion' # me == the sender's email address # family = the list of all recipients' email addresses msg['From'] = me -msg['To'] = COMMASPACE.join(family) +msg['To'] = ', '.join(family) msg.preamble = 'Our family reunion' -# Assume we know that the image files are all in PNG format +# Open the files in binary mode. Use imghdr to figure out the +# MIME subtype for each specific image. for file in pngfiles: - # Open the files in binary mode. Let the MIMEImage class automatically - # guess the specific image type. with open(file, 'rb') as fp: - img = MIMEImage(fp.read()) - msg.attach(img) + img_data = fp.read() + msg.add_attachment(img_data, maintype='image', + subtype=imghdr.what(None, img_data)) # Send the email via our own SMTP server. -s = smtplib.SMTP('localhost') -s.send_message(msg) -s.quit() +with smtplib.SMTP('localhost') as s: + s.send_message(msg) diff --git a/Doc/includes/email-read-alternative-new-api.py b/Doc/includes/email-read-alternative.py index 3f5ab24c0f..3f5ab24c0f 100644 --- a/Doc/includes/email-read-alternative-new-api.py +++ b/Doc/includes/email-read-alternative.py diff --git a/Doc/includes/email-simple.py b/Doc/includes/email-simple.py index b9b8b410a6..f69ef40ff0 100644 --- a/Doc/includes/email-simple.py +++ b/Doc/includes/email-simple.py @@ -2,13 +2,13 @@ import smtplib # Import the email modules we'll need -from email.mime.text import MIMEText +from email.message import EmailMessage -# Open a plain text file for reading. For this example, assume that -# the text file contains only ASCII characters. +# Open the plain text file whose name is in textfile for reading. with open(textfile) as fp: # Create a text/plain message - msg = MIMEText(fp.read()) + msg = EmailMessage() + msg.set_content(fp.read()) # me == the sender's email address # you == the recipient's email address diff --git a/Doc/includes/email-unpack.py b/Doc/includes/email-unpack.py index 574a0b6d72..e0a7f01f58 100644 --- a/Doc/includes/email-unpack.py +++ b/Doc/includes/email-unpack.py @@ -3,11 +3,11 @@ """Unpack a MIME message into a directory of files.""" import os -import sys import email -import errno import mimetypes +from email.policy import default + from argparse import ArgumentParser @@ -22,8 +22,8 @@ Unpack a MIME message into a directory of files. parser.add_argument('msgfile') args = parser.parse_args() - with open(args.msgfile) as fp: - msg = email.message_from_file(fp) + with open(args.msgfile, 'rb') as fp: + msg = email.message_from_binary_file(fp, policy=default) try: os.mkdir(args.directory) diff --git a/Doc/includes/run-func.c b/Doc/includes/run-func.c index 986d670319..ead7bdd232 100644 --- a/Doc/includes/run-func.c +++ b/Doc/includes/run-func.c @@ -63,6 +63,8 @@ main(int argc, char *argv[]) fprintf(stderr, "Failed to load \"%s\"\n", argv[1]); return 1; } - Py_Finalize(); + if (Py_FinalizeEx() < 0) { + return 120; + } return 0; } diff --git a/Doc/includes/test.py b/Doc/includes/test.py index 7ebf46afd1..9e9d4a6712 100644 --- a/Doc/includes/test.py +++ b/Doc/includes/test.py @@ -204,7 +204,7 @@ Test cyclic gc(?) import os import sys from distutils.util import get_platform -PLAT_SPEC = "%s-%s" % (get_platform(), sys.version[0:3]) +PLAT_SPEC = "%s-%d.%d" % (get_platform(), *sys.version_info[:2]) src = os.path.join("build", "lib.%s" % PLAT_SPEC) sys.path.append(src) diff --git a/Doc/includes/tzinfo-examples.py b/Doc/includes/tzinfo-examples.py index 3a8cf47eaf..ae5a509266 100644 --- a/Doc/includes/tzinfo-examples.py +++ b/Doc/includes/tzinfo-examples.py @@ -1,46 +1,13 @@ -from datetime import tzinfo, timedelta, datetime +from datetime import tzinfo, timedelta, datetime, timezone ZERO = timedelta(0) HOUR = timedelta(hours=1) - -# A UTC class. - -class UTC(tzinfo): - """UTC""" - - def utcoffset(self, dt): - return ZERO - - def tzname(self, dt): - return "UTC" - - def dst(self, dt): - return ZERO - -utc = UTC() - -# A class building tzinfo objects for fixed-offset time zones. -# Note that FixedOffset(0, "UTC") is a different way to build a -# UTC tzinfo object. - -class FixedOffset(tzinfo): - """Fixed offset in minutes east from UTC.""" - - def __init__(self, offset, name): - self.__offset = timedelta(minutes=offset) - self.__name = name - - def utcoffset(self, dt): - return self.__offset - - def tzname(self, dt): - return self.__name - - def dst(self, dt): - return ZERO +SECOND = timedelta(seconds=1) # A class capturing the platform's idea of local time. - +# (May result in wrong values on historical times in +# timezones where UTC offset and/or the DST rules had +# changed in the past.) import time as _time STDOFFSET = timedelta(seconds = -_time.timezone) @@ -53,6 +20,16 @@ DSTDIFF = DSTOFFSET - STDOFFSET class LocalTimezone(tzinfo): + def fromutc(self, dt): + assert dt.tzinfo is self + stamp = (dt - datetime(1970, 1, 1, tzinfo=self)) // SECOND + args = _time.localtime(stamp)[:6] + dst_diff = DSTDIFF // SECOND + # Detect fold + fold = (args == _time.localtime(stamp - dst_diff)) + return datetime(*args, microsecond=dt.microsecond, + tzinfo=self, fold=fold) + def utcoffset(self, dt): if self._isdst(dt): return DSTOFFSET @@ -99,20 +76,37 @@ def first_sunday_on_or_after(dt): # In the US, since 2007, DST starts at 2am (standard time) on the second # Sunday in March, which is the first Sunday on or after Mar 8. DSTSTART_2007 = datetime(1, 3, 8, 2) -# and ends at 2am (DST time; 1am standard time) on the first Sunday of Nov. -DSTEND_2007 = datetime(1, 11, 1, 1) +# and ends at 2am (DST time) on the first Sunday of Nov. +DSTEND_2007 = datetime(1, 11, 1, 2) # From 1987 to 2006, DST used to start at 2am (standard time) on the first -# Sunday in April and to end at 2am (DST time; 1am standard time) on the last +# Sunday in April and to end at 2am (DST time) on the last # Sunday of October, which is the first Sunday on or after Oct 25. DSTSTART_1987_2006 = datetime(1, 4, 1, 2) -DSTEND_1987_2006 = datetime(1, 10, 25, 1) +DSTEND_1987_2006 = datetime(1, 10, 25, 2) # From 1967 to 1986, DST used to start at 2am (standard time) on the last -# Sunday in April (the one on or after April 24) and to end at 2am (DST time; -# 1am standard time) on the last Sunday of October, which is the first Sunday +# Sunday in April (the one on or after April 24) and to end at 2am (DST time) +# on the last Sunday of October, which is the first Sunday # on or after Oct 25. DSTSTART_1967_1986 = datetime(1, 4, 24, 2) DSTEND_1967_1986 = DSTEND_1987_2006 +def us_dst_range(year): + # Find start and end times for US DST. For years before 1967, return + # start = end for no DST. + if 2006 < year: + dststart, dstend = DSTSTART_2007, DSTEND_2007 + elif 1986 < year < 2007: + dststart, dstend = DSTSTART_1987_2006, DSTEND_1987_2006 + elif 1966 < year < 1987: + dststart, dstend = DSTSTART_1967_1986, DSTEND_1967_1986 + else: + return (datetime(year, 1, 1), ) * 2 + + start = first_sunday_on_or_after(dststart.replace(year=year)) + end = first_sunday_on_or_after(dstend.replace(year=year)) + return start, end + + class USTimeZone(tzinfo): def __init__(self, hours, reprname, stdname, dstname): @@ -141,27 +135,39 @@ class USTimeZone(tzinfo): # implementation) passes a datetime with dt.tzinfo is self. return ZERO assert dt.tzinfo is self - - # Find start and end times for US DST. For years before 1967, return - # ZERO for no DST. - if 2006 < dt.year: - dststart, dstend = DSTSTART_2007, DSTEND_2007 - elif 1986 < dt.year < 2007: - dststart, dstend = DSTSTART_1987_2006, DSTEND_1987_2006 - elif 1966 < dt.year < 1987: - dststart, dstend = DSTSTART_1967_1986, DSTEND_1967_1986 - else: - return ZERO - - start = first_sunday_on_or_after(dststart.replace(year=dt.year)) - end = first_sunday_on_or_after(dstend.replace(year=dt.year)) - + start, end = us_dst_range(dt.year) # Can't compare naive to aware objects, so strip the timezone from # dt first. - if start <= dt.replace(tzinfo=None) < end: + dt = dt.replace(tzinfo=None) + if start + HOUR <= dt < end - HOUR: + # DST is in effect. return HOUR - else: - return ZERO + if end - HOUR <= dt < end: + # Fold (an ambiguous hour): use dt.fold to disambiguate. + return ZERO if dt.fold else HOUR + if start <= dt < start + HOUR: + # Gap (a non-existent hour): reverse the fold rule. + return HOUR if dt.fold else ZERO + # DST is off. + return ZERO + + def fromutc(self, dt): + assert dt.tzinfo is self + start, end = us_dst_range(dt.year) + start = start.replace(tzinfo=self) + end = end.replace(tzinfo=self) + std_time = dt + self.stdoffset + dst_time = std_time + HOUR + if end <= dst_time < end + HOUR: + # Repeated hour + return std_time.replace(fold=1) + if std_time < start or dst_time >= end: + # Standard time + return std_time + if start <= std_time < end - HOUR: + # Daylight saving time + return dst_time + Eastern = USTimeZone(-5, "Eastern", "EST", "EDT") Central = USTimeZone(-6, "Central", "CST", "CDT") diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst index 1ef314999b..b22465df29 100644 --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -2,9 +2,9 @@ .. _installing-index: -***************************** - Installing Python Modules -***************************** +************************* +Installing Python Modules +************************* :Email: distutils-sig@python.org @@ -34,24 +34,24 @@ Key terms * ``pip`` is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers. -* a virtual environment is a semi-isolated Python environment that allows +* A *virtual environment* is a semi-isolated Python environment that allows packages to be installed for use by a particular application, rather than - being installed system wide -* ``pyvenv`` is the standard tool for creating virtual environments, and has + being installed system wide. +* ``venv`` is the standard tool for creating virtual environments, and has been part of Python since Python 3.3. Starting with Python 3.4, it - defaults to installing ``pip`` into all created virtual environments + defaults to installing ``pip`` into all created virtual environments. * ``virtualenv`` is a third party alternative (and predecessor) to - ``pyvenv``. It allows virtual environments to be used on versions of - Python prior to 3.4, which either don't provide ``pyvenv`` at all, or + ``venv``. It allows virtual environments to be used on versions of + Python prior to 3.4, which either don't provide ``venv`` at all, or aren't able to automatically install ``pip`` into created environments. -* the `Python Packaging Index <https://pypi.python.org/pypi>`__ is a public +* The `Python Packaging Index <https://pypi.python.org/pypi>`__ is a public repository of open source licensed packages made available for use by - other Python users + other Python users. * the `Python Packaging Authority <https://www.pypa.io/en/latest/>`__ are the group of developers and documentation authors responsible for the maintenance and evolution of the standard packaging tools and the associated metadata and - file format standards. They maintain a variety of tools, documentation + file format standards. They maintain a variety of tools, documentation, and issue trackers on both `GitHub <https://github.com/pypa>`__ and `BitBucket <https://bitbucket.org/pypa/>`__. * ``distutils`` is the original build and distribution system first added to @@ -62,6 +62,19 @@ Key terms of the mailing list used to coordinate Python packaging standards development). +.. deprecated:: 3.6 + ``pyvenv`` was the recommended tool for creating virtual environments for + Python 3.3 and 3.4, and is `deprecated in Python 3.6 + <https://docs.python.org/dev/whatsnew/3.6.html#deprecated-features>`_. + +.. versionchanged:: 3.5 + The use of ``venv`` is now recommended for creating virtual environments. + +.. seealso:: + + `Python Packaging User Guide: Creating and using virtual environments + <https://packaging.python.org/installing/#creating-virtual-environments>`__ + Basic usage =========== @@ -100,13 +113,14 @@ explicitly:: More information and resources regarding ``pip`` and its capabilities can be found in the `Python Packaging User Guide <https://packaging.python.org>`__. -``pyvenv`` has its own documentation at :ref:`scripts-pyvenv`. Installing -into an active virtual environment uses the commands shown above. +Creation of virtual environments is done through the :mod:`venv` module. +Installing packages into an active virtual environment uses the commands shown +above. .. seealso:: `Python Packaging User Guide: Installing Python Distribution Packages - <https://packaging.python.org/en/latest/installing/>`__ + <https://packaging.python.org/installing/>`__ How do I ...? @@ -124,7 +138,7 @@ User Guide. .. seealso:: `Python Packaging User Guide: Requirements for Installing Packages - <https://packaging.python.org/en/latest/installing/#requirements-for-installing-packages>`__ + <https://packaging.python.org/installing/#requirements-for-installing-packages>`__ .. installing-per-user-installation: @@ -142,20 +156,19 @@ package just for the current user, rather than for all users of the system. A number of scientific Python packages have complex binary dependencies, and aren't currently easy to install using ``pip`` directly. At this point in time, it will often be easier for users to install these packages by -`other means -<https://packaging.python.org/en/latest/science/>`__ +`other means <https://packaging.python.org/science/>`__ rather than attempting to install them with ``pip``. .. seealso:: `Python Packaging User Guide: Installing Scientific Packages - <https://packaging.python.org/en/latest/science/>`__ + <https://packaging.python.org/science/>`__ ... work with multiple versions of Python installed in parallel? ---------------------------------------------------------------- -On Linux, Mac OS X and other POSIX systems, use the versioned Python commands +On Linux, Mac OS X, and other POSIX systems, use the versioned Python commands in combination with the ``-m`` switch to run the appropriate copy of ``pip``:: @@ -164,7 +177,7 @@ in combination with the ``-m`` switch to run the appropriate copy of python3 -m pip install SomePackage # default Python 3 python3.4 -m pip install SomePackage # specifically Python 3.4 -(appropriately versioned ``pip`` commands may also be available) +Appropriately versioned ``pip`` commands may also be available. On Windows, use the ``py`` Python launcher in combination with the ``-m`` switch:: @@ -212,11 +225,11 @@ as users are more regularly able to install pre-built extensions rather than needing to build them themselves. Some of the solutions for installing `scientific software -<https://packaging.python.org/en/latest/science/>`__ -that is not yet available as pre-built ``wheel`` files may also help with +<https://packaging.python.org/science/>`__ +that are not yet available as pre-built ``wheel`` files may also help with obtaining other binary extensions without needing to build them locally. .. seealso:: `Python Packaging User Guide: Binary Extensions - <https://packaging.python.org/en/latest/extensions/>`__ + <https://packaging.python.org/extensions/>`__ diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index ae72d26122..e02360c883 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -58,13 +58,13 @@ connection requests. The asynchronous output buffer size (default ``4096``). Unlike :class:`asyncore.dispatcher`, :class:`async_chat` allows you to - define a first-in-first-out queue (fifo) of *producers*. A producer need + define a :abbr:`FIFO (first-in, first-out)` queue of *producers*. A producer need have only one method, :meth:`more`, which should return data to be transmitted on the channel. The producer indicates exhaustion (*i.e.* that it contains no more data) by having its :meth:`more` method return the empty bytes object. At this point - the :class:`async_chat` object removes the producer from the fifo and starts - using the next producer, if any. When the producer fifo is empty the + the :class:`async_chat` object removes the producer from the queue and starts + using the next producer, if any. When the producer queue is empty the :meth:`handle_write` method does nothing. You use the channel object's :meth:`set_terminator` method to describe how to recognize the end of, or an important breakpoint in, an incoming transmission from the remote @@ -78,8 +78,8 @@ connection requests. .. method:: async_chat.close_when_done() - Pushes a ``None`` on to the producer fifo. When this producer is popped off - the fifo it causes the channel to be closed. + Pushes a ``None`` on to the producer queue. When this producer is popped off + the queue it causes the channel to be closed. .. method:: async_chat.collect_incoming_data(data) @@ -92,7 +92,7 @@ connection requests. .. method:: async_chat.discard_buffers() In emergencies this method will discard any data held in the input and/or - output buffers and the producer fifo. + output buffers and the producer queue. .. method:: async_chat.found_terminator() @@ -110,7 +110,7 @@ connection requests. .. method:: async_chat.push(data) - Pushes data on to the channel's fifo to ensure its transmission. + Pushes data on to the channel's queue to ensure its transmission. This is all you need to do to have the channel write the data out to the network, although it is possible to use your own producers in more complex schemes to implement encryption and chunking, for example. @@ -118,7 +118,7 @@ connection requests. .. method:: async_chat.push_with_producer(producer) - Takes a producer object and adds it to the producer fifo associated with + Takes a producer object and adds it to the producer queue associated with the channel. When all currently-pushed producers have been exhausted the channel will consume this producer's data by calling its :meth:`more` method and send the data to the remote endpoint. diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index ba52d9bed3..ed1d773578 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -110,8 +110,9 @@ keywords to your callback, use :func:`functools.partial`. For example, called after :meth:`call_soon` returns, when control returns to the event loop. - This operates as a FIFO queue, callbacks are called in the order in - which they are registered. Each callback will be called exactly once. + This operates as a :abbr:`FIFO (first-in, first-out)` queue, callbacks + are called in the order in which they are registered. Each callback + will be called exactly once. Any positional arguments after the callback will be passed to the callback when it is called. diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst index 878d8db53c..4f5f0f26ad 100644 --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -53,13 +53,14 @@ The :mod:`binascii` module defines the following functions: than one line may be passed at a time. -.. function:: b2a_base64(data) +.. function:: b2a_base64(data, \*, newline=True) Convert binary data to a line of ASCII characters in base64 coding. The return - value is the converted line, including a newline char. The newline is - added because the original use case for this function was to feed it a - series of 57 byte input lines to get output lines that conform to the - MIME-base64 standard. Otherwise the output conforms to :rfc:`3548`. + value is the converted line, including a newline char if *newline* is + true. The output of this function conforms to :rfc:`3548`. + + .. versionchanged:: 3.6 + Added the *newline* parameter. .. function:: a2b_qp(data, header=False) diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 6c49d9fffe..d5f622515a 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -61,6 +61,9 @@ All of the classes in this module may safely be accessed from multiple threads. .. versionchanged:: 3.4 The ``'x'`` (exclusive creation) mode was added. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. class:: BZ2File(filename, mode='r', buffering=None, compresslevel=9) @@ -128,6 +131,9 @@ All of the classes in this module may safely be accessed from multiple threads. The :meth:`~io.BufferedIOBase.read` method now accepts an argument of ``None``. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + Incremental (de)compression --------------------------- diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst index 62ddb6bc48..d935c41d0b 100644 --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -253,6 +253,40 @@ Constants The mathematical constant *e*, as a float. +.. data:: tau + + The mathematical constant *τ*, as a float. + + .. versionadded:: 3.6 + +.. data:: inf + + Floating-point positive infinity. Equivalent to ``float('inf')``. + + .. versionadded:: 3.6 + +.. data:: infj + + Complex number with zero real part and positive infinity imaginary + part. Equivalent to ``complex(0.0, float('inf'))``. + + .. versionadded:: 3.6 + +.. data:: nan + + A floating-point "not a number" (NaN) value. Equivalent to + ``float('nan')``. + + .. versionadded:: 3.6 + +.. data:: nanj + + Complex number with zero real part and NaN imaginary part. Equivalent to + ``complex(0.0, float('nan'))``. + + .. versionadded:: 3.6 + + .. index:: module: math Note that the selection of functions is similar, but not identical, to that in @@ -276,5 +310,3 @@ cuts for numerical purposes, a good reference should be the following: Kahan, W: Branch cuts for complex elementary functions; or, Much ado about nothing's sign bit. In Iserles, A., and Powell, M. (eds.), The state of the art in numerical analysis. Clarendon Press (1987) pp165-211. - - diff --git a/Doc/library/code.rst b/Doc/library/code.rst index 443af699f2..e2c47bab5a 100644 --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -30,15 +30,19 @@ build applications which provide an interactive interpreter prompt. ``sys.ps1`` and ``sys.ps2``, and input buffering. -.. function:: interact(banner=None, readfunc=None, local=None) +.. function:: interact(banner=None, readfunc=None, local=None, exitmsg=None) Convenience function to run a read-eval-print loop. This creates a new instance of :class:`InteractiveConsole` and sets *readfunc* to be used as the :meth:`InteractiveConsole.raw_input` method, if provided. If *local* is provided, it is passed to the :class:`InteractiveConsole` constructor for use as the default namespace for the interpreter loop. The :meth:`interact` - method of the instance is then run with *banner* passed as the banner to - use, if provided. The console object is discarded after use. + method of the instance is then run with *banner* and *exitmsg* passed as the + banner and exit message to use, if provided. The console object is discarded + after use. + + .. versionchanged:: 3.6 + Added *exitmsg* parameter. .. function:: compile_command(source, filename="<input>", symbol="single") @@ -136,7 +140,7 @@ The :class:`InteractiveConsole` class is a subclass of interpreter objects as well as the following additions. -.. method:: InteractiveConsole.interact(banner=None) +.. method:: InteractiveConsole.interact(banner=None, exitmsg=None) Closely emulate the interactive Python console. The optional *banner* argument specify the banner to print before the first interaction; by default it prints a @@ -144,9 +148,16 @@ interpreter objects as well as the following additions. by the class name of the console object in parentheses (so as not to confuse this with the real interpreter -- since it's so close!). + The optional *exitmsg* argument specifies an exit message printed when exiting. + Pass the empty string to suppress the exit message. If *exitmsg* is not given or + ``None``, a default message is printed. + .. versionchanged:: 3.4 To suppress printing any banner, pass an empty string. + .. versionchanged:: 3.6 + Print an exit message when exiting. + .. method:: InteractiveConsole.push(line) diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 03f0228476..aa1d123758 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -1263,10 +1263,16 @@ encodings. | | | Only ``errors='strict'`` | | | | is supported. | +--------------------+---------+---------------------------+ -| mbcs | dbcs | Windows only: Encode | -| | | operand according to the | +| mbcs | ansi, | Windows only: Encode | +| | dbcs | operand according to the | | | | ANSI codepage (CP_ACP) | +--------------------+---------+---------------------------+ +| oem | | Windows only: Encode | +| | | operand according to the | +| | | OEM codepage (CP_OEMCP) | +| | | | +| | | .. versionadded:: 3.6 | ++--------------------+---------+---------------------------+ | palmos | | Encoding of PalmOS 3.5 | +--------------------+---------+---------------------------+ | punycode | | Implements :rfc:`3492`. | diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index aeb6a73f2e..e4b75a0f0a 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -41,13 +41,16 @@ ABC Inherits from Abstract Methods Mixin :class:`Hashable` ``__hash__`` :class:`Iterable` ``__iter__`` :class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__`` +:class:`Reversible` :class:`Iterable` ``__reversed__`` :class:`Generator` :class:`Iterator` ``send``, ``throw`` ``close``, ``__iter__``, ``__next__`` :class:`Sized` ``__len__`` :class:`Callable` ``__call__`` +:class:`Collection` :class:`Sized`, ``__contains__``, + :class:`Iterable`, ``__iter__``, + :class:`Container` ``__len__`` -:class:`Sequence` :class:`Sized`, ``__getitem__``, ``__contains__``, ``__iter__``, ``__reversed__``, - :class:`Iterable`, ``__len__`` ``index``, and ``count`` - :class:`Container` +:class:`Sequence` :class:`Reversible`, ``__getitem__``, ``__contains__``, ``__iter__``, ``__reversed__``, + :class:`Collection` ``__len__`` ``index``, and ``count`` :class:`MutableSequence` :class:`Sequence` ``__getitem__``, Inherited :class:`Sequence` methods and ``__setitem__``, ``append``, ``reverse``, ``extend``, ``pop``, @@ -58,9 +61,9 @@ ABC Inherits from Abstract Methods Mixin :class:`ByteString` :class:`Sequence` ``__getitem__``, Inherited :class:`Sequence` methods ``__len__`` -:class:`Set` :class:`Sized`, ``__contains__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, - :class:`Iterable`, ``__iter__``, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, - :class:`Container` ``__len__`` ``__sub__``, ``__xor__``, and ``isdisjoint`` +:class:`Set` :class:`Collection` ``__contains__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, + ``__iter__``, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, + ``__len__`` ``__sub__``, ``__xor__``, and ``isdisjoint`` :class:`MutableSet` :class:`Set` ``__contains__``, Inherited :class:`Set` methods and ``__iter__``, ``clear``, ``pop``, ``remove``, ``__ior__``, @@ -68,9 +71,9 @@ ABC Inherits from Abstract Methods Mixin ``add``, ``discard`` -:class:`Mapping` :class:`Sized`, ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``, - :class:`Iterable`, ``__iter__``, ``get``, ``__eq__``, and ``__ne__`` - :class:`Container` ``__len__`` +:class:`Mapping` :class:`Collection` ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``, + ``__iter__``, ``get``, ``__eq__``, and ``__ne__`` + ``__len__`` :class:`MutableMapping` :class:`Mapping` ``__getitem__``, Inherited :class:`Mapping` methods and ``__setitem__``, ``pop``, ``popitem``, ``clear``, ``update``, @@ -105,12 +108,25 @@ ABC Inherits from Abstract Methods Mixin ABC for classes that provide the :meth:`__iter__` method. See also the definition of :term:`iterable`. +.. class:: Collection + + ABC for sized iterable container classes. + + .. versionadded:: 3.6 + .. class:: Iterator ABC for classes that provide the :meth:`~iterator.__iter__` and :meth:`~iterator.__next__` methods. See also the definition of :term:`iterator`. +.. class:: Reversible + + ABC for iterable classes that also provide the :meth:`__reversed__` + method. + + .. versionadded:: 3.6 + .. class:: Generator ABC for generator classes that implement the protocol defined in diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index d0aa62e368..9f3f5dc8ff 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -763,7 +763,7 @@ Named tuples assign meaning to each position in a tuple and allow for more reada self-documenting code. They can be used wherever regular tuples are used, and they add the ability to access fields by name instead of position index. -.. function:: namedtuple(typename, field_names, verbose=False, rename=False) +.. function:: namedtuple(typename, field_names, *, verbose=False, rename=False, module=None) Returns a new tuple subclass named *typename*. The new subclass is used to create tuple-like objects that have fields accessible by attribute lookup as @@ -790,12 +790,21 @@ they add the ability to access fields by name instead of position index. built. This option is outdated; instead, it is simpler to print the :attr:`_source` attribute. + If *module* is defined, the ``__module__`` attribute of the named tuple is + set to that value. + Named tuple instances do not have per-instance dictionaries, so they are lightweight and require no more memory than regular tuples. .. versionchanged:: 3.1 - Added support for *rename*. + Added support for *rename*. + + .. versionchanged:: 3.6 + The *verbose* and *rename* parameters became + :ref:`keyword-only arguments <keyword-only_parameter>`. + .. versionchanged:: 3.6 + Added the *module* parameter. .. doctest:: :options: +NORMALIZE_WHITESPACE @@ -846,7 +855,9 @@ field names, the method and attribute names start with an underscore. .. method:: somenamedtuple._asdict() Return a new :class:`OrderedDict` which maps field names to their corresponding - values:: + values: + + .. doctest:: >>> p = Point(x=11, y=22) >>> p._asdict() @@ -908,7 +919,9 @@ Since a named tuple is a regular Python class, it is easy to add or change functionality with a subclass. Here is how to add a calculated field and a fixed-width print format: - >>> class Point(namedtuple('Point', 'x y')): +.. doctest:: + + >>> class Point(namedtuple('Point', ['x', 'y'])): ... __slots__ = () ... @property ... def hypot(self): @@ -959,6 +972,11 @@ customize a prototype instance: constructor that is convenient for use cases where named tuples are being subclassed. + * See :meth:`types.SimpleNamespace` for a mutable namespace based on an + underlying dictionary instead of a tuple. + + * See :meth:`typing.NamedTuple` for a way to add type hints for named tuples. + :class:`OrderedDict` objects ---------------------------- @@ -980,8 +998,9 @@ the items are returned in the order their keys were first added. .. method:: popitem(last=True) The :meth:`popitem` method for ordered dictionaries returns and removes a - (key, value) pair. The pairs are returned in LIFO order if *last* is true - or FIFO order if false. + (key, value) pair. The pairs are returned in + :abbr:`LIFO (last-in, first-out)` order if *last* is true + or :abbr:`FIFO (first-in, first-out)` order if false. .. method:: move_to_end(key, last=True) diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 511c581ad0..c1af02b0d8 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -102,7 +102,8 @@ Public functions .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` - files along the way. + files along the way. Return a true value if all the files compiled successfully, + and a false value otherwise. The *maxlevels* parameter is used to limit the depth of the recursion; it defaults to ``10``. @@ -152,9 +153,13 @@ Public functions The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files no matter what the value of *optimize* is. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1) - Compile the file with path *fullname*. + Compile the file with path *fullname*. Return a true value if the file + compiled successfully, and a false value otherwise. If *ddir* is given, it is prepended to the path to the file being compiled for use in compilation time tracebacks, and is also compiled in to the @@ -190,8 +195,10 @@ Public functions .. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1) - Byte-compile all the :file:`.py` files found along ``sys.path``. If - *skip_curdir* is true (the default), the current directory is not included + Byte-compile all the :file:`.py` files found along ``sys.path``. Return a + true value if all the files compiled successfully, and a false value otherwise. + + If *skip_curdir* is true (the default), the current directory is not included in the search. All other parameters are passed to the :func:`compile_dir` function. Note that unlike the other compile functions, ``maxlevels`` defaults to ``0``. @@ -217,6 +224,9 @@ subdirectory and all its subdirectories:: import re compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True) + # pathlib.Path objects can also be used. + import pathlib + compileall.compile_dir(pathlib.Path('Lib/'), force=True) .. seealso:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index ae03f4b8f4..d85576b8be 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -124,7 +124,7 @@ And:: executor.submit(wait_on_future) -.. class:: ThreadPoolExecutor(max_workers=None) +.. class:: ThreadPoolExecutor(max_workers=None, thread_name_prefix='') An :class:`Executor` subclass that uses a pool of at most *max_workers* threads to execute calls asynchronously. @@ -137,6 +137,10 @@ And:: should be higher than the number of workers for :class:`ProcessPoolExecutor`. + .. versionadded:: 3.6 + The *thread_name_prefix* argument was added to allow users to + control the threading.Thread names for worker threads created by + the pool for easier debugging. .. _threadpoolexecutor-example: diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index d5a0f09173..f0742cee55 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -33,16 +33,22 @@ A small number of constants live in the built-in namespace. They are: (e.g. :meth:`__imul__`, :meth:`__iand__`, etc.) for the same purpose. Its truth value is true. -.. note:: + .. note:: + + When a binary (or in-place) method returns ``NotImplemented`` the + interpreter will try the reflected operation on the other type (or some + other fallback, depending on the operator). If all attempts return + ``NotImplemented``, the interpreter will raise an appropriate exception. + Incorrectly returning ``NotImplemented`` will result in a misleading + error message or the ``NotImplemented`` value being returned to Python code. + + See :ref:`implementing-the-arithmetic-operations` for examples. - When ``NotImplemented`` is returned, the interpreter will then try the - reflected operation on the other type, or some other fallback, depending - on the operator. If all attempted operations return ``NotImplemented``, the - interpreter will raise an appropriate exception. + .. note:: - See - :ref:`implementing-the-arithmetic-operations` - for more details. + ``NotImplentedError`` and ``NotImplemented`` are not interchangeable, + even though they have similar names and purposes. + See :exc:`NotImplementedError` for details on when to use it. .. data:: Ellipsis diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index cf85fcd4b2..dd34c96c8f 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -18,6 +18,18 @@ Utilities Functions and classes provided: +.. class:: AbstractContextManager + + An :term:`abstract base class` for classes that implement + :meth:`object.__enter__` and :meth:`object.__exit__`. A default + implementation for :meth:`object.__enter__` is provided which returns + ``self`` while :meth:`object.__exit__` is an abstract method which by default + returns ``None``. See also the definition of :ref:`typecontextmanager`. + + .. versionadded:: 3.6 + + + .. decorator:: contextmanager This function is a :term:`decorator` that can be used to define a factory @@ -447,9 +459,9 @@ Here's an example of doing this for a context manager that accepts resource acquisition and release functions, along with an optional validation function, and maps them to the context management protocol:: - from contextlib import contextmanager, ExitStack + from contextlib import contextmanager, AbstractContextManager, ExitStack - class ResourceManager: + class ResourceManager(AbstractContextManager): def __init__(self, acquire_resource, release_resource, check_resource_ok=None): self.acquire_resource = acquire_resource @@ -578,10 +590,10 @@ single definition:: self.name = name def __enter__(self): - logging.info('Entering: {}'.format(self.name)) + logging.info('Entering: %s', self.name) def __exit__(self, exc_type, exc, exc_tb): - logging.info('Exiting: {}'.format(self.name)) + logging.info('Exiting: %s', self.name) Instances of this class can be used as both a context manager:: diff --git a/Doc/library/crypt.rst b/Doc/library/crypt.rst index a21c1e7a75..dbd4274038 100644 --- a/Doc/library/crypt.rst +++ b/Doc/library/crypt.rst @@ -68,7 +68,7 @@ Module Attributes A list of available password hashing algorithms, as ``crypt.METHOD_*`` objects. This list is sorted from strongest to - weakest, and is guaranteed to have at least ``crypt.METHOD_CRYPT``. + weakest. Module Functions diff --git a/Doc/library/crypto.rst b/Doc/library/crypto.rst index 1eddfdc12c..8eb4b81d00 100644 --- a/Doc/library/crypto.rst +++ b/Doc/library/crypto.rst @@ -15,4 +15,6 @@ Here's an overview: .. toctree:: hashlib.rst + hashlib-blake2.rst hmac.rst + secrets.rst diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 7fb4fc8256..f916572fd5 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -149,18 +149,25 @@ The :mod:`csv` module defines the following classes: .. class:: DictReader(csvfile, fieldnames=None, restkey=None, restval=None, \ dialect='excel', *args, **kwds) - Create an object which operates like a regular reader but maps the - information read into a dict whose keys are given by the optional - *fieldnames* parameter. The *fieldnames* parameter is a :mod:`sequence - <collections.abc>` whose elements are associated with the fields of the - input data in order. These elements become the keys of the resulting - dictionary. If the *fieldnames* parameter is omitted, the values in the - first row of the *csvfile* will be used as the fieldnames. If the row read - has more fields than the fieldnames sequence, the remaining data is added as - a sequence keyed by the value of *restkey*. If the row read has fewer - fields than the fieldnames sequence, the remaining keys take the value of - the optional *restval* parameter. Any other optional or keyword arguments - are passed to the underlying :class:`reader` instance. + Create an object that operates like a regular reader but maps the + information in each row to an :mod:`OrderedDict <collections.OrderedDict>` + whose keys are given by the optional *fieldnames* parameter. + + The *fieldnames* parameter is a :term:`sequence`. If *fieldnames* is + omitted, the values in the first row of the *csvfile* will be used as the + fieldnames. Regardless of how the fieldnames are determined, the ordered + dictionary preserves their original ordering. + + If a row has more fields than fieldnames, the remaining data is put in a + list and stored with the fieldname specified by *restkey* (which defaults + to ``None``). If a non-blank row has fewer fields than fieldnames, the + missing values are filled-in with ``None``. + + All other optional or keyword arguments are passed to the underlying + :class:`reader` instance. + + .. versionchanged:: 3.6 + Returned rows are now of type :class:`OrderedDict`. A short usage example:: @@ -170,9 +177,11 @@ The :mod:`csv` module defines the following classes: ... for row in reader: ... print(row['first_name'], row['last_name']) ... - Baked Beans - Lovely Spam - Wonderful Spam + Eric Idle + John Cleese + + >>> print(row) + OrderedDict([('first_name', 'John'), ('last_name', 'Cleese')]) .. class:: DictWriter(csvfile, fieldnames, restval='', extrasaction='raise', \ @@ -186,10 +195,12 @@ The :mod:`csv` module defines the following classes: written if the dictionary is missing a key in *fieldnames*. If the dictionary passed to the :meth:`writerow` method contains a key not found in *fieldnames*, the optional *extrasaction* parameter indicates what action to - take. If it is set to ``'raise'`` a :exc:`ValueError` is raised. If it is - set to ``'ignore'``, extra values in the dictionary are ignored. Any other - optional or keyword arguments are passed to the underlying :class:`writer` - instance. + take. + If it is set to ``'raise'``, the default value, a :exc:`ValueError` + is raised. + If it is set to ``'ignore'``, extra values in the dictionary are ignored. + Any other optional or keyword arguments are passed to the underlying + :class:`writer` instance. Note that unlike the :class:`DictReader` class, the *fieldnames* parameter of the :class:`DictWriter` is not optional. Since Python's :class:`dict` diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index 94822ba0d1..3381a9d3db 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -1239,9 +1239,10 @@ When programming in a compiled language, shared libraries are accessed when compiling/linking a program, and when the program is run. The purpose of the :func:`find_library` function is to locate a library in a way -similar to what the compiler does (on platforms with several versions of a -shared library the most recent should be loaded), while the ctypes library -loaders act like when a program is run, and call the runtime loader directly. +similar to what the compiler or runtime loader does (on platforms with several +versions of a shared library the most recent should be loaded), while the ctypes +library loaders act like when a program is run, and call the runtime loader +directly. The :mod:`ctypes.util` module provides a function which can help to determine the library to load. @@ -1259,8 +1260,14 @@ the library to load. The exact functionality is system dependent. On Linux, :func:`find_library` tries to run external programs -(``/sbin/ldconfig``, ``gcc``, and ``objdump``) to find the library file. It -returns the filename of the library file. Here are some examples:: +(``/sbin/ldconfig``, ``gcc``, ``objdump`` and ``ld``) to find the library file. +It returns the filename of the library file. + +.. versionchanged:: 3.6 + On Linux, the value of the environment variable ``LD_LIBRARY_PATH`` is used + when searching for libraries, if a library cannot be found by any other means. + +Here are some examples:: >>> from ctypes.util import find_library >>> find_library("m") diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 9254ae8cfa..ecaad06ccc 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -522,7 +522,7 @@ objects are considered to be true. Instance methods: -.. method:: date.replace(year, month, day) +.. method:: date.replace(year=self.year, month=self.month, day=self.day) Return a date with the same value, except for those parameters given new values by whichever keyword arguments are specified. For example, if ``d == @@ -610,7 +610,8 @@ Instance methods: .. method:: date.__format__(format) Same as :meth:`.date.strftime`. This makes it possible to specify a format - string for a :class:`.date` object when using :meth:`str.format`. For a + string for a :class:`.date` object in :ref:`formatted string + literals <f-strings>` and when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -682,22 +683,26 @@ both directions; like a time object, :class:`.datetime` assumes there are exactl Constructor: -.. class:: datetime(year, month, day, hour=0, minute=0, second=0, microsecond=0, tzinfo=None) +.. class:: datetime(year, month, day, hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0) The year, month and day arguments are required. *tzinfo* may be ``None``, or an instance of a :class:`tzinfo` subclass. The remaining arguments may be integers, in the following ranges: - * ``MINYEAR <= year <= MAXYEAR`` - * ``1 <= month <= 12`` - * ``1 <= day <= number of days in the given month and year`` - * ``0 <= hour < 24`` - * ``0 <= minute < 60`` - * ``0 <= second < 60`` - * ``0 <= microsecond < 1000000`` + * ``MINYEAR <= year <= MAXYEAR``, + * ``1 <= month <= 12``, + * ``1 <= day <= number of days in the given month and year``, + * ``0 <= hour < 24``, + * ``0 <= minute < 60``, + * ``0 <= second < 60``, + * ``0 <= microsecond < 1000000``, + * ``fold in [0, 1]``. If an argument outside those ranges is given, :exc:`ValueError` is raised. + .. versionadded:: 3.6 + Added the ``fold`` argument. + Other constructors, all class methods: .. classmethod:: datetime.today() @@ -757,6 +762,8 @@ Other constructors, all class methods: instead of :exc:`ValueError` on :c:func:`localtime` or :c:func:`gmtime` failure. + .. versionchanged:: 3.6 + :meth:`fromtimestamp` may return instances with :attr:`.fold` set to 1. .. classmethod:: datetime.utcfromtimestamp(timestamp) @@ -793,16 +800,23 @@ Other constructors, all class methods: microsecond of the result are all 0, and :attr:`.tzinfo` is ``None``. -.. classmethod:: datetime.combine(date, time) +.. classmethod:: datetime.combine(date, time, tzinfo=self.tzinfo) Return a new :class:`.datetime` object whose date components are equal to the - given :class:`date` object's, and whose time components and :attr:`.tzinfo` - attributes are equal to the given :class:`.time` object's. For any - :class:`.datetime` object *d*, - ``d == datetime.combine(d.date(), d.timetz())``. If date is a + given :class:`date` object's, and whose time components + are equal to the given :class:`.time` object's. If the *tzinfo* + argument is provided, its value is used to set the :attr:`.tzinfo` attribute + of the result, otherwise the :attr:`~.time.tzinfo` attribute of the *time* argument + is used. + + For any :class:`.datetime` object *d*, + ``d == datetime.combine(d.date(), d.time(), d.tzinfo)``. If date is a :class:`.datetime` object, its time components and :attr:`.tzinfo` attributes are ignored. + .. versionchanged:: 3.6 + Added the *tzinfo* argument. + .. classmethod:: datetime.strptime(date_string, format) @@ -878,6 +892,16 @@ Instance attributes (read-only): or ``None`` if none was passed. +.. attribute:: datetime.fold + + In ``[0, 1]``. Used to disambiguate wall times during a repeated interval. (A + repeated interval occurs when clocks are rolled back at the end of daylight saving + time or when the UTC offset for the current zone is decreased for political reasons.) + The value 0 (1) represents the earlier (later) of the two moments with the same wall + time representation. + + .. versionadded:: 3.6 + Supported operations: +---------------------------------------+--------------------------------+ @@ -966,23 +990,34 @@ Instance methods: .. method:: datetime.time() - Return :class:`.time` object with same hour, minute, second and microsecond. + Return :class:`.time` object with same hour, minute, second, microsecond and fold. :attr:`.tzinfo` is ``None``. See also method :meth:`timetz`. + .. versionchanged:: 3.6 + The fold value is copied to the returned :class:`.time` object. + .. method:: datetime.timetz() - Return :class:`.time` object with same hour, minute, second, microsecond, and + Return :class:`.time` object with same hour, minute, second, microsecond, fold, and tzinfo attributes. See also method :meth:`time`. + .. versionchanged:: 3.6 + The fold value is copied to the returned :class:`.time` object. -.. method:: datetime.replace([year[, month[, day[, hour[, minute[, second[, microsecond[, tzinfo]]]]]]]]) + +.. method:: datetime.replace(year=self.year, month=self.month, day=self.day, \ + hour=self.hour, minute=self.minute, second=self.second, microsecond=self.microsecond, \ + tzinfo=self.tzinfo, * fold=0) Return a datetime with the same attributes, except for those attributes given new values by whichever keyword arguments are specified. Note that ``tzinfo=None`` can be specified to create a naive datetime from an aware datetime with no conversion of date and time data. + .. versionadded:: 3.6 + Added the ``fold`` argument. + .. method:: datetime.astimezone(tz=None) @@ -991,23 +1026,20 @@ Instance methods: *self*, but in *tz*'s local time. If provided, *tz* must be an instance of a :class:`tzinfo` subclass, and its - :meth:`utcoffset` and :meth:`dst` methods must not return ``None``. *self* must - be aware (``self.tzinfo`` must not be ``None``, and ``self.utcoffset()`` must - not return ``None``). + :meth:`utcoffset` and :meth:`dst` methods must not return ``None``. If *self* + is naive (``self.tzinfo is None``), it is presumed to represent time in the + system timezone. If called without arguments (or with ``tz=None``) the system local - timezone is assumed. The ``.tzinfo`` attribute of the converted + timezone is assumed for the target timezone. The ``.tzinfo`` attribute of the converted datetime instance will be set to an instance of :class:`timezone` with the zone name and offset obtained from the OS. If ``self.tzinfo`` is *tz*, ``self.astimezone(tz)`` is equal to *self*: no adjustment of date or time data is performed. Else the result is local - time in time zone *tz*, representing the same UTC time as *self*: after - ``astz = dt.astimezone(tz)``, ``astz - astz.utcoffset()`` will usually have - the same date and time data as ``dt - dt.utcoffset()``. The discussion - of class :class:`tzinfo` explains the cases at Daylight Saving Time transition - boundaries where this cannot be achieved (an issue only if *tz* models both - standard and daylight time). + time in the timezone *tz*, representing the same UTC time as *self*: after + ``astz = dt.astimezone(tz)``, ``astz - astz.utcoffset()`` will have + the same date and time data as ``dt - dt.utcoffset()``. If you merely want to attach a time zone object *tz* to a datetime *dt* without adjustment of date and time data, use ``dt.replace(tzinfo=tz)``. If you @@ -1029,6 +1061,10 @@ Instance methods: .. versionchanged:: 3.3 *tz* now can be omitted. + .. versionchanged:: 3.6 + The :meth:`astimezone` method can now be called on naive instances that + are presumed to represent system local time. + .. method:: datetime.utcoffset() @@ -1105,6 +1141,10 @@ Instance methods: .. versionadded:: 3.3 + .. versionchanged:: 3.6 + The :meth:`timestamp` method uses the :attr:`.fold` attribute to + disambiguate the times during a repeated interval. + .. note:: There is no method to obtain the POSIX timestamp directly from a @@ -1138,7 +1178,7 @@ Instance methods: ``self.date().isocalendar()``. -.. method:: datetime.isoformat(sep='T') +.. method:: datetime.isoformat(sep='T', timespec='auto') Return a string representing the date and time in ISO 8601 format, YYYY-MM-DDTHH:MM:SS.mmmmmm or, if :attr:`microsecond` is 0, @@ -1159,6 +1199,37 @@ Instance methods: >>> datetime(2002, 12, 25, tzinfo=TZ()).isoformat(' ') '2002-12-25 00:00:00-06:39' + The optional argument *timespec* specifies the number of additional + components of the time to include (the default is ``'auto'``). + It can be one of the following: + + - ``'auto'``: Same as ``'seconds'`` if :attr:`microsecond` is 0, + same as ``'microseconds'`` otherwise. + - ``'hours'``: Include the :attr:`hour` in the two-digit HH format. + - ``'minutes'``: Include :attr:`hour` and :attr:`minute` in HH:MM format. + - ``'seconds'``: Include :attr:`hour`, :attr:`minute`, and :attr:`second` + in HH:MM:SS format. + - ``'milliseconds'``: Include full time, but truncate fractional second + part to milliseconds. HH:MM:SS.sss format. + - ``'microseconds'``: Include full time in HH:MM:SS.mmmmmm format. + + .. note:: + + Excluded time components are truncated, not rounded. + + :exc:`ValueError` will be raised on an invalid *timespec* argument. + + + >>> from datetime import datetime + >>> datetime.now().isoformat(timespec='minutes') + '2002-12-25T00:00' + >>> dt = datetime(2015, 1, 1, 12, 30, 59, 0) + >>> dt.isoformat(timespec='microseconds') + '2015-01-01T12:30:59.000000' + + .. versionadded:: 3.6 + Added the *timespec* argument. + .. method:: datetime.__str__() @@ -1185,7 +1256,8 @@ Instance methods: .. method:: datetime.__format__(format) Same as :meth:`.datetime.strftime`. This makes it possible to specify a format - string for a :class:`.datetime` object when using :meth:`str.format`. For a + string for a :class:`.datetime` object in :ref:`formatted string + literals <f-strings>` and when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -1302,16 +1374,17 @@ Using datetime with tzinfo: A time object represents a (local) time of day, independent of any particular day, and subject to adjustment via a :class:`tzinfo` object. -.. class:: time(hour=0, minute=0, second=0, microsecond=0, tzinfo=None) +.. class:: time(hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0) All arguments are optional. *tzinfo* may be ``None``, or an instance of a :class:`tzinfo` subclass. The remaining arguments may be integers, in the following ranges: - * ``0 <= hour < 24`` - * ``0 <= minute < 60`` - * ``0 <= second < 60`` - * ``0 <= microsecond < 1000000``. + * ``0 <= hour < 24``, + * ``0 <= minute < 60``, + * ``0 <= second < 60``, + * ``0 <= microsecond < 1000000``, + * ``fold in [0, 1]``. If an argument outside those ranges is given, :exc:`ValueError` is raised. All default to ``0`` except *tzinfo*, which defaults to :const:`None`. @@ -1364,6 +1437,17 @@ Instance attributes (read-only): ``None`` if none was passed. +.. attribute:: time.fold + + In ``[0, 1]``. Used to disambiguate wall times during a repeated interval. (A + repeated interval occurs when clocks are rolled back at the end of daylight saving + time or when the UTC offset for the current zone is decreased for political reasons.) + The value 0 (1) represents the earlier (later) of the two moments with the same wall + time representation. + + .. versionadded:: 3.6 + + Supported operations: * comparison of :class:`.time` to :class:`.time`, where *a* is considered less @@ -1399,21 +1483,58 @@ In boolean contexts, a :class:`.time` object is always considered to be true. Instance methods: -.. method:: time.replace([hour[, minute[, second[, microsecond[, tzinfo]]]]]) +.. method:: time.replace(hour=self.hour, minute=self.minute, second=self.second, \ + microsecond=self.microsecond, tzinfo=self.tzinfo, * fold=0) Return a :class:`.time` with the same value, except for those attributes given new values by whichever keyword arguments are specified. Note that ``tzinfo=None`` can be specified to create a naive :class:`.time` from an aware :class:`.time`, without conversion of the time data. + .. versionadded:: 3.6 + Added the ``fold`` argument. + -.. method:: time.isoformat() +.. method:: time.isoformat(timespec='auto') Return a string representing the time in ISO 8601 format, HH:MM:SS.mmmmmm or, if - self.microsecond is 0, HH:MM:SS If :meth:`utcoffset` does not return ``None``, a + :attr:`microsecond` is 0, HH:MM:SS If :meth:`utcoffset` does not return ``None``, a 6-character string is appended, giving the UTC offset in (signed) hours and minutes: HH:MM:SS.mmmmmm+HH:MM or, if self.microsecond is 0, HH:MM:SS+HH:MM + The optional argument *timespec* specifies the number of additional + components of the time to include (the default is ``'auto'``). + It can be one of the following: + + - ``'auto'``: Same as ``'seconds'`` if :attr:`microsecond` is 0, + same as ``'microseconds'`` otherwise. + - ``'hours'``: Include the :attr:`hour` in the two-digit HH format. + - ``'minutes'``: Include :attr:`hour` and :attr:`minute` in HH:MM format. + - ``'seconds'``: Include :attr:`hour`, :attr:`minute`, and :attr:`second` + in HH:MM:SS format. + - ``'milliseconds'``: Include full time, but truncate fractional second + part to milliseconds. HH:MM:SS.sss format. + - ``'microseconds'``: Include full time in HH:MM:SS.mmmmmm format. + + .. note:: + + Excluded time components are truncated, not rounded. + + :exc:`ValueError` will be raised on an invalid *timespec* argument. + + + >>> from datetime import time + >>> time(hour=12, minute=34, second=56, microsecond=123456).isoformat(timespec='minutes') + '12:34' + >>> dt = time(hour=12, minute=34, second=56, microsecond=0) + >>> dt.isoformat(timespec='microseconds') + '12:34:56.000000' + >>> dt.isoformat(timespec='auto') + '12:34:56' + + .. versionadded:: 3.6 + Added the *timespec* argument. + .. method:: time.__str__() @@ -1430,7 +1551,8 @@ Instance methods: .. method:: time.__format__(format) Same as :meth:`.time.strftime`. This makes it possible to specify a format string - for a :class:`.time` object when using :meth:`str.format`. For a + for a :class:`.time` object in :ref:`formatted string + literals <f-strings>` and when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -1680,9 +1802,19 @@ minute after 1:59 (EST) on the second Sunday in March, and ends the minute after When DST starts (the "start" line), the local wall clock leaps from 1:59 to 3:00. A wall time of the form 2:MM doesn't really make sense on that day, so ``astimezone(Eastern)`` won't deliver a result with ``hour == 2`` on the day DST -begins. In order for :meth:`astimezone` to make this guarantee, the -:meth:`tzinfo.dst` method must consider times in the "missing hour" (2:MM for -Eastern) to be in daylight time. +begins. For example, at the Spring forward transition of 2016, we get + + >>> u0 = datetime(2016, 3, 13, 5, tzinfo=timezone.utc) + >>> for i in range(4): + ... u = u0 + i*HOUR + ... t = u.astimezone(Eastern) + ... print(u.time(), 'UTC =', t.time(), t.tzname()) + ... + 05:00:00 UTC = 00:00:00 EST + 06:00:00 UTC = 01:00:00 EST + 07:00:00 UTC = 03:00:00 EDT + 08:00:00 UTC = 04:00:00 EDT + When DST ends (the "end" line), there's a potentially worse problem: there's an hour that can't be spelled unambiguously in local wall time: the last hour of @@ -1691,28 +1823,41 @@ daylight time ends. The local wall clock leaps from 1:59 (daylight time) back to 1:00 (standard time) again. Local times of the form 1:MM are ambiguous. :meth:`astimezone` mimics the local clock's behavior by mapping two adjacent UTC hours into the same local hour then. In the Eastern example, UTC times of the -form 5:MM and 6:MM both map to 1:MM when converted to Eastern. In order for -:meth:`astimezone` to make this guarantee, the :meth:`tzinfo.dst` method must -consider times in the "repeated hour" to be in standard time. This is easily -arranged, as in the example, by expressing DST switch times in the time zone's -standard local time. +form 5:MM and 6:MM both map to 1:MM when converted to Eastern, but earlier times +have the :attr:`~datetime.fold` attribute set to 0 and the later times have it set to 1. +For example, at the Fall back transition of 2016, we get + + >>> u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc) + >>> for i in range(4): + ... u = u0 + i*HOUR + ... t = u.astimezone(Eastern) + ... print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold) + ... + 04:00:00 UTC = 00:00:00 EDT 0 + 05:00:00 UTC = 01:00:00 EDT 0 + 06:00:00 UTC = 01:00:00 EST 1 + 07:00:00 UTC = 02:00:00 EST 0 + +Note that the :class:`datetime` instances that differ only by the value of the +:attr:`~datetime.fold` attribute are considered equal in comparisons. -Applications that can't bear such ambiguities should avoid using hybrid +Applications that can't bear wall-time ambiguities should explicitly check the +value of the :attr:`~datetime.fold` attribute or avoid using hybrid :class:`tzinfo` subclasses; there are no ambiguities when using :class:`timezone`, or any other fixed-offset :class:`tzinfo` subclass (such as a class representing only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). .. seealso:: - `pytz <https://pypi.python.org/pypi/pytz/>`_ + `datetuil.tz <https://dateutil.readthedocs.io/en/stable/tz.html>`_ The standard library has :class:`timezone` class for handling arbitrary fixed offsets from UTC and :attr:`timezone.utc` as UTC timezone instance. - *pytz* library brings the *IANA timezone database* (also known as the + *datetuil.tz* library brings the *IANA timezone database* (also known as the Olson database) to Python and its usage is recommended. `IANA timezone database <https://www.iana.org/time-zones>`_ - The Time Zone Database (often called tz or zoneinfo) contains code and + The Time Zone Database (often called tz, tzdata or zoneinfo) contains code and data that represent the history of local time for many representative locations around the globe. It is updated periodically to reflect changes made by political bodies to time zone boundaries, UTC offsets, and @@ -1732,7 +1877,7 @@ in different days of the year or where historical changes have been made to civil time. -.. class:: timezone(offset[, name]) +.. class:: timezone(offset, name=None) The *offset* argument must be specified as a :class:`timedelta` object representing the difference between the local time and UTC. It must @@ -1741,10 +1886,7 @@ made to civil time. otherwise :exc:`ValueError` is raised. The *name* argument is optional. If specified it must be a string that - is used as the value returned by the ``tzname(dt)`` method. Otherwise, - ``tzname(dt)`` returns a string 'UTCsHH:MM', where s is the sign of - *offset*, HH and MM are two digits of ``offset.hours`` and - ``offset.minutes`` respectively. + will be used as the value returned by the :meth:`datetime.tzname` method. .. versionadded:: 3.2 @@ -1757,11 +1899,19 @@ made to civil time. .. method:: timezone.tzname(dt) - Return the fixed value specified when the :class:`timezone` instance is - constructed or a string 'UTCsHH:MM', where s is the sign of - *offset*, HH and MM are two digits of ``offset.hours`` and + Return the fixed value specified when the :class:`timezone` instance + is constructed. If *name* is not provided in the constructor, the + name returned by ``tzname(dt)`` is generated from the value of the + ``offset`` as follows. If *offset* is ``timedelta(0)``, the name + is "UTC", otherwise it is a string 'UTC±HH:MM', where ± is the sign + of ``offset``, HH and MM are two digits of ``offset.hours`` and ``offset.minutes`` respectively. + .. versionchanged:: 3.6 + Name generated from ``offset=timedelta(0)`` is now plain 'UTC', not + 'UTC+00:00'. + + .. method:: timezone.dst(dt) Always returns ``None``. @@ -1911,6 +2061,34 @@ format codes. | ``%%`` | A literal ``'%'`` character. | % | | +-----------+--------------------------------+------------------------+-------+ +Several additional directives not required by the C89 standard are included for +convenience. These parameters all correspond to ISO 8601 date values. These +may not be available on all platforms when used with the :meth:`strftime` +method. The ISO 8601 year and ISO 8601 week directives are not interchangeable +with the year and week number directives above. Calling :meth:`strptime` with +incomplete or ambiguous ISO 8601 directives will raise a :exc:`ValueError`. + ++-----------+--------------------------------+------------------------+-------+ +| Directive | Meaning | Example | Notes | ++===========+================================+========================+=======+ +| ``%G`` | ISO 8601 year with century | 0001, 0002, ..., 2013, | \(8) | +| | representing the year that | 2014, ..., 9998, 9999 | | +| | contains the greater part of | | | +| | the ISO week (``%V``). | | | ++-----------+--------------------------------+------------------------+-------+ +| ``%u`` | ISO 8601 weekday as a decimal | 1, 2, ..., 7 | | +| | number where 1 is Monday. | | | ++-----------+--------------------------------+------------------------+-------+ +| ``%V`` | ISO 8601 week as a decimal | 01, 02, ..., 53 | \(8) | +| | number with Monday as | | | +| | the first day of the week. | | | +| | Week 01 is the week containing | | | +| | Jan 4. | | | ++-----------+--------------------------------+------------------------+-------+ + +.. versionadded:: 3.6 + ``%G``, ``%u`` and ``%V`` were added. + Notes: (1) @@ -1975,7 +2153,14 @@ Notes: (7) When used with the :meth:`strptime` method, ``%U`` and ``%W`` are only used - in calculations when the day of the week and the year are specified. + in calculations when the day of the week and the calendar year (``%Y``) + are specified. + +(8) + Similar to ``%U`` and ``%W``, ``%V`` is only used in calculations when the + day of the week and the ISO year (``%G``) are specified in a + :meth:`strptime` format string. Also note that ``%G`` and ``%Y`` are not + interchangeable. .. rubric:: Footnotes diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index 2a1db91427..32e80b2cf6 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -351,6 +351,10 @@ The module defines the following: :func:`.open` always creates a new database when the flag has the value ``'n'``. + .. deprecated-removed:: 3.6 3.8 + Creating database in ``'r'`` and ``'w'`` modes. Modifying database in + ``'r'`` mode. + In addition to the methods provided by the :class:`collections.abc.MutableMapping` class, :class:`dumbdbm` objects provide the following methods: diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index b5ce0b1a8e..e984edcb75 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -345,7 +345,7 @@ Decimal objects *value* can be an integer, string, tuple, :class:`float`, or another :class:`Decimal` object. If no *value* is given, returns ``Decimal('0')``. If *value* is a string, it should conform to the decimal numeric string syntax after leading - and trailing whitespace characters are removed:: + and trailing whitespace characters, as well as underscores throughout, are removed:: sign ::= '+' | '-' digit ::= '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' @@ -394,6 +394,10 @@ Decimal objects :class:`float` arguments raise an exception if the :exc:`FloatOperation` trap is set. By default the trap is off. + .. versionchanged:: 3.6 + Underscores are allowed for grouping, as with integral and floating-point + literals in code. + Decimal floating point objects share many properties with the other built-in numeric types such as :class:`float` and :class:`int`. All of the usual math operations and special methods apply. Likewise, decimal objects can be @@ -447,6 +451,19 @@ Decimal objects ``Decimal('321e+5').adjusted()`` returns seven. Used for determining the position of the most significant digit with respect to the decimal point. + .. method:: as_integer_ratio() + + Return a pair ``(n, d)`` of integers that represent the given + :class:`Decimal` instance as a fraction, in lowest terms and + with a positive denominator:: + + >>> Decimal('-3.14').as_integer_ratio() + (-157, 50) + + The conversion is exact. Raise OverflowError on infinities and ValueError + on NaNs. + + .. versionadded:: 3.6 .. method:: as_tuple() @@ -1062,8 +1079,8 @@ In addition to the three supplied contexts, new contexts can be created with the Decimal('4.44') This method implements the to-number operation of the IBM specification. - If the argument is a string, no leading or trailing whitespace is - permitted. + If the argument is a string, no leading or trailing whitespace or + underscores are permitted. .. method:: create_decimal_from_float(f) diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index f86725b601..a15690ba48 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -31,9 +31,9 @@ the following command can be used to display the disassembly of >>> dis.dis(myfunc) 2 0 LOAD_GLOBAL 0 (len) - 3 LOAD_FAST 0 (alist) - 6 CALL_FUNCTION 1 - 9 RETURN_VALUE + 2 LOAD_FAST 0 (alist) + 4 CALL_FUNCTION 1 + 6 RETURN_VALUE (The "2" is a line number). @@ -607,6 +607,14 @@ iterations of the loop. .. versionadded:: 3.3 +.. opcode:: SETUP_ANNOTATIONS + + Checks whether ``__annotations__`` is defined in ``locals()``, if not it is + set up to an empty ``dict``. This opcode is only emitted if a class + or module body contains :term:`variable annotations <variable annotation>` + statically. + + .. versionadded:: 3.6 .. opcode:: IMPORT_STAR @@ -682,8 +690,7 @@ iterations of the loop. .. XXX explain the WHY stuff! -All of the following opcodes expect arguments. An argument is two bytes, with -the more significant byte last. +All of the following opcodes use their arguments. .. opcode:: STORE_NAME (namei) @@ -769,6 +776,23 @@ the more significant byte last. to hold *count* entries. +.. opcode:: BUILD_CONST_KEY_MAP (count) + + The version of :opcode:`BUILD_MAP` specialized for constant keys. *count* + values are consumed from the stack. The top element on the stack contains + a tuple of keys. + + .. versionadded:: 3.6 + + +.. opcode:: BUILD_STRING (count) + + Concatenates *count* strings from the stack and pushes the resulting string + onto the stack. + + .. versionadded:: 3.6 + + .. opcode:: LOAD_ATTR (namei) Replaces TOS with ``getattr(TOS, co_names[namei])``. @@ -874,6 +898,13 @@ the more significant byte last. Deletes local ``co_varnames[var_num]``. +.. opcode:: STORE_ANNOTATION (namei) + + Stores TOS as ``locals()['__annotations__'][co_names[namei]] = TOS``. + + .. versionadded:: 3.6 + + .. opcode:: LOAD_CLOSURE (i) Pushes a reference to the cell contained in slot *i* of the cell and free @@ -929,27 +960,16 @@ the more significant byte last. .. opcode:: MAKE_FUNCTION (argc) Pushes a new function object on the stack. From bottom to top, the consumed - stack must consist of - - * ``argc & 0xFF`` default argument objects in positional order - * ``(argc >> 8) & 0xFF`` pairs of name and default argument, with the name - just below the object on the stack, for keyword-only parameters - * ``(argc >> 16) & 0x7FFF`` parameter annotation objects - * a tuple listing the parameter names for the annotations (only if there are - ony annotation objects) + stack must consist of values if the argument carries a specified flag value + + * ``0x01`` a tuple of default argument objects in positional order + * ``0x02`` a dictionary of keyword-only parameters' default values + * ``0x04`` an annotation dictionary + * ``0x08`` a tuple containing cells for free variables, making a closure * the code associated with the function (at TOS1) * the :term:`qualified name` of the function (at TOS) -.. opcode:: MAKE_CLOSURE (argc) - - Creates a new function object, sets its *__closure__* slot, and pushes it on - the stack. TOS is the :term:`qualified name` of the function, TOS1 is the - code associated with the function, and TOS2 is the tuple containing cells for - the closure's free variables. *argc* is interpreted as in ``MAKE_FUNCTION``; - the annotations and defaults are also in the same order below TOS2. - - .. opcode:: BUILD_SLICE (argc) .. index:: builtin: slice @@ -989,6 +1009,28 @@ the more significant byte last. arguments. +.. opcode:: FORMAT_VALUE (flags) + + Used for implementing formatted literal strings (f-strings). Pops + an optional *fmt_spec* from the stack, then a required *value*. + *flags* is interpreted as follows: + + * ``(flags & 0x03) == 0x00``: *value* is formatted as-is. + * ``(flags & 0x03) == 0x01``: call :func:`str` on *value* before + formatting it. + * ``(flags & 0x03) == 0x02``: call :func:`repr` on *value* before + formatting it. + * ``(flags & 0x03) == 0x03``: call :func:`ascii` on *value* before + formatting it. + * ``(flags & 0x04) == 0x04``: pop *fmt_spec* from the stack and use + it, else use an empty *fmt_spec*. + + Formatting is performed using :c:func:`PyObject_Format`. The + result is pushed on the stack. + + .. versionadded:: 3.6 + + .. opcode:: HAVE_ARGUMENT This is not really an opcode. It identifies the dividing line between diff --git a/Doc/library/email.charset.rst b/Doc/library/email.charset.rst index 161d86a3b7..053463f974 100644 --- a/Doc/library/email.charset.rst +++ b/Doc/library/email.charset.rst @@ -8,6 +8,11 @@ -------------- +This module is part of the legacy (``Compat32``) email API. In the new +API only the aliases table is used. + +The remaining text in this section is the original documentation of the module. + This module provides a class :class:`Charset` for representing character sets and character set conversions in email messages, as well as a character set registry and several convenience methods for manipulating this registry. diff --git a/Doc/library/email.compat32-message.rst b/Doc/library/email.compat32-message.rst new file mode 100644 index 0000000000..2c65079ebd --- /dev/null +++ b/Doc/library/email.compat32-message.rst @@ -0,0 +1,754 @@ +.. _compat32_message: + +:mod:`email.message.Message`: Representing an email message using the :data:`~email.policy.compat32` API +-------------------------------------------------------------------------------------------------------- + +.. module:: email.message + :synopsis: The base class representing email messages in a fashion + backward compatible with python3.2 + + +The :class:`Message` class is very similar to the +:class:`~email.message.EmailMessage` class, without the methods added by that +class, and with the default behavior of certain other methods being slightly +different. We also document here some methods that, while supported by the +:class:`~email.message.EmailMessage` class, are not recommended unless you are +dealing with legacy code. + +The philosophy and structure of the two classes is otherwise the same. + +This document describes the behavior under the default (for :class:`Message`) +policy :attr:`~email.policy.Compat32`. If you are going to use another policy, +you should be using the :class:`~email.message.EmailMessage` class instead. + +An email message consists of *headers* and a *payload*. Headers must be +:rfc:`5233` style names and values, where the field name and value are +separated by a colon. The colon is not part of either the field name or the +field value. The payload may be a simple text message, or a binary object, or +a structured sequence of sub-messages each with their own set of headers and +their own payload. The latter type of payload is indicated by the message +having a MIME type such as :mimetype:`multipart/\*` or +:mimetype:`message/rfc822`. + +The conceptual model provided by a :class:`Message` object is that of an +ordered dictionary of headers with additional methods for accessing both +specialized information from the headers, for accessing the payload, for +generating a serialized version of the mssage, and for recursively walking over +the object tree. Note that duplicate headers are supported but special methods +must be used to access them. + +The :class:`Message` psuedo-dictionary is indexed by the header names, which +must be ASCII values. The values of the dictionary are strings that are +supposed to contain only ASCII characters; there is some special handling for +non-ASCII input, but it doesn't always produce the correct results. Headers +are stored and returned in case-preserving form, but field names are matched +case-insensitively. There may also be a single envelope header, also known as +the *Unix-From* header or the ``From_`` header. The *payload* is either a +string or bytes, in the case of simple message objects, or a list of +:class:`Message` objects, for MIME container documents (e.g. +:mimetype:`multipart/\*` and :mimetype:`message/rfc822`). + +Here are the methods of the :class:`Message` class: + + +.. class:: Message(policy=compat32) + + If *policy* is specified (it must be an instance of a :mod:`~email.policy` + class) use the rules it specifies to update and serialize the representation + of the message. If *policy* is not set, use the :class:`compat32 + <email.policy.Compat32>` policy, which maintains backward compatibility with + the Python 3.2 version of the email package. For more information see the + :mod:`~email.policy` documentation. + + .. versionchanged:: 3.3 The *policy* keyword argument was added. + + + .. method:: as_string(unixfrom=False, maxheaderlen=0, policy=None) + + Return the entire message flattened as a string. When optional *unixfrom* + is true, the envelope header is included in the returned string. + *unixfrom* defaults to ``False``. For backward compabitility reasons, + *maxheaderlen* defaults to ``0``, so if you want a different value you + must override it explicitly (the value specified for *max_line_length* in + the policy will be ignored by this method). The *policy* argument may be + used to override the default policy obtained from the message instance. + This can be used to control some of the formatting produced by the + method, since the specified *policy* will be passed to the ``Generator``. + + Flattening the message may trigger changes to the :class:`Message` if + defaults need to be filled in to complete the transformation to a string + (for example, MIME boundaries may be generated or modified). + + Note that this method is provided as a convenience and may not always + format the message the way you want. For example, by default it does + not do the mangling of lines that begin with ``From`` that is + required by the unix mbox format. For more flexibility, instantiate a + :class:`~email.generator.Generator` instance and use its + :meth:`~email.generator.Generator.flatten` method directly. For example:: + + from io import StringIO + from email.generator import Generator + fp = StringIO() + g = Generator(fp, mangle_from_=True, maxheaderlen=60) + g.flatten(msg) + text = fp.getvalue() + + If the message object contains binary data that is not encoded according + to RFC standards, the non-compliant data will be replaced by unicode + "unknown character" code points. (See also :meth:`.as_bytes` and + :class:`~email.generator.BytesGenerator`.) + + .. versionchanged:: 3.4 the *policy* keyword argument was added. + + + .. method:: __str__() + + Equivalent to :meth:`.as_string()`. Allows ``str(msg)`` to produce a + string containing the formatted message. + + + .. method:: as_bytes(unixfrom=False, policy=None) + + Return the entire message flattened as a bytes object. When optional + *unixfrom* is true, the envelope header is included in the returned + string. *unixfrom* defaults to ``False``. The *policy* argument may be + used to override the default policy obtained from the message instance. + This can be used to control some of the formatting produced by the + method, since the specified *policy* will be passed to the + ``BytesGenerator``. + + Flattening the message may trigger changes to the :class:`Message` if + defaults need to be filled in to complete the transformation to a string + (for example, MIME boundaries may be generated or modified). + + Note that this method is provided as a convenience and may not always + format the message the way you want. For example, by default it does + not do the mangling of lines that begin with ``From`` that is + required by the unix mbox format. For more flexibility, instantiate a + :class:`~email.generator.BytesGenerator` instance and use its + :meth:`~email.generator.BytesGenerator.flatten` method directly. + For example:: + + from io import BytesIO + from email.generator import BytesGenerator + fp = BytesIO() + g = BytesGenerator(fp, mangle_from_=True, maxheaderlen=60) + g.flatten(msg) + text = fp.getvalue() + + .. versionadded:: 3.4 + + + .. method:: __bytes__() + + Equivalent to :meth:`.as_bytes()`. Allows ``bytes(msg)`` to produce a + bytes object containing the formatted message. + + .. versionadded:: 3.4 + + + .. method:: is_multipart() + + Return ``True`` if the message's payload is a list of sub-\ + :class:`Message` objects, otherwise return ``False``. When + :meth:`is_multipart` returns ``False``, the payload should be a string + object (which might be a CTE encoded binary payload. (Note that + :meth:`is_multipart` returning ``True`` does not necessarily mean that + "msg.get_content_maintype() == 'multipart'" will return the ``True``. + For example, ``is_multipart`` will return ``True`` when the + :class:`Message` is of type ``message/rfc822``.) + + + .. method:: set_unixfrom(unixfrom) + + Set the message's envelope header to *unixfrom*, which should be a string. + + + .. method:: get_unixfrom() + + Return the message's envelope header. Defaults to ``None`` if the + envelope header was never set. + + + .. method:: attach(payload) + + Add the given *payload* to the current payload, which must be ``None`` or + a list of :class:`Message` objects before the call. After the call, the + payload will always be a list of :class:`Message` objects. If you want to + set the payload to a scalar object (e.g. a string), use + :meth:`set_payload` instead. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by :meth:`~email.message.EmailMessage.set_content` and the + realted ``make`` and ``add`` methods. + + + .. method:: get_payload(i=None, decode=False) + + Return the current payload, which will be a list of + :class:`Message` objects when :meth:`is_multipart` is ``True``, or a + string when :meth:`is_multipart` is ``False``. If the payload is a list + and you mutate the list object, you modify the message's payload in place. + + With optional argument *i*, :meth:`get_payload` will return the *i*-th + element of the payload, counting from zero, if :meth:`is_multipart` is + ``True``. An :exc:`IndexError` will be raised if *i* is less than 0 or + greater than or equal to the number of items in the payload. If the + payload is a string (i.e. :meth:`is_multipart` is ``False``) and *i* is + given, a :exc:`TypeError` is raised. + + Optional *decode* is a flag indicating whether the payload should be + decoded or not, according to the :mailheader:`Content-Transfer-Encoding` + header. When ``True`` and the message is not a multipart, the payload will + be decoded if this header's value is ``quoted-printable`` or ``base64``. + If some other encoding is used, or :mailheader:`Content-Transfer-Encoding` + header is missing, the payload is + returned as-is (undecoded). In all cases the returned value is binary + data. If the message is a multipart and the *decode* flag is ``True``, + then ``None`` is returned. If the payload is base64 and it was not + perfectly formed (missing padding, characters outside the base64 + alphabet), then an appropriate defect will be added to the message's + defect property (:class:`~email.errors.InvalidBase64PaddingDefect` or + :class:`~email.errors.InvalidBase64CharactersDefect`, respectively). + + When *decode* is ``False`` (the default) the body is returned as a string + without decoding the :mailheader:`Content-Transfer-Encoding`. However, + for a :mailheader:`Content-Transfer-Encoding` of 8bit, an attempt is made + to decode the original bytes using the ``charset`` specified by the + :mailheader:`Content-Type` header, using the ``replace`` error handler. + If no ``charset`` is specified, or if the ``charset`` given is not + recognized by the email package, the body is decoded using the default + ASCII charset. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by :meth:`~email.message.EmailMessage.get_content` and + :meth:`~email.message.EmailMessage.iter_parts`. + + + .. method:: set_payload(payload, charset=None) + + Set the entire message object's payload to *payload*. It is the client's + responsibility to ensure the payload invariants. Optional *charset* sets + the message's default character set; see :meth:`set_charset` for details. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by :meth:`~email.message.EmailMessage.set_content`. + + + .. method:: set_charset(charset) + + Set the character set of the payload to *charset*, which can either be a + :class:`~email.charset.Charset` instance (see :mod:`email.charset`), a + string naming a character set, or ``None``. If it is a string, it will + be converted to a :class:`~email.charset.Charset` instance. If *charset* + is ``None``, the ``charset`` parameter will be removed from the + :mailheader:`Content-Type` header (the message will not be otherwise + modified). Anything else will generate a :exc:`TypeError`. + + If there is no existing :mailheader:`MIME-Version` header one will be + added. If there is no existing :mailheader:`Content-Type` header, one + will be added with a value of :mimetype:`text/plain`. Whether the + :mailheader:`Content-Type` header already exists or not, its ``charset`` + parameter will be set to *charset.output_charset*. If + *charset.input_charset* and *charset.output_charset* differ, the payload + will be re-encoded to the *output_charset*. If there is no existing + :mailheader:`Content-Transfer-Encoding` header, then the payload will be + transfer-encoded, if needed, using the specified + :class:`~email.charset.Charset`, and a header with the appropriate value + will be added. If a :mailheader:`Content-Transfer-Encoding` header + already exists, the payload is assumed to already be correctly encoded + using that :mailheader:`Content-Transfer-Encoding` and is not modified. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by the *charset* parameter of the + :meth:`email.emailmessage.EmailMessage.set_content` method. + + + .. method:: get_charset() + + Return the :class:`~email.charset.Charset` instance associated with the + message's payload. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class it always returns + ``None``. + + + The following methods implement a mapping-like interface for accessing the + message's :rfc:`2822` headers. Note that there are some semantic differences + between these methods and a normal mapping (i.e. dictionary) interface. For + example, in a dictionary there are no duplicate keys, but here there may be + duplicate message headers. Also, in dictionaries there is no guaranteed + order to the keys returned by :meth:`keys`, but in a :class:`Message` object, + headers are always returned in the order they appeared in the original + message, or were added to the message later. Any header deleted and then + re-added are always appended to the end of the header list. + + These semantic differences are intentional and are biased toward maximal + convenience. + + Note that in all cases, any envelope header present in the message is not + included in the mapping interface. + + In a model generated from bytes, any header values that (in contravention of + the RFCs) contain non-ASCII bytes will, when retrieved through this + interface, be represented as :class:`~email.header.Header` objects with + a charset of `unknown-8bit`. + + + .. method:: __len__() + + Return the total number of headers, including duplicates. + + + .. method:: __contains__(name) + + Return true if the message object has a field named *name*. Matching is + done case-insensitively and *name* should not include the trailing colon. + Used for the ``in`` operator, e.g.:: + + if 'message-id' in myMessage: + print('Message-ID:', myMessage['message-id']) + + + .. method:: __getitem__(name) + + Return the value of the named header field. *name* should not include the + colon field separator. If the header is missing, ``None`` is returned; a + :exc:`KeyError` is never raised. + + Note that if the named field appears more than once in the message's + headers, exactly which of those field values will be returned is + undefined. Use the :meth:`get_all` method to get the values of all the + extant named headers. + + + .. method:: __setitem__(name, val) + + Add a header to the message with field name *name* and value *val*. The + field is appended to the end of the message's existing fields. + + Note that this does *not* overwrite or delete any existing header with the same + name. If you want to ensure that the new header is the only one present in the + message with field name *name*, delete the field first, e.g.:: + + del msg['subject'] + msg['subject'] = 'Python roolz!' + + + .. method:: __delitem__(name) + + Delete all occurrences of the field with name *name* from the message's + headers. No exception is raised if the named field isn't present in the + headers. + + + .. method:: keys() + + Return a list of all the message's header field names. + + + .. method:: values() + + Return a list of all the message's field values. + + + .. method:: items() + + Return a list of 2-tuples containing all the message's field headers and + values. + + + .. method:: get(name, failobj=None) + + Return the value of the named header field. This is identical to + :meth:`__getitem__` except that optional *failobj* is returned if the + named header is missing (defaults to ``None``). + + Here are some additional useful methods: + + + .. method:: get_all(name, failobj=None) + + Return a list of all the values for the field named *name*. If there are + no such named headers in the message, *failobj* is returned (defaults to + ``None``). + + + .. method:: add_header(_name, _value, **_params) + + Extended header setting. This method is similar to :meth:`__setitem__` + except that additional header parameters can be provided as keyword + arguments. *_name* is the header field to add and *_value* is the + *primary* value for the header. + + For each item in the keyword argument dictionary *_params*, the key is + taken as the parameter name, with underscores converted to dashes (since + dashes are illegal in Python identifiers). Normally, the parameter will + be added as ``key="value"`` unless the value is ``None``, in which case + only the key will be added. If the value contains non-ASCII characters, + it can be specified as a three tuple in the format + ``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string naming the + charset to be used to encode the value, ``LANGUAGE`` can usually be set + to ``None`` or the empty string (see :rfc:`2231` for other possibilities), + and ``VALUE`` is the string value containing non-ASCII code points. If + a three tuple is not passed and the value contains non-ASCII characters, + it is automatically encoded in :rfc:`2231` format using a ``CHARSET`` + of ``utf-8`` and a ``LANGUAGE`` of ``None``. + + Here's an example:: + + msg.add_header('Content-Disposition', 'attachment', filename='bud.gif') + + This will add a header that looks like :: + + Content-Disposition: attachment; filename="bud.gif" + + An example with non-ASCII characters:: + + msg.add_header('Content-Disposition', 'attachment', + filename=('iso-8859-1', '', 'Fußballer.ppt')) + + Which produces :: + + Content-Disposition: attachment; filename*="iso-8859-1''Fu%DFballer.ppt" + + + .. method:: replace_header(_name, _value) + + Replace a header. Replace the first header found in the message that + matches *_name*, retaining header order and field name case. If no + matching header was found, a :exc:`KeyError` is raised. + + + .. method:: get_content_type() + + Return the message's content type. The returned string is coerced to + lower case of the form :mimetype:`maintype/subtype`. If there was no + :mailheader:`Content-Type` header in the message the default type as given + by :meth:`get_default_type` will be returned. Since according to + :rfc:`2045`, messages always have a default type, :meth:`get_content_type` + will always return a value. + + :rfc:`2045` defines a message's default type to be :mimetype:`text/plain` + unless it appears inside a :mimetype:`multipart/digest` container, in + which case it would be :mimetype:`message/rfc822`. If the + :mailheader:`Content-Type` header has an invalid type specification, + :rfc:`2045` mandates that the default type be :mimetype:`text/plain`. + + + .. method:: get_content_maintype() + + Return the message's main content type. This is the :mimetype:`maintype` + part of the string returned by :meth:`get_content_type`. + + + .. method:: get_content_subtype() + + Return the message's sub-content type. This is the :mimetype:`subtype` + part of the string returned by :meth:`get_content_type`. + + + .. method:: get_default_type() + + Return the default content type. Most messages have a default content + type of :mimetype:`text/plain`, except for messages that are subparts of + :mimetype:`multipart/digest` containers. Such subparts have a default + content type of :mimetype:`message/rfc822`. + + + .. method:: set_default_type(ctype) + + Set the default content type. *ctype* should either be + :mimetype:`text/plain` or :mimetype:`message/rfc822`, although this is not + enforced. The default content type is not stored in the + :mailheader:`Content-Type` header. + + + .. method:: get_params(failobj=None, header='content-type', unquote=True) + + Return the message's :mailheader:`Content-Type` parameters, as a list. + The elements of the returned list are 2-tuples of key/value pairs, as + split on the ``'='`` sign. The left hand side of the ``'='`` is the key, + while the right hand side is the value. If there is no ``'='`` sign in + the parameter the value is the empty string, otherwise the value is as + described in :meth:`get_param` and is unquoted if optional *unquote* is + ``True`` (the default). + + Optional *failobj* is the object to return if there is no + :mailheader:`Content-Type` header. Optional *header* is the header to + search instead of :mailheader:`Content-Type`. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by the *params* property of the individual header objects + returned by the header access methods. + + + .. method:: get_param(param, failobj=None, header='content-type', unquote=True) + + Return the value of the :mailheader:`Content-Type` header's parameter + *param* as a string. If the message has no :mailheader:`Content-Type` + header or if there is no such parameter, then *failobj* is returned + (defaults to ``None``). + + Optional *header* if given, specifies the message header to use instead of + :mailheader:`Content-Type`. + + Parameter keys are always compared case insensitively. The return value + can either be a string, or a 3-tuple if the parameter was :rfc:`2231` + encoded. When it's a 3-tuple, the elements of the value are of the form + ``(CHARSET, LANGUAGE, VALUE)``. Note that both ``CHARSET`` and + ``LANGUAGE`` can be ``None``, in which case you should consider ``VALUE`` + to be encoded in the ``us-ascii`` charset. You can usually ignore + ``LANGUAGE``. + + If your application doesn't care whether the parameter was encoded as in + :rfc:`2231`, you can collapse the parameter value by calling + :func:`email.utils.collapse_rfc2231_value`, passing in the return value + from :meth:`get_param`. This will return a suitably decoded Unicode + string when the value is a tuple, or the original string unquoted if it + isn't. For example:: + + rawparam = msg.get_param('foo') + param = email.utils.collapse_rfc2231_value(rawparam) + + In any case, the parameter value (either the returned string, or the + ``VALUE`` item in the 3-tuple) is always unquoted, unless *unquote* is set + to ``False``. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by the *params* property of the individual header objects + returned by the header access methods. + + + .. method:: set_param(param, value, header='Content-Type', requote=True, \ + charset=None, language='', replace=False) + + Set a parameter in the :mailheader:`Content-Type` header. If the + parameter already exists in the header, its value will be replaced with + *value*. If the :mailheader:`Content-Type` header as not yet been defined + for this message, it will be set to :mimetype:`text/plain` and the new + parameter value will be appended as per :rfc:`2045`. + + Optional *header* specifies an alternative header to + :mailheader:`Content-Type`, and all parameters will be quoted as necessary + unless optional *requote* is ``False`` (the default is ``True``). + + If optional *charset* is specified, the parameter will be encoded + according to :rfc:`2231`. Optional *language* specifies the RFC 2231 + language, defaulting to the empty string. Both *charset* and *language* + should be strings. + + If *replace* is ``False`` (the default) the header is moved to the + end of the list of headers. If *replace* is ``True``, the header + will be updated in place. + + .. versionchanged:: 3.4 ``replace`` keyword was added. + + + .. method:: del_param(param, header='content-type', requote=True) + + Remove the given parameter completely from the :mailheader:`Content-Type` + header. The header will be re-written in place without the parameter or + its value. All values will be quoted as necessary unless *requote* is + ``False`` (the default is ``True``). Optional *header* specifies an + alternative to :mailheader:`Content-Type`. + + + .. method:: set_type(type, header='Content-Type', requote=True) + + Set the main type and subtype for the :mailheader:`Content-Type` + header. *type* must be a string in the form :mimetype:`maintype/subtype`, + otherwise a :exc:`ValueError` is raised. + + This method replaces the :mailheader:`Content-Type` header, keeping all + the parameters in place. If *requote* is ``False``, this leaves the + existing header's quoting as is, otherwise the parameters will be quoted + (the default). + + An alternative header can be specified in the *header* argument. When the + :mailheader:`Content-Type` header is set a :mailheader:`MIME-Version` + header is also added. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by the ``make_`` and ``add_`` methods. + + + .. method:: get_filename(failobj=None) + + Return the value of the ``filename`` parameter of the + :mailheader:`Content-Disposition` header of the message. If the header + does not have a ``filename`` parameter, this method falls back to looking + for the ``name`` parameter on the :mailheader:`Content-Type` header. If + neither is found, or the header is missing, then *failobj* is returned. + The returned string will always be unquoted as per + :func:`email.utils.unquote`. + + + .. method:: get_boundary(failobj=None) + + Return the value of the ``boundary`` parameter of the + :mailheader:`Content-Type` header of the message, or *failobj* if either + the header is missing, or has no ``boundary`` parameter. The returned + string will always be unquoted as per :func:`email.utils.unquote`. + + + .. method:: set_boundary(boundary) + + Set the ``boundary`` parameter of the :mailheader:`Content-Type` header to + *boundary*. :meth:`set_boundary` will always quote *boundary* if + necessary. A :exc:`~email.errors.HeaderParseError` is raised if the + message object has no :mailheader:`Content-Type` header. + + Note that using this method is subtly different than deleting the old + :mailheader:`Content-Type` header and adding a new one with the new + boundary via :meth:`add_header`, because :meth:`set_boundary` preserves + the order of the :mailheader:`Content-Type` header in the list of + headers. However, it does *not* preserve any continuation lines which may + have been present in the original :mailheader:`Content-Type` header. + + + .. method:: get_content_charset(failobj=None) + + Return the ``charset`` parameter of the :mailheader:`Content-Type` header, + coerced to lower case. If there is no :mailheader:`Content-Type` header, or if + that header has no ``charset`` parameter, *failobj* is returned. + + Note that this method differs from :meth:`get_charset` which returns the + :class:`~email.charset.Charset` instance for the default encoding of the message body. + + + .. method:: get_charsets(failobj=None) + + Return a list containing the character set names in the message. If the + message is a :mimetype:`multipart`, then the list will contain one element + for each subpart in the payload, otherwise, it will be a list of length 1. + + Each item in the list will be a string which is the value of the + ``charset`` parameter in the :mailheader:`Content-Type` header for the + represented subpart. However, if the subpart has no + :mailheader:`Content-Type` header, no ``charset`` parameter, or is not of + the :mimetype:`text` main MIME type, then that item in the returned list + will be *failobj*. + + + .. method:: get_content_disposition() + + Return the lowercased value (without parameters) of the message's + :mailheader:`Content-Disposition` header if it has one, or ``None``. The + possible values for this method are *inline*, *attachment* or ``None`` + if the message follows :rfc:`2183`. + + .. versionadded:: 3.5 + + .. method:: walk() + + The :meth:`walk` method is an all-purpose generator which can be used to + iterate over all the parts and subparts of a message object tree, in + depth-first traversal order. You will typically use :meth:`walk` as the + iterator in a ``for`` loop; each iteration returns the next subpart. + + Here's an example that prints the MIME type of every part of a multipart + message structure: + + .. testsetup:: + + >>> from email import message_from_binary_file + >>> with open('Lib/test/test_email/data/msg_16.txt', 'rb') as f: + ... msg = message_from_binary_file(f) + >>> from email.iterators import _structure + + .. doctest:: + + >>> for part in msg.walk(): + ... print(part.get_content_type()) + multipart/report + text/plain + message/delivery-status + text/plain + text/plain + message/rfc822 + text/plain + + ``walk`` iterates over the subparts of any part where + :meth:`is_multipart` returns ``True``, even though + ``msg.get_content_maintype() == 'multipart'`` may return ``False``. We + can see this in our example by making use of the ``_structure`` debug + helper function: + + .. doctest:: + + >>> for part in msg.walk(): + ... print(part.get_content_maintype() == 'multipart'), + ... part.is_multipart()) + True True + False False + False True + False False + False False + False True + False False + >>> _structure(msg) + multipart/report + text/plain + message/delivery-status + text/plain + text/plain + message/rfc822 + text/plain + + Here the ``message`` parts are not ``multiparts``, but they do contain + subparts. ``is_multipart()`` returns ``True`` and ``walk`` descends + into the subparts. + + + :class:`Message` objects can also optionally contain two instance attributes, + which can be used when generating the plain text of a MIME message. + + + .. attribute:: preamble + + The format of a MIME document allows for some text between the blank line + following the headers, and the first multipart boundary string. Normally, + this text is never visible in a MIME-aware mail reader because it falls + outside the standard MIME armor. However, when viewing the raw text of + the message, or when viewing the message in a non-MIME aware reader, this + text can become visible. + + The *preamble* attribute contains this leading extra-armor text for MIME + documents. When the :class:`~email.parser.Parser` discovers some text + after the headers but before the first boundary string, it assigns this + text to the message's *preamble* attribute. When the + :class:`~email.generator.Generator` is writing out the plain text + representation of a MIME message, and it finds the + message has a *preamble* attribute, it will write this text in the area + between the headers and the first boundary. See :mod:`email.parser` and + :mod:`email.generator` for details. + + Note that if the message object has no preamble, the *preamble* attribute + will be ``None``. + + + .. attribute:: epilogue + + The *epilogue* attribute acts the same way as the *preamble* attribute, + except that it contains text that appears between the last boundary and + the end of the message. + + You do not need to set the epilogue to the empty string in order for the + :class:`~email.generator.Generator` to print a newline at the end of the + file. + + + .. attribute:: defects + + The *defects* attribute contains a list of all the problems found when + parsing this message. See :mod:`email.errors` for a detailed description + of the possible parsing defects. diff --git a/Doc/library/email.contentmanager.rst b/Doc/library/email.contentmanager.rst index a9c078bd60..57743d5a29 100644 --- a/Doc/library/email.contentmanager.rst +++ b/Doc/library/email.contentmanager.rst @@ -7,251 +7,12 @@ .. moduleauthor:: R. David Murray <rdmurray@bitdance.com> .. sectionauthor:: R. David Murray <rdmurray@bitdance.com> -.. versionadded:: 3.4 - as a :term:`provisional module <provisional package>`. - **Source code:** :source:`Lib/email/contentmanager.py` -.. note:: - - The contentmanager module has been included in the standard library on a - :term:`provisional basis <provisional package>`. Backwards incompatible - changes (up to and including removal of the module) may occur if deemed - necessary by the core developers. - --------------- - -The :mod:`~email.message` module provides a class that can represent an -arbitrary email message. That basic message model has a useful and flexible -API, but it provides only a lower-level API for interacting with the generic -parts of a message (the headers, generic header parameters, and the payload, -which may be a list of sub-parts). This module provides classes and tools -that provide an enhanced and extensible API for dealing with various specific -types of content, including the ability to retrieve the content of the message -as a specialized object type rather than as a simple bytes object. The module -automatically takes care of the RFC-specified MIME details (required headers -and parameters, etc.) for the certain common content types content properties, -and support for additional types can be added by an application using the -extension mechanisms. - -This module defines the eponymous "Content Manager" classes. The base -:class:`.ContentManager` class defines an API for registering content -management functions which extract data from ``Message`` objects or insert data -and headers into ``Message`` objects, thus providing a way of converting -between ``Message`` objects containing data and other representations of that -data (Python data types, specialized Python objects, external files, etc). The -module also defines one concrete content manager: :data:`raw_data_manager` -converts between MIME content types and ``str`` or ``bytes`` data. It also -provides a convenient API for managing the MIME parameters when inserting -content into ``Message``\ s. It also handles inserting and extracting -``Message`` objects when dealing with the ``message/rfc822`` content type. - -Another part of the enhanced interface is subclasses of -:class:`~email.message.Message` that provide new convenience API functions, -including convenience methods for calling the Content Managers derived from -this module. - -.. note:: - - Although :class:`.EmailMessage` and :class:`.MIMEPart` are currently - documented in this module because of the provisional nature of the code, the - implementation lives in the :mod:`email.message` module. - -.. currentmodule:: email.message - -.. class:: EmailMessage(policy=default) - - If *policy* is specified (it must be an instance of a :mod:`~email.policy` - class) use the rules it specifies to udpate and serialize the representation - of the message. If *policy* is not set, use the - :class:`~email.policy.default` policy, which follows the rules of the email - RFCs except for line endings (instead of the RFC mandated ``\r\n``, it uses - the Python standard ``\n`` line endings). For more information see the - :mod:`~email.policy` documentation. - - This class is a subclass of :class:`~email.message.Message`. It adds - the following methods: - - - .. method:: is_attachment - - Return ``True`` if there is a :mailheader:`Content-Disposition` header - and its (case insensitive) value is ``attachment``, ``False`` otherwise. - - .. versionchanged:: 3.4.2 - is_attachment is now a method instead of a property, for consistency - with :meth:`~email.message.Message.is_multipart`. - - - .. method:: get_body(preferencelist=('related', 'html', 'plain')) - - Return the MIME part that is the best candidate to be the "body" of the - message. - - *preferencelist* must be a sequence of strings from the set ``related``, - ``html``, and ``plain``, and indicates the order of preference for the - content type of the part returned. - - Start looking for candidate matches with the object on which the - ``get_body`` method is called. - - If ``related`` is not included in *preferencelist*, consider the root - part (or subpart of the root part) of any related encountered as a - candidate if the (sub-)part matches a preference. - - When encountering a ``multipart/related``, check the ``start`` parameter - and if a part with a matching :mailheader:`Content-ID` is found, consider - only it when looking for candidate matches. Otherwise consider only the - first (default root) part of the ``multipart/related``. - - If a part has a :mailheader:`Content-Disposition` header, only consider - the part a candidate match if the value of the header is ``inline``. - - If none of the candidates matches any of the preferences in - *preferneclist*, return ``None``. - - Notes: (1) For most applications the only *preferencelist* combinations - that really make sense are ``('plain',)``, ``('html', 'plain')``, and the - default, ``('related', 'html', 'plain')``. (2) Because matching starts - with the object on which ``get_body`` is called, calling ``get_body`` on - a ``multipart/related`` will return the object itself unless - *preferencelist* has a non-default value. (3) Messages (or message parts) - that do not specify a :mailheader:`Content-Type` or whose - :mailheader:`Content-Type` header is invalid will be treated as if they - are of type ``text/plain``, which may occasionally cause ``get_body`` to - return unexpected results. - - - .. method:: iter_attachments() - - Return an iterator over all of the parts of the message that are not - candidate "body" parts. That is, skip the first occurrence of each of - ``text/plain``, ``text/html``, ``multipart/related``, or - ``multipart/alternative`` (unless they are explicitly marked as - attachments via :mailheader:`Content-Disposition: attachment`), and - return all remaining parts. When applied directly to a - ``multipart/related``, return an iterator over the all the related parts - except the root part (ie: the part pointed to by the ``start`` parameter, - or the first part if there is no ``start`` parameter or the ``start`` - parameter doesn't match the :mailheader:`Content-ID` of any of the - parts). When applied directly to a ``multipart/alternative`` or a - non-``multipart``, return an empty iterator. - - - .. method:: iter_parts() - - Return an iterator over all of the immediate sub-parts of the message, - which will be empty for a non-``multipart``. (See also - :meth:`~email.message.walk`.) - - - .. method:: get_content(*args, content_manager=None, **kw) - - Call the ``get_content`` method of the *content_manager*, passing self - as the message object, and passing along any other arguments or keywords - as additional arguments. If *content_manager* is not specified, use - the ``content_manager`` specified by the current :mod:`~email.policy`. +------------ +.. versionadded:: 3.6 [1]_ - .. method:: set_content(*args, content_manager=None, **kw) - - Call the ``set_content`` method of the *content_manager*, passing self - as the message object, and passing along any other arguments or keywords - as additional arguments. If *content_manager* is not specified, use - the ``content_manager`` specified by the current :mod:`~email.policy`. - - - .. method:: make_related(boundary=None) - - Convert a non-``multipart`` message into a ``multipart/related`` message, - moving any existing :mailheader:`Content-` headers and payload into a - (new) first part of the ``multipart``. If *boundary* is specified, use - it as the boundary string in the multipart, otherwise leave the boundary - to be automatically created when it is needed (for example, when the - message is serialized). - - - .. method:: make_alternative(boundary=None) - - Convert a non-``multipart`` or a ``multipart/related`` into a - ``multipart/alternative``, moving any existing :mailheader:`Content-` - headers and payload into a (new) first part of the ``multipart``. If - *boundary* is specified, use it as the boundary string in the multipart, - otherwise leave the boundary to be automatically created when it is - needed (for example, when the message is serialized). - - - .. method:: make_mixed(boundary=None) - - Convert a non-``multipart``, a ``multipart/related``, or a - ``multipart-alternative`` into a ``multipart/mixed``, moving any existing - :mailheader:`Content-` headers and payload into a (new) first part of the - ``multipart``. If *boundary* is specified, use it as the boundary string - in the multipart, otherwise leave the boundary to be automatically - created when it is needed (for example, when the message is serialized). - - - .. method:: add_related(*args, content_manager=None, **kw) - - If the message is a ``multipart/related``, create a new message - object, pass all of the arguments to its :meth:`set_content` method, - and :meth:`~email.message.Message.attach` it to the ``multipart``. If - the message is a non-``multipart``, call :meth:`make_related` and then - proceed as above. If the message is any other type of ``multipart``, - raise a :exc:`TypeError`. If *content_manager* is not specified, use - the ``content_manager`` specified by the current :mod:`~email.policy`. - If the added part has no :mailheader:`Content-Disposition` header, - add one with the value ``inline``. - - - .. method:: add_alternative(*args, content_manager=None, **kw) - - If the message is a ``multipart/alternative``, create a new message - object, pass all of the arguments to its :meth:`set_content` method, and - :meth:`~email.message.Message.attach` it to the ``multipart``. If the - message is a non-``multipart`` or ``multipart/related``, call - :meth:`make_alternative` and then proceed as above. If the message is - any other type of ``multipart``, raise a :exc:`TypeError`. If - *content_manager* is not specified, use the ``content_manager`` specified - by the current :mod:`~email.policy`. - - - .. method:: add_attachment(*args, content_manager=None, **kw) - - If the message is a ``multipart/mixed``, create a new message object, - pass all of the arguments to its :meth:`set_content` method, and - :meth:`~email.message.Message.attach` it to the ``multipart``. If the - message is a non-``multipart``, ``multipart/related``, or - ``multipart/alternative``, call :meth:`make_mixed` and then proceed as - above. If *content_manager* is not specified, use the ``content_manager`` - specified by the current :mod:`~email.policy`. If the added part - has no :mailheader:`Content-Disposition` header, add one with the value - ``attachment``. This method can be used both for explicit attachments - (:mailheader:`Content-Disposition: attachment` and ``inline`` attachments - (:mailheader:`Content-Disposition: inline`), by passing appropriate - options to the ``content_manager``. - - - .. method:: clear() - - Remove the payload and all of the headers. - - - .. method:: clear_content() - - Remove the payload and all of the :exc:`Content-` headers, leaving - all other headers intact and in their original order. - - -.. class:: MIMEPart(policy=default) - - This class represents a subpart of a MIME message. It is identical to - :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are - added when :meth:`~EmailMessage.set_content` is called, since sub-parts do - not need their own :mailheader:`MIME-Version` headers. - - -.. currentmodule:: email.contentmanager .. class:: ContentManager() @@ -362,7 +123,7 @@ Currently the email package provides only one concrete content manager, set_content(msg, <'bytes'>, maintype, subtype, cte="base64", \ disposition=None, filename=None, cid=None, \ params=None, headers=None) - set_content(msg, <'Message'>, cte=None, \ + set_content(msg, <'EmailMessage'>, cte=None, \ disposition=None, filename=None, cid=None, \ params=None, headers=None) set_content(msg, <'list'>, subtype='mixed', \ @@ -378,14 +139,14 @@ Currently the email package provides only one concrete content manager, subtype to *subtype* if it is specified, or ``plain`` if it is not. * For ``bytes``, use the specified *maintype* and *subtype*, or raise a :exc:`TypeError` if they are not specified. - * For :class:`~email.message.Message` objects, set the maintype to - ``message``, and set the subtype to *subtype* if it is specified - or ``rfc822`` if it is not. If *subtype* is ``partial``, raise an - error (``bytes`` objects must be used to construct - ``message/partial`` parts). + * For :class:`~email.message.EmailMessage` objects, set the maintype + to ``message``, and set the subtype to *subtype* if it is + specified or ``rfc822`` if it is not. If *subtype* is + ``partial``, raise an error (``bytes`` objects must be used to + construct ``message/partial`` parts). * For *<'list'>*, which should be a list of - :class:`~email.message.Message` objects, set the ``maintype`` to - ``multipart``, and the ``subtype`` to *subtype* if it is + :class:`~email.message.EmailMessage` objects, set the ``maintype`` + to ``multipart``, and the ``subtype`` to *subtype* if it is specified, and ``mixed`` if it is not. If the message parts in the *<'list'>* have :mailheader:`MIME-Version` headers, remove them. @@ -397,32 +158,35 @@ Currently the email package provides only one concrete content manager, If *cte* is set, encode the payload using the specified content transfer encoding, and set the :mailheader:`Content-Transfer-Endcoding` header to - that value. For ``str`` objects, if it is not set use heuristics to - determine the most compact encoding. Possible values for *cte* are - ``quoted-printable``, ``base64``, ``7bit``, ``8bit``, and ``binary``. - If the input cannot be encoded in the specified encoding (eg: ``7bit``), - raise a :exc:`ValueError`. For :class:`~email.message.Message`, per - :rfc:`2046`, raise an error if a *cte* of ``quoted-printable`` or - ``base64`` is requested for *subtype* ``rfc822``, and for any *cte* - other than ``7bit`` for *subtype* ``external-body``. For - ``message/rfc822``, use ``8bit`` if *cte* is not specified. For all - other values of *subtype*, use ``7bit``. + that value. Possible values for *cte* are ``quoted-printable``, + ``base64``, ``7bit``, ``8bit``, and ``binary``. If the input cannot be + encoded in the specified encoding (for example, specifying a *cte* of + ``7bit`` for an input that contains non-ASCII values), raise a + :exc:`ValueError`. + + * For ``str`` objects, if *cte* is not set use heuristics to + determine the most compact encoding. + * For :class:`~email.message.EmailMessage`, per :rfc:`2046`, raise + an error if a *cte* of ``quoted-printable`` or ``base64`` is + requested for *subtype* ``rfc822``, and for any *cte* other than + ``7bit`` for *subtype* ``external-body``. For + ``message/rfc822``, use ``8bit`` if *cte* is not specified. For + all other values of *subtype*, use ``7bit``. .. note:: A *cte* of ``binary`` does not actually work correctly yet. - The ``Message`` object as modified by ``set_content`` is correct, but - :class:`~email.generator.BytesGenerator` does not serialize it - correctly. + The ``EmailMessage`` object as modified by ``set_content`` is + correct, but :class:`~email.generator.BytesGenerator` does not + serialize it correctly. If *disposition* is set, use it as the value of the :mailheader:`Content-Disposition` header. If not specified, and *filename* is specified, add the header with the value ``attachment``. - If it is not specified and *filename* is also not specified, do not add - the header. The only valid values for *disposition* are ``attachment`` - and ``inline``. + If *disposition* is not specified and *filename* is also not specified, + do not add the header. The only valid values for *disposition* are + ``attachment`` and ``inline``. If *filename* is specified, use it as the value of the ``filename`` - parameter of the :mailheader:`Content-Disposition` header. There is no - default. + parameter of the :mailheader:`Content-Disposition` header. If *cid* is specified, add a :mailheader:`Content-ID` header with *cid* as its value. @@ -435,3 +199,9 @@ Currently the email package provides only one concrete content manager, ``headername: headervalue`` or a list of ``header`` objects (distinguished from strings by having a ``name`` attribute), add the headers to *msg*. + + +.. rubric:: Footnotes + +.. [1] Oringally added in 3.4 as a :term:`provisional module <provisional + package>` diff --git a/Doc/library/email.encoders.rst b/Doc/library/email.encoders.rst index 9d7f9bf5e9..e24ac7b90d 100644 --- a/Doc/library/email.encoders.rst +++ b/Doc/library/email.encoders.rst @@ -8,6 +8,12 @@ -------------- +This module is part of the legacy (``Compat32``) email API. In the +new API the functionality is provided by the *cte* parameter of +the :meth:`~email.message.EmailMessage.set_content` method. + +The remaining text in this section is the original documentation of the module. + When creating :class:`~email.message.Message` objects from scratch, you often need to encode the payloads for transport through compliant mail servers. This is especially true for :mimetype:`image/\*` and :mimetype:`text/\*` type messages diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst index 8470783e81..2d0d1923cd 100644 --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -20,33 +20,27 @@ The following exception classes are defined in the :mod:`email.errors` module: .. exception:: MessageParseError() - This is the base class for exceptions raised by the :class:`~email.parser.Parser` - class. It is derived from :exc:`MessageError`. + This is the base class for exceptions raised by the + :class:`~email.parser.Parser` class. It is derived from + :exc:`MessageError`. This class is also used internally by the parser used + by :mod:`~email.headerregistry`. .. exception:: HeaderParseError() - Raised under some error conditions when parsing the :rfc:`2822` headers of a - message, this class is derived from :exc:`MessageParseError`. It can be raised - from the :meth:`Parser.parse <email.parser.Parser.parse>` or - :meth:`Parser.parsestr <email.parser.Parser.parsestr>` methods. - - Situations where it can be raised include finding an envelope header after the - first :rfc:`2822` header of the message, finding a continuation line before the - first :rfc:`2822` header is found, or finding a line in the headers which is - neither a header or a continuation line. + Raised under some error conditions when parsing the :rfc:`5322` headers of a + message, this class is derived from :exc:`MessageParseError`. The + :meth:`~email.message.EmailMessage.set_boundary` method will raise this + error if the content type is unknown when the method is called. + :class:`~email.header.Header` may raise this error for certain base64 + decoding errors, and when an attempt is made to create a header that appears + to contain an embedded header (that is, there is what is supposed to be a + continuation line that has no leading whitespace and looks like a header). .. exception:: BoundaryError() - Raised under some error conditions when parsing the :rfc:`2822` headers of a - message, this class is derived from :exc:`MessageParseError`. It can be raised - from the :meth:`Parser.parse <email.parser.Parser.parse>` or - :meth:`Parser.parsestr <email.parser.Parser.parsestr>` methods. - - Situations where it can be raised include not being able to find the starting or - terminating boundary in a :mimetype:`multipart/\*` message when strict parsing - is used. + Deprecated and no longer used. .. exception:: MultipartConversionError() @@ -64,14 +58,14 @@ The following exception classes are defined in the :mod:`email.errors` module: :class:`~email.mime.nonmultipart.MIMENonMultipart` (e.g. :class:`~email.mime.image.MIMEImage`). -Here's the list of the defects that the :class:`~email.parser.FeedParser` + +Here is the list of the defects that the :class:`~email.parser.FeedParser` can find while parsing messages. Note that the defects are added to the message where the problem was found, so for example, if a message nested inside a :mimetype:`multipart/alternative` had a malformed header, that nested message object would have a defect, but the containing messages would not. -All defect classes are subclassed from :class:`email.errors.MessageDefect`, but -this class is *not* an exception! +All defect classes are subclassed from :class:`email.errors.MessageDefect`. * :class:`NoBoundaryInMultipartDefect` -- A message claimed to be a multipart, but had no :mimetype:`boundary` parameter. diff --git a/Doc/library/email-examples.rst b/Doc/library/email.examples.rst index ad93b5ccb8..84e9aee0bc 100644 --- a/Doc/library/email-examples.rst +++ b/Doc/library/email.examples.rst @@ -6,13 +6,14 @@ Here are a few examples of how to use the :mod:`email` package to read, write, and send simple email messages, as well as more complex MIME messages. -First, let's see how to create and send a simple text message: +First, let's see how to create and send a simple text message (both the +text content and the addresses may contain unicode characters): .. literalinclude:: ../includes/email-simple.py -And parsing RFC822 headers can easily be done by the parse(filename) or -parsestr(message_as_string) methods of the Parser() class: +Parsing RFC822 headers can easily be done by the using the classes +from the :mod:`~email.parser` module: .. literalinclude:: ../includes/email-headers.py @@ -34,30 +35,19 @@ above, into a directory of files: .. literalinclude:: ../includes/email-unpack.py + Here's an example of how to create an HTML message with an alternative plain -text version: [2]_ +text version. To make things a bit more interesting, we include a related +image in the html part, and we save a copy of what we are going to send to +disk, as well as sending it. .. literalinclude:: ../includes/email-alternative.py -.. _email-contentmanager-api-examples: - -Examples using the Provisional API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Here is a reworking of the last example using the provisional API. To make -things a bit more interesting, we include a related image in the html part, and -we save a copy of what we are going to send to disk, as well as sending it. - -This example also shows how easy it is to include non-ASCII, and simplifies the -sending of the message using the :meth:`.send_message` method of the -:mod:`smtplib` module. - -.. literalinclude:: ../includes/email-alternative-new-api.py - -If we were instead sent the message from the last example, here is one -way we could process it: +If we were sent the message from the last example, here is one way we could +process it: -.. literalinclude:: ../includes/email-read-alternative-new-api.py +.. literalinclude:: ../includes/email-read-alternative.py Up to the prompt, the output from the above is: @@ -75,4 +65,3 @@ Up to the prompt, the output from the above is: .. rubric:: Footnotes .. [1] Thanks to Matthew Dixon Cowles for the original inspiration and examples. -.. [2] Contributed by Martin Matejek. diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst index d596ed8d85..ab0fbc29d1 100644 --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -8,210 +8,244 @@ -------------- -One of the most common tasks is to generate the flat text of the email message -represented by a message object structure. You will need to do this if you want -to send your message via the :mod:`smtplib` module or the :mod:`nntplib` module, -or print the message on the console. Taking a message object structure and -producing a flat text document is the job of the :class:`Generator` class. - -Again, as with the :mod:`email.parser` module, you aren't limited to the -functionality of the bundled generator; you could write one from scratch -yourself. However the bundled generator knows how to generate most email in a -standards-compliant way, should handle MIME and non-MIME email messages just -fine, and is designed so that the transformation from flat text, to a message -structure via the :class:`~email.parser.Parser` class, and back to flat text, -is idempotent (the input is identical to the output) [#]_. On the other hand, -using the Generator on a :class:`~email.message.Message` constructed by program -may result in changes to the :class:`~email.message.Message` object as defaults -are filled in. - -:class:`bytes` output can be generated using the :class:`BytesGenerator` class. -If the message object structure contains non-ASCII bytes, this generator's -:meth:`~BytesGenerator.flatten` method will emit the original bytes. Parsing a -binary message and then flattening it with :class:`BytesGenerator` should be -idempotent for standards compliant messages. - -Here are the public methods of the :class:`Generator` class, imported from the -:mod:`email.generator` module: - - -.. class:: Generator(outfp, mangle_from_=True, maxheaderlen=78, *, policy=None) - - The constructor for the :class:`Generator` class takes a :term:`file-like object` - called *outfp* for an argument. *outfp* must support the :meth:`write` method - and be usable as the output file for the :func:`print` function. - - Optional *mangle_from_* is a flag that, when ``True``, puts a ``>`` character in - front of any line in the body that starts exactly as ``From``, i.e. ``From`` - followed by a space at the beginning of the line. This is the only guaranteed - portable way to avoid having such lines be mistaken for a Unix mailbox format - envelope header separator (see `WHY THE CONTENT-LENGTH FORMAT IS BAD - <https://www.jwz.org/doc/content-length.html>`_ for details). *mangle_from_* - defaults to ``True``, but you might want to set this to ``False`` if you are not - writing Unix mailbox format files. - - Optional *maxheaderlen* specifies the longest length for a non-continued header. - When a header line is longer than *maxheaderlen* (in characters, with tabs - expanded to 8 spaces), the header will be split as defined in the - :class:`~email.header.Header` class. Set to zero to disable header wrapping. - The default is 78, as recommended (but not required) by :rfc:`2822`. - - The *policy* keyword specifies a :mod:`~email.policy` object that controls a - number of aspects of the generator's operation. If no *policy* is specified, - then the *policy* attached to the message object passed to :attr:`flatten` - is used. +One of the most common tasks is to generate the flat (serialized) version of +the email message represented by a message object structure. You will need to +do this if you want to send your message via :meth:`smtplib.SMTP.sendmail` or +the :mod:`nntplib` module, or print the message on the console. Taking a +message object structure and producing a serialized representation is the job +of the generator classes. + +As with the :mod:`email.parser` module, you aren't limited to the functionality +of the bundled generator; you could write one from scratch yourself. However +the bundled generator knows how to generate most email in a standards-compliant +way, should handle MIME and non-MIME email messages just fine, and is designed +so that the bytes-oriented parsing and generation operations are inverses, +assuming the same non-transforming :mod:`~email.policy` is used for both. That +is, parsing the serialized byte stream via the +:class:`~email.parser.BytesParser` class and then regenerating the serialized +byte stream using :class:`BytesGenerator` should produce output identical to +the input [#]_. (On the other hand, using the generator on an +:class:`~email.message.EmailMessage` constructed by program may result in +changes to the :class:`~email.message.EmailMessage` object as defaults are +filled in.) + +The :class:`Generator` class can be used to flatten a message into a text (as +opposed to binary) serialized representation, but since Unicode cannot +represent binary data directly, the message is of necessity transformed into +something that contains only ASCII characters, using the standard email RFC +Content Transfer Encoding techniques for encoding email messages for transport +over channels that are not "8 bit clean". + + +.. class:: BytesGenerator(outfp, mangle_from_=None, maxheaderlen=None, *, \ + policy=None) - .. versionchanged:: 3.3 Added the *policy* keyword. + Return a :class:`BytesGenerator` object that will write any message provided + to the :meth:`flatten` method, or any surrogateescape encoded text provided + to the :meth:`write` method, to the :term:`file-like object` *outfp*. + *outfp* must support a ``write`` method that accepts binary data. + + If optional *mangle_from_* is ``True``, put a ``>`` character in front of + any line in the body that starts with the exact string ``"From "``, that is + ``From`` followed by a space at the beginning of a line. *mangle_from_* + defaults to the value of the :attr:`~email.policy.Policy.mangle_from_` + setting of the *policy* (which is ``True`` for the + :data:`~email.policy.compat32` policy and ``False`` for all others). + *mangle_from_* is intended for use when messages are stored in unix mbox + format (see :mod:`mailbox` and `WHY THE CONTENT-LENGTH FORMAT IS BAD + <http://www.jwz.org/doc/content-length.html>`_). + + If *maxheaderlen* is not ``None``, refold any header lines that are longer + than *maxheaderlen*, or if ``0``, do not rewrap any headers. If + *manheaderlen* is ``None`` (the default), wrap headers and other message + lines according to the *policy* settings. + + If *policy* is specified, use that policy to control message generation. If + *policy* is ``None`` (the default), use the policy associated with the + :class:`~email.message.Message` or :class:`~email.message.EmailMessage` + object passed to ``flatten`` to control the message generation. See + :mod:`email.policy` for details on what *policy* controls. - The other public :class:`Generator` methods are: + .. versionadded:: 3.2 + .. versionchanged:: 3.3 Added the *policy* keyword. - .. method:: flatten(msg, unixfrom=False, linesep=None) + .. versionchanged:: 3.6 The default behavior of the *mangle_from_* + and *maxheaderlen* parameters is to follow the policy. - Print the textual representation of the message object structure rooted at - *msg* to the output file specified when the :class:`Generator` instance - was created. Subparts are visited depth-first and the resulting text will - be properly MIME encoded. - Optional *unixfrom* is a flag that forces the printing of the envelope - header delimiter before the first :rfc:`2822` header of the root message - object. If the root object has no envelope header, a standard one is - crafted. By default, this is set to ``False`` to inhibit the printing of - the envelope delimiter. + .. method:: flatten(msg, unixfrom=False, linesep=None) + Print the textual representation of the message object structure rooted + at *msg* to the output file specified when the :class:`BytesGenerator` + instance was created. + + If the :mod:`~email.policy` option :attr:`~email.policy.Policy.cte_type` + is ``8bit`` (the default), copy any headers in the original parsed + message that have not been modified to the output with any bytes with the + high bit set reproduced as in the original, and preserve the non-ASCII + :mailheader:`Content-Transfer-Encoding` of any body parts that have them. + If ``cte_type`` is ``7bit``, convert the bytes with the high bit set as + needed using an ASCII-compatible :mailheader:`Content-Transfer-Encoding`. + That is, transform parts with non-ASCII + :mailheader:`Cotnent-Transfer-Encoding` + (:mailheader:`Content-Transfer-Encoding: 8bit`) to an ASCII compatibile + :mailheader:`Content-Transfer-Encoding`, and encode RFC-invalid non-ASCII + bytes in headers using the MIME ``unknown-8bit`` character set, thus + rendering them RFC-compliant. + + .. XXX: There should be an option that just does the RFC + compliance transformation on headers but leaves CTE 8bit parts alone. + + If *unixfrom* is ``True``, print the envelope header delimiter used by + the Unix mailbox format (see :mod:`mailbox`) before the first of the + :rfc:`5322` headers of the root message object. If the root object has + no envelope header, craft a standard one. The default is ``False``. Note that for subparts, no envelope header is ever printed. - Optional *linesep* specifies the line separator character used to - terminate lines in the output. If specified it overrides the value - specified by the *msg*\'s or ``Generator``\'s ``policy``. + If *linesep* is not ``None``, use it as the separator character between + all the lines of the flattened message. If *linesep* is ``None`` (the + default), use the value specified in the *policy*. - Because strings cannot represent non-ASCII bytes, if the policy that - applies when ``flatten`` is run has :attr:`~email.policy.Policy.cte_type` - set to ``8bit``, ``Generator`` will operate as if it were set to - ``7bit``. This means that messages parsed with a Bytes parser that have - a :mailheader:`Content-Transfer-Encoding` of ``8bit`` will be converted - to a use a ``7bit`` Content-Transfer-Encoding. Non-ASCII bytes in the - headers will be :rfc:`2047` encoded with a charset of ``unknown-8bit``. + .. XXX: flatten should take a *policy* keyword. - .. versionchanged:: 3.2 - Added support for re-encoding ``8bit`` message bodies, and the - *linesep* argument. .. method:: clone(fp) - Return an independent clone of this :class:`Generator` instance with the - exact same options. - - .. method:: write(s) - - Write the string *s* to the underlying file object, i.e. *outfp* passed to - :class:`Generator`'s constructor. This provides just enough file-like API - for :class:`Generator` instances to be used in the :func:`print` function. + Return an independent clone of this :class:`BytesGenerator` instance with + the exact same option settings, and *fp* as the new *outfp*. -As a convenience, see the :class:`~email.message.Message` methods -:meth:`~email.message.Message.as_string` and ``str(aMessage)``, a.k.a. -:meth:`~email.message.Message.__str__`, which simplify the generation of a -formatted string representation of a message object. For more detail, see -:mod:`email.message`. -.. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78, *, \ - policy=None) + .. method:: write(s) - The constructor for the :class:`BytesGenerator` class takes a binary - :term:`file-like object` called *outfp* for an argument. *outfp* must - support a :meth:`write` method that accepts binary data. + Encode *s* using the ``ASCII`` codec and the ``surrogateescape`` error + handler, and pass it to the *write* method of the *outfp* passed to the + :class:`BytesGenerator`'s constructor. - Optional *mangle_from_* is a flag that, when ``True``, puts a ``>`` - character in front of any line in the body that starts exactly as ``From``, - i.e. ``From`` followed by a space at the beginning of the line. This is the - only guaranteed portable way to avoid having such lines be mistaken for a - Unix mailbox format envelope header separator (see `WHY THE CONTENT-LENGTH - FORMAT IS BAD <https://www.jwz.org/doc/content-length.html>`_ for details). - *mangle_from_* defaults to ``True``, but you might want to set this to - ``False`` if you are not writing Unix mailbox format files. - Optional *maxheaderlen* specifies the longest length for a non-continued - header. When a header line is longer than *maxheaderlen* (in characters, - with tabs expanded to 8 spaces), the header will be split as defined in the - :class:`~email.header.Header` class. Set to zero to disable header - wrapping. The default is 78, as recommended (but not required) by - :rfc:`2822`. +As a convenience, :class:`~email.message.EmailMessage` provides the methods +:meth:`~email.message.EmailMessage.as_bytes` and ``bytes(aMessage)`` (a.k.a. +:meth:`~email.message.EmailMessage.__bytes__`), which simplify the generation of +a serialized binary representation of a message object. For more detail, see +:mod:`email.message`. - The *policy* keyword specifies a :mod:`~email.policy` object that controls a - number of aspects of the generator's operation. If no *policy* is specified, - then the *policy* attached to the message object passed to :attr:`flatten` - is used. +Because strings cannot represent binary data, the :class:`Generator` class must +convert any binary data in any message it flattens to an ASCII compatible +format, by converting them to an ASCII compatible +:mailheader:`Content-Transfer_Encoding`. Using the terminology of the email +RFCs, you can think of this as :class:`Generator` serializing to an I/O stream +that is not "8 bit clean". In other words, most applications will want +to be using :class:`BytesGenerator`, and not :class:`Generator`. + +.. class:: Generator(outfp, mangle_from_=None, maxheaderlen=None, *, \ + policy=None) + + Return a :class:`Generator` object that will write any message provided + to the :meth:`flatten` method, or any text provided to the :meth:`write` + method, to the :term:`file-like object` *outfp*. *outfp* must support a + ``write`` method that accepts string data. + + If optional *mangle_from_* is ``True``, put a ``>`` character in front of + any line in the body that starts with the exact string ``"From "``, that is + ``From`` followed by a space at the beginning of a line. *mangle_from_* + defaults to the value of the :attr:`~email.policy.Policy.mangle_from_` + setting of the *policy* (which is ``True`` for the + :data:`~email.policy.compat32` policy and ``False`` for all others). + *mangle_from_* is intended for use when messages are stored in unix mbox + format (see :mod:`mailbox` and `WHY THE CONTENT-LENGTH FORMAT IS BAD + <http://www.jwz.org/doc/content-length.html>`_). + + If *maxheaderlen* is not ``None``, refold any header lines that are longer + than *maxheaderlen*, or if ``0``, do not rewrap any headers. If + *manheaderlen* is ``None`` (the default), wrap headers and other message + lines according to the *policy* settings. + + If *policy* is specified, use that policy to control message generation. If + *policy* is ``None`` (the default), use the policy associated with the + :class:`~email.message.Message` or :class:`~email.message.EmailMessage` + object passed to ``flatten`` to control the message generation. See + :mod:`email.policy` for details on what *policy* controls. .. versionchanged:: 3.3 Added the *policy* keyword. - The other public :class:`BytesGenerator` methods are: + .. versionchanged:: 3.6 The default behavior of the *mangle_from_* + and *maxheaderlen* parameters is to follow the policy. .. method:: flatten(msg, unixfrom=False, linesep=None) Print the textual representation of the message object structure rooted - at *msg* to the output file specified when the :class:`BytesGenerator` - instance was created. Subparts are visited depth-first and the resulting - text will be properly MIME encoded. If the :mod:`~email.policy` option - :attr:`~email.policy.Policy.cte_type` is ``8bit`` (the default), - then any bytes with the high bit set in the original parsed message that - have not been modified will be copied faithfully to the output. If - ``cte_type`` is ``7bit``, the bytes will be converted as needed - using an ASCII-compatible Content-Transfer-Encoding. In particular, - RFC-invalid non-ASCII bytes in headers will be encoded using the MIME - ``unknown-8bit`` character set, thus rendering them RFC-compliant. - - .. XXX: There should be a complementary option that just does the RFC - compliance transformation but leaves CTE 8bit parts alone. - - Messages parsed with a Bytes parser that have a - :mailheader:`Content-Transfer-Encoding` of 8bit will be reconstructed - as 8bit if they have not been modified. - - Optional *unixfrom* is a flag that forces the printing of the envelope - header delimiter before the first :rfc:`2822` header of the root message - object. If the root object has no envelope header, a standard one is - crafted. By default, this is set to ``False`` to inhibit the printing of - the envelope delimiter. - + at *msg* to the output file specified when the :class:`Generator` + instance was created. + + If the :mod:`~email.policy` option :attr:`~email.policy.Policy.cte_type` + is ``8bit``, generate the message as if the option were set to ``7bit``. + (This is required because strings cannot represent non-ASCII bytes.) + Convert any bytes with the high bit set as needed using an + ASCII-compatible :mailheader:`Content-Transfer-Encoding`. That is, + transform parts with non-ASCII :mailheader:`Cotnent-Transfer-Encoding` + (:mailheader:`Content-Transfer-Encoding: 8bit`) to an ASCII compatibile + :mailheader:`Content-Transfer-Encoding`, and encode RFC-invalid non-ASCII + bytes in headers using the MIME ``unknown-8bit`` character set, thus + rendering them RFC-compliant. + + If *unixfrom* is ``True``, print the envelope header delimiter used by + the Unix mailbox format (see :mod:`mailbox`) before the first of the + :rfc:`5322` headers of the root message object. If the root object has + no envelope header, craft a standard one. The default is ``False``. Note that for subparts, no envelope header is ever printed. - Optional *linesep* specifies the line separator character used to - terminate lines in the output. If specified it overrides the value - specified by the ``Generator``\ or *msg*\ 's ``policy``. + If *linesep* is not ``None``, use it as the separator character between + all the lines of the flattened message. If *linesep* is ``None`` (the + default), use the value specified in the *policy*. + + .. XXX: flatten should take a *policy* keyword. + + .. versionchanged:: 3.2 + Added support for re-encoding ``8bit`` message bodies, and the + *linesep* argument. + .. method:: clone(fp) - Return an independent clone of this :class:`BytesGenerator` instance with - the exact same options. + Return an independent clone of this :class:`Generator` instance with the + exact same options, and *fp* as the new *outfp*. + .. method:: write(s) - Write the string *s* to the underlying file object. *s* is encoded using - the ``ASCII`` codec and written to the *write* method of the *outfp* - *outfp* passed to the :class:`BytesGenerator`'s constructor. This - provides just enough file-like API for :class:`BytesGenerator` instances - to be used in the :func:`print` function. + Write *s* to the *write* method of the *outfp* passed to the + :class:`Generator`'s constructor. This provides just enough file-like + API for :class:`Generator` instances to be used in the :func:`print` + function. - .. versionadded:: 3.2 -The :mod:`email.generator` module also provides a derived class, called -:class:`DecodedGenerator` which is like the :class:`Generator` base class, -except that non-\ :mimetype:`text` parts are substituted with a format string -representing the part. +As a convenience, :class:`~email.message.EmailMessage` provides the methods +:meth:`~email.message.EmailMessage.as_string` and ``str(aMessage)`` (a.k.a. +:meth:`~email.message.EmailMessage.__str__`), which simplify the generation of +a formatted string representation of a message object. For more detail, see +:mod:`email.message`. + +The :mod:`email.generator` module also provides a derived class, +:class:`DecodedGenerator`, which is like the :class:`Generator` base class, +except that non-\ :mimetype:`text` parts are not serialized, but are instead +represented in the output stream by a string derived from a template filled +in with information about the part. -.. class:: DecodedGenerator(outfp, mangle_from_=True, maxheaderlen=78, fmt=None) +.. class:: DecodedGenerator(outfp, mangle_from_=None, maxheaderlen=None, \ + fmt=None, *, policy=None) - This class, derived from :class:`Generator` walks through all the subparts of a - message. If the subpart is of main type :mimetype:`text`, then it prints the - decoded payload of the subpart. Optional *_mangle_from_* and *maxheaderlen* are - as with the :class:`Generator` base class. + Act like :class:`Generator`, except that for any subpart of the message + passed to :meth:`Generator.flatten`, if the subpart is of main type + :mimetype:`text`, print the decoded payload of the subpart, and if the main + type is not :mimetype:`text`, instead of printing it fill in the string + *fmt* using information from the part and print the resulting + filled-in string. - If the subpart is not of main type :mimetype:`text`, optional *fmt* is a format - string that is used instead of the message payload. *fmt* is expanded with the - following keywords, ``%(keyword)s`` format: + To fill in *fmt*, execute ``fmt % part_info``, where ``part_info`` + is a dictionary composed of the following keys and values: * ``type`` -- Full MIME type of the non-\ :mimetype:`text` part @@ -225,15 +259,21 @@ representing the part. * ``encoding`` -- Content transfer encoding of the non-\ :mimetype:`text` part - The default value for *fmt* is ``None``, meaning :: + If *fmt* is ``None``, use the following default *fmt*: + + "[Non-text (%(type)s) part of message omitted, filename %(filename)s]" - [Non-text (%(type)s) part of message omitted, filename %(filename)s] + Optional *_mangle_from_* and *maxheaderlen* are as with the + :class:`Generator` base class. .. rubric:: Footnotes -.. [#] This statement assumes that you use the appropriate setting for the - ``unixfrom`` argument, and that you set maxheaderlen=0 (which will - preserve whatever the input line lengths were). It is also not strictly - true, since in many cases runs of whitespace in headers are collapsed - into single blanks. The latter is a bug that will eventually be fixed. +.. [#] This statement assumes that you use the appropriate setting for + ``unixfrom``, and that there are no :mod:`policy` settings calling for + automatic adjustments (for example, + :attr:`~email.policy.Policy.refold_source` must be ``none``, which is + *not* the default). It is also not 100% true, since if the message + does not conform to the RFC standards occasionally information about the + exact original text is lost during parsing error recovery. It is a goal + to fix these latter edge cases when possible. diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst index e94837c1c9..07152c224f 100644 --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -8,6 +8,14 @@ -------------- +This module is part of the legacy (``Compat32``) email API. In the current API +encoding and decoding of headers is handled transparently by the +dictionary-like API of the :class:`~email.message.EmailMessage` class. In +addition to uses in legacy code, this module can be useful in applications that +need to completely control the character sets used when encoding headers. + +The remaining text in this section is the original documentation of the module. + :rfc:`2822` is the base standard that describes the format of email messages. It derives from the older :rfc:`822` standard which came into widespread use at a time when most email was composed of ASCII characters only. :rfc:`2822` is a diff --git a/Doc/library/email.headerregistry.rst b/Doc/library/email.headerregistry.rst index 0707bd858a..2c830cfd81 100644 --- a/Doc/library/email.headerregistry.rst +++ b/Doc/library/email.headerregistry.rst @@ -7,20 +7,12 @@ .. moduleauthor:: R. David Murray <rdmurray@bitdance.com> .. sectionauthor:: R. David Murray <rdmurray@bitdance.com> -.. versionadded:: 3.3 - as a :term:`provisional module <provisional package>`. - **Source code:** :source:`Lib/email/headerregistry.py` -.. note:: - - The headerregistry module has been included in the standard library on a - :term:`provisional basis <provisional package>`. Backwards incompatible - changes (up to and including removal of the module) may occur if deemed - necessary by the core developers. - -------------- +.. versionadded:: 3.6 [1]_ + Headers are represented by customized subclasses of :class:`str`. The particular class used to represent a given header is determined by the :attr:`~email.policy.EmailPolicy.header_factory` of the :mod:`~email.policy` in @@ -86,10 +78,11 @@ headers. .. method:: fold(*, policy) Return a string containing :attr:`~email.policy.Policy.linesep` - characters as required to correctly fold the header according - to *policy*. A :attr:`~email.policy.Policy.cte_type` of - ``8bit`` will be treated as if it were ``7bit``, since strings - may not contain binary data. + characters as required to correctly fold the header according to + *policy*. A :attr:`~email.policy.Policy.cte_type` of ``8bit`` will be + treated as if it were ``7bit``, since headers may not contain arbitrary + binary data. If :attr:`~email.policy.EmailPolicy.utf8` is ``False``, + non-ASCII data will be :rfc:`2047` encoded. ``BaseHeader`` by itself cannot be used to create a header object. It @@ -106,7 +99,7 @@ headers. values for at least the keys ``decoded`` and ``defects``. ``decoded`` should be the string value for the header (that is, the header value fully decoded to unicode). The parse method should assume that *string* may - contain transport encoded parts, but should correctly handle all valid + contain content-transfer-encoded parts, but should correctly handle all valid unicode characters as well so that it can parse un-encoded header values. ``BaseHeader``'s ``__new__`` then creates the header instance, and calls its @@ -135,11 +128,10 @@ headers. mechanism for encoding non-ASCII text as ASCII characters within a header value. When a *value* containing encoded words is passed to the constructor, the ``UnstructuredHeader`` parser converts such encoded words - back in to the original unicode, following the :rfc:`2047` rules for - unstructured text. The parser uses heuristics to attempt to decode certain - non-compliant encoded words. Defects are registered in such cases, as well - as defects for issues such as invalid characters within the encoded words or - the non-encoded text. + into unicode, following the :rfc:`2047` rules for unstructured text. The + parser uses heuristics to attempt to decode certain non-compliant encoded + words. Defects are registered in such cases, as well as defects for issues + such as invalid characters within the encoded words or the non-encoded text. This header type provides no additional attributes. @@ -213,15 +205,16 @@ headers. the list of addresses is "flattened" into a one dimensional list). The ``decoded`` value of the header will have all encoded words decoded to - unicode. :class:`~encodings.idna` encoded domain names are also decoded to unicode. The - ``decoded`` value is set by :attr:`~str.join`\ ing the :class:`str` value of - the elements of the ``groups`` attribute with ``', '``. + unicode. :class:`~encodings.idna` encoded domain names are also decoded to + unicode. The ``decoded`` value is set by :attr:`~str.join`\ ing the + :class:`str` value of the elements of the ``groups`` attribute with ``', + '``. A list of :class:`.Address` and :class:`.Group` objects in any combination may be used to set the value of an address header. ``Group`` objects whose ``display_name`` is ``None`` will be interpreted as single addresses, which allows an address list to be copied with groups intact by using the list - obtained ``groups`` attribute of the source header. + obtained from the ``groups`` attribute of the source header. .. class:: SingleAddressHeader @@ -267,7 +260,7 @@ variant, :attr:`~.BaseHeader.max_count` is set to 1. .. class:: ParameterizedMIMEHeader - MOME headers all start with the prefix 'Content-'. Each specific header has + MIME headers all start with the prefix 'Content-'. Each specific header has a certain value, described under the class for that header. Some can also take a list of supplemental parameters, which have a common format. This class serves as a base for all the MIME headers that take parameters. @@ -454,3 +447,9 @@ construct structured values to assign to specific headers. ``display_name`` is none and there is a single ``Address`` in the ``addresses`` list, the ``str`` value will be the same as the ``str`` of that single ``Address``. + + +.. rubric:: Footnotes + +.. [1] Oringally added in 3.3 as a :term:`provisional module <provisional + package>` diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index 2907975357..95136d2b84 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -3,91 +3,102 @@ .. module:: email.message :synopsis: The base class representing email messages. +.. moduleauthor:: R. David Murray <rdmurray@bitdance.com> +.. sectionauthor:: R. David Murray <rdmurray@bitdance.com>, + Barry A. Warsaw <barry@python.org> **Source code:** :source:`Lib/email/message.py` -------------- -The central class in the :mod:`email` package is the :class:`Message` class, -imported from the :mod:`email.message` module. It is the base class for the -:mod:`email` object model. :class:`Message` provides the core functionality for -setting and querying header fields, and for accessing message bodies. - -Conceptually, a :class:`Message` object consists of *headers* and *payloads*. -Headers are :rfc:`2822` style field names and values where the field name and -value are separated by a colon. The colon is not part of either the field name -or the field value. - -Headers are stored and returned in case-preserving form but are matched -case-insensitively. There may also be a single envelope header, also known as -the *Unix-From* header or the ``From_`` header. The payload is either a string -in the case of simple message objects or a list of :class:`Message` objects for -MIME container documents (e.g. :mimetype:`multipart/\*` and -:mimetype:`message/rfc822`). - -:class:`Message` objects provide a mapping style interface for accessing the -message headers, and an explicit interface for accessing both the headers and -the payload. It provides convenience methods for generating a flat text -representation of the message object tree, for accessing commonly used header -parameters, and for recursively walking over the object tree. - -Here are the methods of the :class:`Message` class: - - -.. class:: Message(policy=compat32) - - If *policy* is specified (it must be an instance of a :mod:`~email.policy` - class) use the rules it specifies to update and serialize the representation - of the message. If *policy* is not set, use the :class:`compat32 - <email.policy.Compat32>` policy, which maintains backward compatibility with - the Python 3.2 version of the email package. For more information see the +.. versionadded:: 3.6 [1]_ + +The central class in the :mod:`email` package is the :class:`EmailMessage` +class, imported from the :mod:`email.message` module. It is the base class for +the :mod:`email` object model. :class:`EmailMessage` provides the core +functionality for setting and querying header fields, for accessing message +bodies, and for creating or modifying structured messages. + +An email message consists of *headers* and a *payload* (which is also referred +to as the *content*). Headers are :rfc:`5322` or :rfc:`6532` style field names +and values, where the field name and value are separated by a colon. The colon +is not part of either the field name or the field value. The payload may be a +simple text message, or a binary object, or a structured sequence of +sub-messages each with their own set of headers and their own payload. The +latter type of payload is indicated by the message having a MIME type such as +:mimetype:`multipart/\*` or :mimetype:`message/rfc822`. + +The conceptual model provided by an :class:`EmailMessage` object is that of an +ordered dictionary of headers coupled with a *payload* that represents the +:rfc:`5322` body of the message, which might be a list of sub-``EmailMessage`` +objects. In addition to the normal dictionary methods for accessing the header +names and values, there are methods for accessing specialized information from +the headers (for example the MIME content type), for operating on the payload, +for generating a serialized version of the message, and for recursively walking +over the object tree. + +The :class:`EmailMessage` dictionary-like interface is indexed by the header +names, which must be ASCII values. The values of the dictionary are strings +with some extra methods. Headers are stored and returned in case-preserving +form, but field names are matched case-insensitively. Unlike a real dict, +there is an ordering to the keys, and there can be duplicate keys. Additional +methods are provided for working with headers that have duplicate keys. + +The *payload* is either a string or bytes object, in the case of simple message +objects, or a list of :class:`EmailMessage` objects, for MIME container +documents such as :mimetype:`multipart/\*` and :mimetype:`message/rfc822` +message objects. + + +.. class:: EmailMessage(policy=default) + + If *policy* is specified use the rules it specifies to udpate and serialize + the representation of the message. If *policy* is not set, use the + :class:`~email.policy.default` policy, which follows the rules of the email + RFCs except for line endings (instead of the RFC mandated ``\r\n``, it uses + the Python standard ``\n`` line endings). For more information see the :mod:`~email.policy` documentation. - .. versionchanged:: 3.3 The *policy* keyword argument was added. - - - .. method:: as_string(unixfrom=False, maxheaderlen=0, policy=None) - - Return the entire message flattened as a string. When optional *unixfrom* - is true, the envelope header is included in the returned string. - *unixfrom* defaults to ``False``. For backward compabitility reasons, - *maxheaderlen* defaults to ``0``, so if you want a different value you - must override it explicitly (the value specified for *max_line_length* in - the policy will be ignored by this method). The *policy* argument may be - used to override the default policy obtained from the message instance. - This can be used to control some of the formatting produced by the - method, since the specified *policy* will be passed to the ``Generator``. - - Flattening the message may trigger changes to the :class:`Message` if - defaults need to be filled in to complete the transformation to a string - (for example, MIME boundaries may be generated or modified). - - Note that this method is provided as a convenience and may not always - format the message the way you want. For example, by default it does - not do the mangling of lines that begin with ``From`` that is - required by the unix mbox format. For more flexibility, instantiate a - :class:`~email.generator.Generator` instance and use its - :meth:`~email.generator.Generator.flatten` method directly. For example:: - - from io import StringIO - from email.generator import Generator - fp = StringIO() - g = Generator(fp, mangle_from_=True, maxheaderlen=60) - g.flatten(msg) - text = fp.getvalue() + .. method:: as_string(unixfrom=False, maxheaderlen=None, policy=None) - If the message object contains binary data that is not encoded according - to RFC standards, the non-compliant data will be replaced by unicode - "unknown character" code points. (See also :meth:`.as_bytes` and - :class:`~email.generator.BytesGenerator`.) - - .. versionchanged:: 3.4 the *policy* keyword argument was added. + Return the entire message flattened as a string. When optional + *unixfrom* is true, the envelope header is included in the returned + string. *unixfrom* defaults to ``False``. For backward compabitility + with the base :class:`~email.message.Message` class *maxheaderlen* is + accepted, but defaults to ``None``, which means that by default the line + length is controlled by the + :attr:`~email.policy.EmailPolicy.max_line_length` of the policy. The + *policy* argument may be used to override the default policy obtained + from the message instance. This can be used to control some of the + formatting produced by the method, since the specified *policy* will be + passed to the :class:`~email.generator.Generator`. + + Flattening the message may trigger changes to the :class:`EmailMessage` + if defaults need to be filled in to complete the transformation to a + string (for example, MIME boundaries may be generated or modified). + + Note that this method is provided as a convenience and may not be the + most useful way to serialize messages in your application, especially if + you are dealing with multiple messages. See + :class:`email.generator.Generator` for a more flexible API for + serializing messages. Note also that this method is restricted to + producing messages serialized as "7 bit clean" when + :attr:`~email.policy.EmailPolicy.utf8` is ``False``, which is the default. + + .. versionchanged:: 3.6 the default behavior when *maxheaderlen* + is not specified was changed from defaulting to 0 to defaulting + to the value of *max_line_length* from the policy. .. method:: __str__() - Equivalent to :meth:`.as_string()`. Allows ``str(msg)`` to produce a - string containing the formatted message. + Equivalent to `as_string(policy=self.policy.clone(utf8=True)`. Allows + ``str(msg)`` to produce a string containing the serialized message in a + readable format. + + .. versionchanged:: 3.4 the method was changed to use ``utf8=True``, + thus producing an :rfc:`6531`-like message representation, instead of + being a direct alias for :meth:`as_string`. .. method:: as_bytes(unixfrom=False, policy=None) @@ -98,52 +109,42 @@ Here are the methods of the :class:`Message` class: used to override the default policy obtained from the message instance. This can be used to control some of the formatting produced by the method, since the specified *policy* will be passed to the - ``BytesGenerator``. - - Flattening the message may trigger changes to the :class:`Message` if - defaults need to be filled in to complete the transformation to a string - (for example, MIME boundaries may be generated or modified). + :class:`~email.generator.BytesGenerator`. - Note that this method is provided as a convenience and may not always - format the message the way you want. For example, by default it does - not do the mangling of lines that begin with ``From`` that is - required by the unix mbox format. For more flexibility, instantiate a - :class:`~email.generator.BytesGenerator` instance and use its - :meth:`~email.generator.BytesGenerator.flatten` method directly. - For example:: + Flattening the message may trigger changes to the :class:`EmailMessage` + if defaults need to be filled in to complete the transformation to a + string (for example, MIME boundaries may be generated or modified). - from io import BytesIO - from email.generator import BytesGenerator - fp = BytesIO() - g = BytesGenerator(fp, mangle_from_=True, maxheaderlen=60) - g.flatten(msg) - text = fp.getvalue() - - .. versionadded:: 3.4 + Note that this method is provided as a convenience and may not be the + most useful way to serialize messages in your application, especially if + you are dealing with multiple messages. See + :class:`email.generator.BytesGenerator` for a more flexible API for + serializing messages. .. method:: __bytes__() Equivalent to :meth:`.as_bytes()`. Allows ``bytes(msg)`` to produce a - bytes object containing the formatted message. - - .. versionadded:: 3.4 + bytes object containing the serialized message. .. method:: is_multipart() Return ``True`` if the message's payload is a list of sub-\ - :class:`Message` objects, otherwise return ``False``. When + :class:`EmailMessage` objects, otherwise return ``False``. When :meth:`is_multipart` returns ``False``, the payload should be a string - object. (Note that :meth:`is_multipart` returning ``True`` does not - necessarily mean that "msg.get_content_maintype() == 'multipart'" will - return the ``True``. For example, ``is_multipart`` will return ``True`` - when the :class:`Message` is of type ``message/rfc822``.) + object (which might be a CTE encoded binary payload). Note that + :meth:`is_multipart` returning ``True`` does not necessarily mean that + "msg.get_content_maintype() == 'multipart'" will return the ``True``. + For example, ``is_multipart`` will return ``True`` when the + :class:`EmailMessage` is of type ``message/rfc822``. .. method:: set_unixfrom(unixfrom) - Set the message's envelope header to *unixfrom*, which should be a string. + Set the message's envelope header to *unixfrom*, which should be a + string. (See :class:`~mailbox.mboxMessage` for a brief description of + this header.) .. method:: get_unixfrom() @@ -152,109 +153,23 @@ Here are the methods of the :class:`Message` class: envelope header was never set. - .. method:: attach(payload) - - Add the given *payload* to the current payload, which must be ``None`` or - a list of :class:`Message` objects before the call. After the call, the - payload will always be a list of :class:`Message` objects. If you want to - set the payload to a scalar object (e.g. a string), use - :meth:`set_payload` instead. - - - .. method:: get_payload(i=None, decode=False) - - Return the current payload, which will be a list of - :class:`Message` objects when :meth:`is_multipart` is ``True``, or a - string when :meth:`is_multipart` is ``False``. If the payload is a list - and you mutate the list object, you modify the message's payload in place. - - With optional argument *i*, :meth:`get_payload` will return the *i*-th - element of the payload, counting from zero, if :meth:`is_multipart` is - ``True``. An :exc:`IndexError` will be raised if *i* is less than 0 or - greater than or equal to the number of items in the payload. If the - payload is a string (i.e. :meth:`is_multipart` is ``False``) and *i* is - given, a :exc:`TypeError` is raised. - - Optional *decode* is a flag indicating whether the payload should be - decoded or not, according to the :mailheader:`Content-Transfer-Encoding` - header. When ``True`` and the message is not a multipart, the payload will - be decoded if this header's value is ``quoted-printable`` or ``base64``. - If some other encoding is used, or :mailheader:`Content-Transfer-Encoding` - header is missing, the payload is - returned as-is (undecoded). In all cases the returned value is binary - data. If the message is a multipart and the *decode* flag is ``True``, - then ``None`` is returned. If the payload is base64 and it was not - perfectly formed (missing padding, characters outside the base64 - alphabet), then an appropriate defect will be added to the message's - defect property (:class:`~email.errors.InvalidBase64PaddingDefect` or - :class:`~email.errors.InvalidBase64CharactersDefect`, respectively). - - When *decode* is ``False`` (the default) the body is returned as a string - without decoding the :mailheader:`Content-Transfer-Encoding`. However, - for a :mailheader:`Content-Transfer-Encoding` of 8bit, an attempt is made - to decode the original bytes using the ``charset`` specified by the - :mailheader:`Content-Type` header, using the ``replace`` error handler. - If no ``charset`` is specified, or if the ``charset`` given is not - recognized by the email package, the body is decoded using the default - ASCII charset. - - - .. method:: set_payload(payload, charset=None) - - Set the entire message object's payload to *payload*. It is the client's - responsibility to ensure the payload invariants. Optional *charset* sets - the message's default character set; see :meth:`set_charset` for details. - - .. method:: set_charset(charset) - - Set the character set of the payload to *charset*, which can either be a - :class:`~email.charset.Charset` instance (see :mod:`email.charset`), a - string naming a character set, or ``None``. If it is a string, it will - be converted to a :class:`~email.charset.Charset` instance. If *charset* - is ``None``, the ``charset`` parameter will be removed from the - :mailheader:`Content-Type` header (the message will not be otherwise - modified). Anything else will generate a :exc:`TypeError`. - - If there is no existing :mailheader:`MIME-Version` header one will be - added. If there is no existing :mailheader:`Content-Type` header, one - will be added with a value of :mimetype:`text/plain`. Whether the - :mailheader:`Content-Type` header already exists or not, its ``charset`` - parameter will be set to *charset.output_charset*. If - *charset.input_charset* and *charset.output_charset* differ, the payload - will be re-encoded to the *output_charset*. If there is no existing - :mailheader:`Content-Transfer-Encoding` header, then the payload will be - transfer-encoded, if needed, using the specified - :class:`~email.charset.Charset`, and a header with the appropriate value - will be added. If a :mailheader:`Content-Transfer-Encoding` header - already exists, the payload is assumed to already be correctly encoded - using that :mailheader:`Content-Transfer-Encoding` and is not modified. - - .. method:: get_charset() - - Return the :class:`~email.charset.Charset` instance associated with the - message's payload. - - The following methods implement a mapping-like interface for accessing the - message's :rfc:`2822` headers. Note that there are some semantic differences + The following methods implement the mapping-like interface for accessing the + message's headers. Note that there are some semantic differences between these methods and a normal mapping (i.e. dictionary) interface. For example, in a dictionary there are no duplicate keys, but here there may be duplicate message headers. Also, in dictionaries there is no guaranteed - order to the keys returned by :meth:`keys`, but in a :class:`Message` object, - headers are always returned in the order they appeared in the original - message, or were added to the message later. Any header deleted and then - re-added are always appended to the end of the header list. + order to the keys returned by :meth:`keys`, but in an :class:`EmailMessage` + object, headers are always returned in the order they appeared in the + original message, or in which they were added to the message later. Any + header deleted and then re-added is always appended to the end of the + header list. - These semantic differences are intentional and are biased toward maximal - convenience. + These semantic differences are intentional and are biased toward + convenience in the most common use cases. Note that in all cases, any envelope header present in the message is not included in the mapping interface. - In a model generated from bytes, any header values that (in contravention of - the RFCs) contain non-ASCII bytes will, when retrieved through this - interface, be represented as :class:`~email.header.Header` objects with - a charset of `unknown-8bit`. - .. method:: __len__() @@ -264,8 +179,8 @@ Here are the methods of the :class:`Message` class: .. method:: __contains__(name) Return true if the message object has a field named *name*. Matching is - done case-insensitively and *name* should not include the trailing colon. - Used for the ``in`` operator, e.g.:: + done without regard to case and *name* does not include the trailing + colon. Used for the ``in`` operator. For example:: if 'message-id' in myMessage: print('Message-ID:', myMessage['message-id']) @@ -273,20 +188,23 @@ Here are the methods of the :class:`Message` class: .. method:: __getitem__(name) - Return the value of the named header field. *name* should not include the + Return the value of the named header field. *name* does not include the colon field separator. If the header is missing, ``None`` is returned; a :exc:`KeyError` is never raised. Note that if the named field appears more than once in the message's headers, exactly which of those field values will be returned is undefined. Use the :meth:`get_all` method to get the values of all the - extant named headers. + extant headers named *name*. + + Using the standard (non-``compat32``) policies, the returned value is an + instance of a subclass of :class:`email.headerregistry.BaseHeader`. .. method:: __setitem__(name, val) Add a header to the message with field name *name* and value *val*. The - field is appended to the end of the message's existing fields. + field is appended to the end of the message's existing headers. Note that this does *not* overwrite or delete any existing header with the same name. If you want to ensure that the new header is the only one present in the @@ -295,6 +213,13 @@ Here are the methods of the :class:`Message` class: del msg['subject'] msg['subject'] = 'Python roolz!' + If the :mod:`policy` defines certain haders to be unique (as the standard + policies do), this method may raise a :exc:`ValueError` when an attempt + is made to assign a value to such a header when one already exists. This + behavior is intentional for consistency's sake, but do not depend on it + as we may choose to make such assignments do an automatic deletion of the + existing header in the future. + .. method:: __delitem__(name) @@ -323,9 +248,10 @@ Here are the methods of the :class:`Message` class: Return the value of the named header field. This is identical to :meth:`__getitem__` except that optional *failobj* is returned if the - named header is missing (defaults to ``None``). + named header is missing (*failobj* defaults to ``None``). - Here are some additional useful methods: + + Here are some additional useful header related methods: .. method:: get_all(name, failobj=None) @@ -346,17 +272,19 @@ Here are the methods of the :class:`Message` class: taken as the parameter name, with underscores converted to dashes (since dashes are illegal in Python identifiers). Normally, the parameter will be added as ``key="value"`` unless the value is ``None``, in which case - only the key will be added. If the value contains non-ASCII characters, - it can be specified as a three tuple in the format - ``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string naming the - charset to be used to encode the value, ``LANGUAGE`` can usually be set - to ``None`` or the empty string (see :rfc:`2231` for other possibilities), - and ``VALUE`` is the string value containing non-ASCII code points. If - a three tuple is not passed and the value contains non-ASCII characters, - it is automatically encoded in :rfc:`2231` format using a ``CHARSET`` - of ``utf-8`` and a ``LANGUAGE`` of ``None``. - - Here's an example:: + only the key will be added. + + If the value contains non-ASCII characters, the charset and language may + be explicitly controlled by specifing the value as a three tuple in the + format ``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string + naming the charset to be used to encode the value, ``LANGUAGE`` can + usually be set to ``None`` or the empty string (see :rfc:`2231` for other + possibilities), and ``VALUE`` is the string value containing non-ASCII + code points. If a three tuple is not passed and the value contains + non-ASCII characters, it is automatically encoded in :rfc:`2231` format + using a ``CHARSET`` of ``utf-8`` and a ``LANGUAGE`` of ``None``. + + Here is an example:: msg.add_header('Content-Disposition', 'attachment', filename='bud.gif') @@ -364,37 +292,35 @@ Here are the methods of the :class:`Message` class: Content-Disposition: attachment; filename="bud.gif" - An example with non-ASCII characters:: + An example of the extended interface with non-ASCII characters:: msg.add_header('Content-Disposition', 'attachment', filename=('iso-8859-1', '', 'Fußballer.ppt')) - Which produces :: - - Content-Disposition: attachment; filename*="iso-8859-1''Fu%DFballer.ppt" - .. method:: replace_header(_name, _value) Replace a header. Replace the first header found in the message that - matches *_name*, retaining header order and field name case. If no - matching header was found, a :exc:`KeyError` is raised. + matches *_name*, retaining header order and field name case of the + original header. If no matching header is found, raise a + :exc:`KeyError`. .. method:: get_content_type() - Return the message's content type. The returned string is coerced to - lower case of the form :mimetype:`maintype/subtype`. If there was no - :mailheader:`Content-Type` header in the message the default type as given - by :meth:`get_default_type` will be returned. Since according to - :rfc:`2045`, messages always have a default type, :meth:`get_content_type` - will always return a value. + Return the message's content type, coerced to lower case of the form + :mimetype:`maintype/subtype`. If there is no :mailheader:`Content-Type` + header in the message return the value returned by + :meth:`get_default_type`. If the :mailheader:`Content-Type` header is + invalid, return ``text/plain``. - :rfc:`2045` defines a message's default type to be :mimetype:`text/plain` - unless it appears inside a :mimetype:`multipart/digest` container, in - which case it would be :mimetype:`message/rfc822`. If the - :mailheader:`Content-Type` header has an invalid type specification, - :rfc:`2045` mandates that the default type be :mimetype:`text/plain`. + (According to :rfc:`2045`, messages always have a default type, + :meth:`get_content_type` will always return a value. :rfc:`2045` defines + a message's default type to be :mimetype:`text/plain` unless it appears + inside a :mimetype:`multipart/digest` container, in which case it would + be :mimetype:`message/rfc822`. If the :mailheader:`Content-Type` header + has an invalid type specification, :rfc:`2045` mandates that the default + type be :mimetype:`text/plain`.) .. method:: get_content_maintype() @@ -420,81 +346,41 @@ Here are the methods of the :class:`Message` class: .. method:: set_default_type(ctype) Set the default content type. *ctype* should either be - :mimetype:`text/plain` or :mimetype:`message/rfc822`, although this is not - enforced. The default content type is not stored in the - :mailheader:`Content-Type` header. - - - .. method:: get_params(failobj=None, header='content-type', unquote=True) - - Return the message's :mailheader:`Content-Type` parameters, as a list. - The elements of the returned list are 2-tuples of key/value pairs, as - split on the ``'='`` sign. The left hand side of the ``'='`` is the key, - while the right hand side is the value. If there is no ``'='`` sign in - the parameter the value is the empty string, otherwise the value is as - described in :meth:`get_param` and is unquoted if optional *unquote* is - ``True`` (the default). - - Optional *failobj* is the object to return if there is no - :mailheader:`Content-Type` header. Optional *header* is the header to - search instead of :mailheader:`Content-Type`. - - - .. method:: get_param(param, failobj=None, header='content-type', unquote=True) - - Return the value of the :mailheader:`Content-Type` header's parameter - *param* as a string. If the message has no :mailheader:`Content-Type` - header or if there is no such parameter, then *failobj* is returned - (defaults to ``None``). - - Optional *header* if given, specifies the message header to use instead of - :mailheader:`Content-Type`. - - Parameter keys are always compared case insensitively. The return value - can either be a string, or a 3-tuple if the parameter was :rfc:`2231` - encoded. When it's a 3-tuple, the elements of the value are of the form - ``(CHARSET, LANGUAGE, VALUE)``. Note that both ``CHARSET`` and - ``LANGUAGE`` can be ``None``, in which case you should consider ``VALUE`` - to be encoded in the ``us-ascii`` charset. You can usually ignore - ``LANGUAGE``. - - If your application doesn't care whether the parameter was encoded as in - :rfc:`2231`, you can collapse the parameter value by calling - :func:`email.utils.collapse_rfc2231_value`, passing in the return value - from :meth:`get_param`. This will return a suitably decoded Unicode - string when the value is a tuple, or the original string unquoted if it - isn't. For example:: - - rawparam = msg.get_param('foo') - param = email.utils.collapse_rfc2231_value(rawparam) - - In any case, the parameter value (either the returned string, or the - ``VALUE`` item in the 3-tuple) is always unquoted, unless *unquote* is set - to ``False``. + :mimetype:`text/plain` or :mimetype:`message/rfc822`, although this is + not enforced. The default content type is not stored in the + :mailheader:`Content-Type` header, so it only affects the return value of + the ``get_content_type`` methods when no :mailheader:`Content-Type` + header is present in the message. .. method:: set_param(param, value, header='Content-Type', requote=True, \ charset=None, language='', replace=False) Set a parameter in the :mailheader:`Content-Type` header. If the - parameter already exists in the header, its value will be replaced with - *value*. If the :mailheader:`Content-Type` header as not yet been defined - for this message, it will be set to :mimetype:`text/plain` and the new - parameter value will be appended as per :rfc:`2045`. - - Optional *header* specifies an alternative header to - :mailheader:`Content-Type`, and all parameters will be quoted as necessary - unless optional *requote* is ``False`` (the default is ``True``). - - If optional *charset* is specified, the parameter will be encoded - according to :rfc:`2231`. Optional *language* specifies the RFC 2231 - language, defaulting to the empty string. Both *charset* and *language* - should be strings. + parameter already exists in the header, replace its value with *value*. + When *header* is ``Content-Type`` (the default) and the header does not + yet exist in the message, add it, set its value to + :mimetype:`text/plain`, and append the new parameter value. Optional + *header* specifies an alternative header to :mailheader:`Content-Type`. + + If the value contains non-ASCII characters, the charset and language may + be explicity specified using the optional *charset* and *language* + parameters. Optional *language* specifies the :rfc:`2231` language, + defaulting to the empty string. Both *charset* and *language* should be + strings. The default is to use the ``utf8`` *charset* and ``None`` for + the *language*. If *replace* is ``False`` (the default) the header is moved to the end of the list of headers. If *replace* is ``True``, the header will be updated in place. + Use of the *requote* parameter with :class:`EmailMessage` objects is + deprecated. + + Note that existing parameter values of headers may be accessed through + the :attr:`~email.headerregistry.BaseHeader.params` attribute of the + header value (for example, ``msg['Content-Type'].params['charset']``. + .. versionchanged:: 3.4 ``replace`` keyword was added. @@ -502,25 +388,11 @@ Here are the methods of the :class:`Message` class: Remove the given parameter completely from the :mailheader:`Content-Type` header. The header will be re-written in place without the parameter or - its value. All values will be quoted as necessary unless *requote* is - ``False`` (the default is ``True``). Optional *header* specifies an - alternative to :mailheader:`Content-Type`. - - - .. method:: set_type(type, header='Content-Type', requote=True) - - Set the main type and subtype for the :mailheader:`Content-Type` - header. *type* must be a string in the form :mimetype:`maintype/subtype`, - otherwise a :exc:`ValueError` is raised. - - This method replaces the :mailheader:`Content-Type` header, keeping all - the parameters in place. If *requote* is ``False``, this leaves the - existing header's quoting as is, otherwise the parameters will be quoted - (the default). + its value. Optional *header* specifies an alternative to + :mailheader:`Content-Type`. - An alternative header can be specified in the *header* argument. When the - :mailheader:`Content-Type` header is set a :mailheader:`MIME-Version` - header is also added. + Use of the *requote* parameter with :class:`EmailMessage` objects is + deprecated. .. method:: get_filename(failobj=None) @@ -549,12 +421,11 @@ Here are the methods of the :class:`Message` class: necessary. A :exc:`~email.errors.HeaderParseError` is raised if the message object has no :mailheader:`Content-Type` header. - Note that using this method is subtly different than deleting the old + Note that using this method is subtly different from deleting the old :mailheader:`Content-Type` header and adding a new one with the new boundary via :meth:`add_header`, because :meth:`set_boundary` preserves the order of the :mailheader:`Content-Type` header in the list of - headers. However, it does *not* preserve any continuation lines which may - have been present in the original :mailheader:`Content-Type` header. + headers. .. method:: get_content_charset(failobj=None) @@ -563,9 +434,6 @@ Here are the methods of the :class:`Message` class: coerced to lower case. If there is no :mailheader:`Content-Type` header, or if that header has no ``charset`` parameter, *failobj* is returned. - Note that this method differs from :meth:`get_charset` which returns the - :class:`~email.charset.Charset` instance for the default encoding of the message body. - .. method:: get_charsets(failobj=None) @@ -575,10 +443,19 @@ Here are the methods of the :class:`Message` class: Each item in the list will be a string which is the value of the ``charset`` parameter in the :mailheader:`Content-Type` header for the - represented subpart. However, if the subpart has no - :mailheader:`Content-Type` header, no ``charset`` parameter, or is not of - the :mimetype:`text` main MIME type, then that item in the returned list - will be *failobj*. + represented subpart. If the subpart has no :mailheader:`Content-Type` + header, no ``charset`` parameter, or is not of the :mimetype:`text` main + MIME type, then that item in the returned list will be *failobj*. + + + .. method:: is_attachment + + Return ``True`` if there is a :mailheader:`Content-Disposition` header + and its (case insensitive) value is ``attachment``, ``False`` otherwise. + + .. versionchanged:: 3.4.2 + is_attachment is now a method instead of a property, for consistency + with :meth:`~email.message.Message.is_multipart`. .. method:: get_content_disposition() @@ -590,6 +467,11 @@ Here are the methods of the :class:`Message` class: .. versionadded:: 3.5 + + The following methods relate to interrogating and manipulating the content + (payload) of the message. + + .. method:: walk() The :meth:`walk` method is an all-purpose generator which can be used to @@ -651,8 +533,169 @@ Here are the methods of the :class:`Message` class: into the subparts. - :class:`Message` objects can also optionally contain two instance attributes, - which can be used when generating the plain text of a MIME message. + .. method:: get_body(preferencelist=('related', 'html', 'plain')) + + Return the MIME part that is the best candidate to be the "body" of the + message. + + *preferencelist* must be a sequence of strings from the set ``related``, + ``html``, and ``plain``, and indicates the order of preference for the + content type of the part returned. + + Start looking for candidate matches with the object on which the + ``get_body`` method is called. + + If ``related`` is not included in *preferencelist*, consider the root + part (or subpart of the root part) of any related encountered as a + candidate if the (sub-)part matches a preference. + + When encountering a ``multipart/related``, check the ``start`` parameter + and if a part with a matching :mailheader:`Content-ID` is found, consider + only it when looking for candidate matches. Otherwise consider only the + first (default root) part of the ``multipart/related``. + + If a part has a :mailheader:`Content-Disposition` header, only consider + the part a candidate match if the value of the header is ``inline``. + + If none of the candidates matches any of the preferences in + *preferneclist*, return ``None``. + + Notes: (1) For most applications the only *preferencelist* combinations + that really make sense are ``('plain',)``, ``('html', 'plain')``, and the + default ``('related', 'html', 'plain')``. (2) Because matching starts + with the object on which ``get_body`` is called, calling ``get_body`` on + a ``multipart/related`` will return the object itself unless + *preferencelist* has a non-default value. (3) Messages (or message parts) + that do not specify a :mailheader:`Content-Type` or whose + :mailheader:`Content-Type` header is invalid will be treated as if they + are of type ``text/plain``, which may occasionally cause ``get_body`` to + return unexpected results. + + + .. method:: iter_attachments() + + Return an iterator over all of the immediate sub-parts of the message + that are not candidate "body" parts. That is, skip the first occurrence + of each of ``text/plain``, ``text/html``, ``multipart/related``, or + ``multipart/alternative`` (unless they are explicitly marked as + attachments via :mailheader:`Content-Disposition: attachment`), and + return all remaining parts. When applied directly to a + ``multipart/related``, return an iterator over the all the related parts + except the root part (ie: the part pointed to by the ``start`` parameter, + or the first part if there is no ``start`` parameter or the ``start`` + parameter doesn't match the :mailheader:`Content-ID` of any of the + parts). When applied directly to a ``multipart/alternative`` or a + non-``multipart``, return an empty iterator. + + + .. method:: iter_parts() + + Return an iterator over all of the immediate sub-parts of the message, + which will be empty for a non-``multipart``. (See also + :meth:`~email.message.EmailMessage.walk`.) + + + .. method:: get_content(*args, content_manager=None, **kw) + + Call the :meth:`~email.contentmanager.ContentManager.get_content` method + of the *content_manager*, passing self as the message object, and passing + along any other arguments or keywords as additional arguments. If + *content_manager* is not specified, use the ``content_manager`` specified + by the current :mod:`~email.policy`. + + + .. method:: set_content(*args, content_manager=None, **kw) + + Call the :meth:`~email.contentmanager.ContentManager.set_content` method + of the *content_manager*, passing self as the message object, and passing + along any other arguments or keywords as additional arguments. If + *content_manager* is not specified, use the ``content_manager`` specified + by the current :mod:`~email.policy`. + + + .. method:: make_related(boundary=None) + + Convert a non-``multipart`` message into a ``multipart/related`` message, + moving any existing :mailheader:`Content-` headers and payload into a + (new) first part of the ``multipart``. If *boundary* is specified, use + it as the boundary string in the multipart, otherwise leave the boundary + to be automatically created when it is needed (for example, when the + message is serialized). + + + .. method:: make_alternative(boundary=None) + + Convert a non-``multipart`` or a ``multipart/related`` into a + ``multipart/alternative``, moving any existing :mailheader:`Content-` + headers and payload into a (new) first part of the ``multipart``. If + *boundary* is specified, use it as the boundary string in the multipart, + otherwise leave the boundary to be automatically created when it is + needed (for example, when the message is serialized). + + + .. method:: make_mixed(boundary=None) + + Convert a non-``multipart``, a ``multipart/related``, or a + ``multipart-alternative`` into a ``multipart/mixed``, moving any existing + :mailheader:`Content-` headers and payload into a (new) first part of the + ``multipart``. If *boundary* is specified, use it as the boundary string + in the multipart, otherwise leave the boundary to be automatically + created when it is needed (for example, when the message is serialized). + + + .. method:: add_related(*args, content_manager=None, **kw) + + If the message is a ``multipart/related``, create a new message + object, pass all of the arguments to its :meth:`set_content` method, + and :meth:`~email.message.Message.attach` it to the ``multipart``. If + the message is a non-``multipart``, call :meth:`make_related` and then + proceed as above. If the message is any other type of ``multipart``, + raise a :exc:`TypeError`. If *content_manager* is not specified, use + the ``content_manager`` specified by the current :mod:`~email.policy`. + If the added part has no :mailheader:`Content-Disposition` header, + add one with the value ``inline``. + + + .. method:: add_alternative(*args, content_manager=None, **kw) + + If the message is a ``multipart/alternative``, create a new message + object, pass all of the arguments to its :meth:`set_content` method, and + :meth:`~email.message.Message.attach` it to the ``multipart``. If the + message is a non-``multipart`` or ``multipart/related``, call + :meth:`make_alternative` and then proceed as above. If the message is + any other type of ``multipart``, raise a :exc:`TypeError`. If + *content_manager* is not specified, use the ``content_manager`` specified + by the current :mod:`~email.policy`. + + + .. method:: add_attachment(*args, content_manager=None, **kw) + + If the message is a ``multipart/mixed``, create a new message object, + pass all of the arguments to its :meth:`set_content` method, and + :meth:`~email.message.Message.attach` it to the ``multipart``. If the + message is a non-``multipart``, ``multipart/related``, or + ``multipart/alternative``, call :meth:`make_mixed` and then proceed as + above. If *content_manager* is not specified, use the ``content_manager`` + specified by the current :mod:`~email.policy`. If the added part + has no :mailheader:`Content-Disposition` header, add one with the value + ``attachment``. This method can be used both for explicit attachments + (:mailheader:`Content-Disposition: attachment` and ``inline`` attachments + (:mailheader:`Content-Disposition: inline`), by passing appropriate + options to the ``content_manager``. + + + .. method:: clear() + + Remove the payload and all of the headers. + + + .. method:: clear_content() + + Remove the payload and all of the :exc:`Content-` headers, leaving + all other headers intact and in their original order. + + + :class:`EmailMessage` objects have the following instance attributes: .. attribute:: preamble @@ -682,11 +725,8 @@ Here are the methods of the :class:`Message` class: The *epilogue* attribute acts the same way as the *preamble* attribute, except that it contains text that appears between the last boundary and - the end of the message. - - You do not need to set the epilogue to the empty string in order for the - :class:`~email.generator.Generator` to print a newline at the end of the - file. + the end of the message. As with the :attr:`~EmailMessage.preamble`, + if there is no epilog text this attribute will be ``None``. .. attribute:: defects @@ -694,3 +734,18 @@ Here are the methods of the :class:`Message` class: The *defects* attribute contains a list of all the problems found when parsing this message. See :mod:`email.errors` for a detailed description of the possible parsing defects. + + +.. class:: MIMEPart(policy=default) + + This class represents a subpart of a MIME message. It is identical to + :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are + added when :meth:`~EmailMessage.set_content` is called, since sub-parts do + not need their own :mailheader:`MIME-Version` headers. + + +.. rubric:: Footnotes + +.. [1] Oringally added in 3.4 as a :term:`provisional module <provisional + package>`. Docs for legacy message class moved to + :ref:`compat32_message`. diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index 8297deaf93..d9dae9f0b9 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -8,6 +8,11 @@ -------------- +This module is part of the legacy (``Compat32``) email API. Its functionality +is partially replaced by the :mod:`~email.contentmanager` in the new API, but +in certain applications these classes may still be useful, even in non-legacy +code. + Ordinarily, you get a message object structure by passing a file or some text to a parser, which parses the text and returns the root message object. However you can also build a complete message structure from scratch, or even individual @@ -25,7 +30,7 @@ Here are the classes: .. currentmodule:: email.mime.base -.. class:: MIMEBase(_maintype, _subtype, **_params) +.. class:: MIMEBase(_maintype, _subtype, *, policy=compat32, **_params) Module: :mod:`email.mime.base` @@ -41,10 +46,17 @@ Here are the classes: key/value dictionary and is passed directly to :meth:`Message.add_header <email.message.Message.add_header>`. + If *policy* is specified, (defaults to the + :class:`compat32 <email.policy.Compat32>` policy) it will be passed to + :class:`~email.message.Message`. + The :class:`MIMEBase` class always adds a :mailheader:`Content-Type` header (based on *_maintype*, *_subtype*, and *_params*), and a :mailheader:`MIME-Version` header (always set to ``1.0``). + .. versionchanged:: 3.6 + Added *policy* keyword-only parameter. + .. currentmodule:: email.mime.nonmultipart @@ -62,7 +74,8 @@ Here are the classes: .. currentmodule:: email.mime.multipart -.. class:: MIMEMultipart(_subtype='mixed', boundary=None, _subparts=None, **_params) +.. class:: MIMEMultipart(_subtype='mixed', boundary=None, _subparts=None, \ + *, policy=compat32, **_params) Module: :mod:`email.mime.multipart` @@ -82,14 +95,20 @@ Here are the classes: to the message by using the :meth:`Message.attach <email.message.Message.attach>` method. + Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`. + Additional parameters for the :mailheader:`Content-Type` header are taken from the keyword arguments, or passed into the *_params* argument, which is a keyword dictionary. + .. versionchanged:: 3.6 + Added *policy* keyword-only parameter. .. currentmodule:: email.mime.application -.. class:: MIMEApplication(_data, _subtype='octet-stream', _encoder=email.encoders.encode_base64, **_params) +.. class:: MIMEApplication(_data, _subtype='octet-stream', \ + _encoder=email.encoders.encode_base64, \ + *, policy=compat32, **_params) Module: :mod:`email.mime.application` @@ -109,12 +128,18 @@ Here are the classes: object as necessary. The default encoding is base64. See the :mod:`email.encoders` module for a list of the built-in encoders. + Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`. + *_params* are passed straight through to the base class constructor. + .. versionchanged:: 3.6 + Added *policy* keyword-only parameter. .. currentmodule:: email.mime.audio -.. class:: MIMEAudio(_audiodata, _subtype=None, _encoder=email.encoders.encode_base64, **_params) +.. class:: MIMEAudio(_audiodata, _subtype=None, \ + _encoder=email.encoders.encode_base64, \ + *, policy=compat32, **_params) Module: :mod:`email.mime.audio` @@ -137,12 +162,18 @@ Here are the classes: object as necessary. The default encoding is base64. See the :mod:`email.encoders` module for a list of the built-in encoders. + Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`. + *_params* are passed straight through to the base class constructor. + .. versionchanged:: 3.6 + Added *policy* keyword-only parameter. .. currentmodule:: email.mime.image -.. class:: MIMEImage(_imagedata, _subtype=None, _encoder=email.encoders.encode_base64, **_params) +.. class:: MIMEImage(_imagedata, _subtype=None, \ + _encoder=email.encoders.encode_base64, \ + *, policy=compat32, **_params) Module: :mod:`email.mime.image` @@ -165,13 +196,17 @@ Here are the classes: object as necessary. The default encoding is base64. See the :mod:`email.encoders` module for a list of the built-in encoders. + Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`. + *_params* are passed straight through to the :class:`~email.mime.base.MIMEBase` constructor. + .. versionchanged:: 3.6 + Added *policy* keyword-only parameter. .. currentmodule:: email.mime.message -.. class:: MIMEMessage(_msg, _subtype='rfc822') +.. class:: MIMEMessage(_msg, _subtype='rfc822', *, policy=compat32) Module: :mod:`email.mime.message` @@ -184,10 +219,14 @@ Here are the classes: Optional *_subtype* sets the subtype of the message; it defaults to :mimetype:`rfc822`. + Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`. + + .. versionchanged:: 3.6 + Added *policy* keyword-only parameter. .. currentmodule:: email.mime.text -.. class:: MIMEText(_text, _subtype='plain', _charset=None) +.. class:: MIMEText(_text, _subtype='plain', _charset=None, *, policy=compat32) Module: :mod:`email.mime.text` @@ -211,5 +250,10 @@ Here are the classes: will automatically encode the new payload (and add a new :mailheader:`Content-Transfer-Encoding` header). + Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`. + .. versionchanged:: 3.5 *_charset* also accepts :class:`~email.charset.Charset` instances. + + .. versionchanged:: 3.6 + Added *policy* keyword-only parameter. diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst index b8eb7c516e..c323ebc640 100644 --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -8,210 +8,223 @@ -------------- -Message object structures can be created in one of two ways: they can be created -from whole cloth by instantiating :class:`~email.message.Message` objects and -stringing them together via :meth:`~email.message.Message.attach` and -:meth:`~email.message.Message.set_payload` calls, or they -can be created by parsing a flat text representation of the email message. +Message object structures can be created in one of two ways: they can be +created from whole cloth by creating an :class:`~email.message.EmailMessage` +object, adding headers using the dictionary interface, and adding payload(s) +using :meth:`~email.message.EmailMessage.set_content` and related methods, or +they can be created by parsing a serialized representation of the email +message. The :mod:`email` package provides a standard parser that understands most email -document structures, including MIME documents. You can pass the parser a string -or a file object, and the parser will return to you the root -:class:`~email.message.Message` instance of the object structure. For simple, -non-MIME messages the payload of this root object will likely be a string -containing the text of the message. For MIME messages, the root object will -return ``True`` from its :meth:`~email.message.Message.is_multipart` method, and -the subparts can be accessed via the :meth:`~email.message.Message.get_payload` -and :meth:`~email.message.Message.walk` methods. - -There are actually two parser interfaces available for use, the classic -:class:`Parser` API and the incremental :class:`FeedParser` API. The classic -:class:`Parser` API is fine if you have the entire text of the message in memory -as a string, or if the entire message lives in a file on the file system. -:class:`FeedParser` is more appropriate for when you're reading the message from -a stream which might block waiting for more input (e.g. reading an email message -from a socket). The :class:`FeedParser` can consume and parse the message -incrementally, and only returns the root object when you close the parser [#]_. +document structures, including MIME documents. You can pass the parser a +bytes, string or file object, and the parser will return to you the root +:class:`~email.message.EmailMessage` instance of the object structure. For +simple, non-MIME messages the payload of this root object will likely be a +string containing the text of the message. For MIME messages, the root object +will return ``True`` from its :meth:`~email.message.EmailMessage.is_multipart` +method, and the subparts can be accessed via the payload manipulation methods, +such as :meth:`~email.message.EmailMessage.get_body`, +:meth:`~email.message.EmailMessage.iter_parts`, and +:meth:`~email.message.EmailMessage.walk`. + +There are actually two parser interfaces available for use, the :class:`Parser` +API and the incremental :class:`FeedParser` API. The :class:`Parser` API is +most useful if you have the entire text of the message in memory, or if the +entire message lives in a file on the file system. :class:`FeedParser` is more +appropriate when you are reading the message from a stream which might block +waiting for more input (such as reading an email message from a socket). The +:class:`FeedParser` can consume and parse the message incrementally, and only +returns the root object when you close the parser. Note that the parser can be extended in limited ways, and of course you can -implement your own parser completely from scratch. There is no magical -connection between the :mod:`email` package's bundled parser and the -:class:`~email.message.Message` class, so your custom parser can create message -object trees any way it finds necessary. +implement your own parser completely from scratch. All of the logic that +connects the :mod:`email` package's bundled parser and the +:class:`~email.message.EmailMessage` class is embodied in the :mod:`policy` +class, so a custom parser can create message object trees any way it finds +necessary by implementing custom versions of the appropriate :mod:`policy` +methods. FeedParser API ^^^^^^^^^^^^^^ -The :class:`FeedParser`, imported from the :mod:`email.feedparser` module, -provides an API that is conducive to incremental parsing of email messages, such -as would be necessary when reading the text of an email message from a source -that can block (e.g. a socket). The :class:`FeedParser` can of course be used -to parse an email message fully contained in a string or a file, but the classic -:class:`Parser` API may be more convenient for such use cases. The semantics -and results of the two parser APIs are identical. - -The :class:`FeedParser`'s API is simple; you create an instance, feed it a bunch -of text until there's no more to feed it, then close the parser to retrieve the -root message object. The :class:`FeedParser` is extremely accurate when parsing -standards-compliant messages, and it does a very good job of parsing -non-compliant messages, providing information about how a message was deemed -broken. It will populate a message object's *defects* attribute with a list of -any problems it found in a message. See the :mod:`email.errors` module for the +The :class:`BytesFeedParser`, imported from the :mod:`email.feedparser` module, +provides an API that is conducive to incremental parsing of email messages, +such as would be necessary when reading the text of an email message from a +source that can block (such as a socket). The :class:`BytesFeedParser` can of +course be used to parse an email message fully contained in a :term:`bytes-like +object`, string, or file, but the :class:`BytesParser` API may be more +convenient for such use cases. The semantics and results of the two parser +APIs are identical. + +The :class:`BytesFeedParser`'s API is simple; you create an instance, feed it a +bunch of bytes until there's no more to feed it, then close the parser to +retrieve the root message object. The :class:`BytesFeedParser` is extremely +accurate when parsing standards-compliant messages, and it does a very good job +of parsing non-compliant messages, providing information about how a message +was deemed broken. It will populate a message object's +:attr:`~email.message.EmailMessage.defects` attribute with a list of any +problems it found in a message. See the :mod:`email.errors` module for the list of defects that it can find. -Here is the API for the :class:`FeedParser`: +Here is the API for the :class:`BytesFeedParser`: -.. class:: FeedParser(_factory=email.message.Message, *, policy=policy.compat32) +.. class:: BytesFeedParser(_factory=None, *, policy=policy.compat32) - Create a :class:`FeedParser` instance. Optional *_factory* is a no-argument - callable that will be called whenever a new message object is needed. It - defaults to the :class:`email.message.Message` class. + Create a :class:`BytesFeedParser` instance. Optional *_factory* is a + no-argument callable; if not specified use the + :attr:`~email.policy.Policy.message_factory` from the *policy*. Call + *_factory* whenever a new message object is needed. - If *policy* is specified (it must be an instance of a :mod:`~email.policy` - class) use the rules it specifies to update the representation of the - message. If *policy* is not set, use the :class:`compat32 - <email.policy.Compat32>` policy, which maintains backward compatibility with - the Python 3.2 version of the email package. For more information see the + If *policy* is specified use the rules it specifies to update the + representation of the message. If *policy* is not set, use the + :class:`compat32 <email.policy.Compat32>` policy, which maintains backward + compatibility with the Python 3.2 version of the email package and provides + :class:`~email.message.Message` as the default factory. All other policies + provide :class:`~email.message.EmailMessage` as the default *_factory*. For + more information on what else *policy* controls, see the :mod:`~email.policy` documentation. + Note: **The policy keyword should always be specified**; The default will + change to :data:`email.policy.default` in a future version of Python. + + .. versionadded:: 3.2 + .. versionchanged:: 3.3 Added the *policy* keyword. + .. versionchanged:: 3.6 *_factory* defaults to the policy ``message_factory``. + .. method:: feed(data) - Feed the :class:`FeedParser` some more data. *data* should be a string - containing one or more lines. The lines can be partial and the - :class:`FeedParser` will stitch such partial lines together properly. The - lines in the string can have any of the common three line endings, - carriage return, newline, or carriage return and newline (they can even be - mixed). + Feed the parser some more data. *data* should be a :term:`bytes-like + object` containing one or more lines. The lines can be partial and the + parser will stitch such partial lines together properly. The lines can + have any of the three common line endings: carriage return, newline, or + carriage return and newline (they can even be mixed). + .. method:: close() - Closing a :class:`FeedParser` completes the parsing of all previously fed - data, and returns the root message object. It is undefined what happens - if you feed more data to a closed :class:`FeedParser`. + Complete the parsing of all previously fed data and return the root + message object. It is undefined what happens if :meth:`~feed` is called + after this method has been called. -.. class:: BytesFeedParser(_factory=email.message.Message) +.. class:: FeedParser(_factory=None, *, policy=policy.compat32) - Works exactly like :class:`FeedParser` except that the input to the - :meth:`~FeedParser.feed` method must be bytes and not string. + Works like :class:`BytesFeedParser` except that the input to the + :meth:`~BytesFeedParser.feed` method must be a string. This is of limited + utility, since the only way for such a message to be valid is for it to + contain only ASCII text or, if :attr:`~email.policy.Policy.utf8` is + ``True``, no binary attachments. - .. versionadded:: 3.2 + .. versionchanged:: 3.3 Added the *policy* keyword. -Parser class API -^^^^^^^^^^^^^^^^ +Parser API +^^^^^^^^^^ -The :class:`Parser` class, imported from the :mod:`email.parser` module, +The :class:`BytesParser` class, imported from the :mod:`email.parser` module, provides an API that can be used to parse a message when the complete contents -of the message are available in a string or file. The :mod:`email.parser` -module also provides header-only parsers, called :class:`HeaderParser` and -:class:`BytesHeaderParser`, which can be used if you're only interested in the -headers of the message. :class:`HeaderParser` and :class:`BytesHeaderParser` +of the message are available in a :term:`bytes-like object` or file. The +:mod:`email.parser` module also provides :class:`Parser` for parsing strings, +and header-only parsers, :class:`BytesHeaderParser` and +:class:`HeaderParser`, which can be used if you're only interested in the +headers of the message. :class:`BytesHeaderParser` and :class:`HeaderParser` can be much faster in these situations, since they do not attempt to parse the -message body, instead setting the payload to the raw body as a string. They -have the same API as the :class:`Parser` and :class:`BytesParser` classes. +message body, instead setting the payload to the raw body. -.. versionadded:: 3.3 - The BytesHeaderParser class. +.. class:: BytesParser(_class=None, *, policy=policy.compat32) -.. class:: Parser(_class=email.message.Message, *, policy=policy.compat32) + Create a :class:`BytesParser` instance. The *_class* and *policy* + arguments have the same meaning and sematnics as the *_factory* + and *policy* arguments of :class:`BytesFeedParser`. - The constructor for the :class:`Parser` class takes an optional argument - *_class*. This must be a callable factory (such as a function or a class), and - it is used whenever a sub-message object needs to be created. It defaults to - :class:`~email.message.Message` (see :mod:`email.message`). The factory will - be called without arguments. - - If *policy* is specified (it must be an instance of a :mod:`~email.policy` - class) use the rules it specifies to update the representation of the - message. If *policy* is not set, use the :class:`compat32 - <email.policy.Compat32>` policy, which maintains backward compatibility with - the Python 3.2 version of the email package. For more information see the - :mod:`~email.policy` documentation. + Note: **The policy keyword should always be specified**; The default will + change to :data:`email.policy.default` in a future version of Python. .. versionchanged:: 3.3 Removed the *strict* argument that was deprecated in 2.4. Added the *policy* keyword. - - The other public :class:`Parser` methods are: + .. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``. .. method:: parse(fp, headersonly=False) - Read all the data from the file-like object *fp*, parse the resulting - text, and return the root message object. *fp* must support both the - :meth:`~io.TextIOBase.readline` and the :meth:`~io.TextIOBase.read` - methods on file-like objects. + Read all the data from the binary file-like object *fp*, parse the + resulting bytes, and return the message object. *fp* must support + both the :meth:`~io.IOBase.readline` and the :meth:`~io.IOBase.read` + methods. - The text contained in *fp* must be formatted as a block of :rfc:`2822` + The bytes contained in *fp* must be formatted as a block of :rfc:`5322` + (or, if :attr:`~email.policy.Policy.utf8` is ``True``, :rfc:`6532`) style headers and header continuation lines, optionally preceded by an envelope header. The header block is terminated either by the end of the data or by a blank line. Following the header block is the body of the - message (which may contain MIME-encoded subparts). + message (which may contain MIME-encoded subparts, including subparts + with a :mailheader:`Content-Transfer-Encoding` of ``8bit``. Optional *headersonly* is a flag specifying whether to stop parsing after reading the headers or not. The default is ``False``, meaning it parses the entire contents of the file. - .. method:: parsestr(text, headersonly=False) - Similar to the :meth:`parse` method, except it takes a string object - instead of a file-like object. Calling this method on a string is exactly - equivalent to wrapping *text* in a :class:`~io.StringIO` instance first and - calling :meth:`parse`. + .. method:: parsebytes(bytes, headersonly=False) + + Similar to the :meth:`parse` method, except it takes a :term:`bytes-like + object` instead of a file-like object. Calling this method on a + :term:`bytes-like object` is equivalent to wrapping *bytes* in a + :class:`~io.BytesIO` instance first and calling :meth:`parse`. Optional *headersonly* is as with the :meth:`parse` method. + .. versionadded:: 3.2 -.. class:: BytesParser(_class=email.message.Message, *, policy=policy.compat32) - This class is exactly parallel to :class:`Parser`, but handles bytes input. - The *_class* and *strict* arguments are interpreted in the same way as for - the :class:`Parser` constructor. +.. class:: BytesHeaderParser(_class=None, *, policy=policy.compat32) - If *policy* is specified (it must be an instance of a :mod:`~email.policy` - class) use the rules it specifies to update the representation of the - message. If *policy* is not set, use the :class:`compat32 - <email.policy.Compat32>` policy, which maintains backward compatibility with - the Python 3.2 version of the email package. For more information see the - :mod:`~email.policy` documentation. + Exactly like :class:`BytesParser`, except that *headersonly* + defaults to ``True``. + + .. versionadded:: 3.3 + + +.. class:: Parser(_class=None, *, policy=policy.compat32) + + This class is parallel to :class:`BytesParser`, but handles string input. .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. + .. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``. + .. method:: parse(fp, headersonly=False) - Read all the data from the binary file-like object *fp*, parse the - resulting bytes, and return the message object. *fp* must support - both the :meth:`~io.IOBase.readline` and the :meth:`~io.IOBase.read` - methods on file-like objects. + Read all the data from the text-mode file-like object *fp*, parse the + resulting text, and return the root message object. *fp* must support + both the :meth:`~io.TextIOBase.readline` and the + :meth:`~io.TextIOBase.read` methods on file-like objects. - The bytes contained in *fp* must be formatted as a block of :rfc:`2822` - style headers and header continuation lines, optionally preceded by an - envelope header. The header block is terminated either by the end of the - data or by a blank line. Following the header block is the body of the - message (which may contain MIME-encoded subparts, including subparts - with a :mailheader:`Content-Transfer-Encoding` of ``8bit``. + Other than the text mode requirement, this method operates like + :meth:`BytesParser.parse`. - Optional *headersonly* is a flag specifying whether to stop parsing after - reading the headers or not. The default is ``False``, meaning it parses - the entire contents of the file. - .. method:: parsebytes(text, headersonly=False) + .. method:: parsestr(text, headersonly=False) - Similar to the :meth:`parse` method, except it takes a :term:`bytes-like - object` instead of a file-like object. Calling this method is equivalent - to wrapping *text* in a :class:`~io.BytesIO` instance first and calling - :meth:`parse`. + Similar to the :meth:`parse` method, except it takes a string object + instead of a file-like object. Calling this method on a string is + equivalent to wrapping *text* in a :class:`~io.StringIO` instance first + and calling :meth:`parse`. Optional *headersonly* is as with the :meth:`parse` method. - .. versionadded:: 3.2 + +.. class:: HeaderParser(_class=None, *, policy=policy.compat32) + + Exactly like :class:`Parser`, except that *headersonly* + defaults to ``True``. Since creating a message object structure from a string or a file object is such @@ -220,55 +233,58 @@ in the top-level :mod:`email` package namespace. .. currentmodule:: email -.. function:: message_from_string(s, _class=email.message.Message, *, \ - policy=policy.compat32) - Return a message object structure from a string. This is exactly equivalent to - ``Parser().parsestr(s)``. *_class* and *policy* are interpreted as - with the :class:`~email.parser.Parser` class constructor. +.. function:: message_from_bytes(s, _class=None, *, policy=policy.compat32) + + Return a message object structure from a :term:`bytes-like object`. This is + equivalent to ``BytesParser().parsebytes(s)``. Optional *_class* and + *strict* are interpreted as with the :class:`~email.parser.BytesParser` class + constructor. + .. versionadded:: 3.2 .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. -.. function:: message_from_bytes(s, _class=email.message.Message, *, \ - policy=policy.compat32) - Return a message object structure from a :term:`bytes-like object`. This is exactly - equivalent to ``BytesParser().parsebytes(s)``. Optional *_class* and - *strict* are interpreted as with the :class:`~email.parser.Parser` class +.. function:: message_from_binary_file(fp, _class=None, *, + policy=policy.compat32) + + Return a message object structure tree from an open binary :term:`file + object`. This is equivalent to ``BytesParser().parse(fp)``. *_class* and + *policy* are interpreted as with the :class:`~email.parser.BytesParser` class constructor. .. versionadded:: 3.2 .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. -.. function:: message_from_file(fp, _class=email.message.Message, *, \ - policy=policy.compat32) - Return a message object structure tree from an open :term:`file object`. - This is exactly equivalent to ``Parser().parse(fp)``. *_class* - and *policy* are interpreted as with the :class:`~email.parser.Parser` class - constructor. +.. function:: message_from_string(s, _class=None, *, policy=policy.compat32) + + Return a message object structure from a string. This is equivalent to + ``Parser().parsestr(s)``. *_class* and *policy* are interpreted as + with the :class:`~email.parser.Parser` class constructor. .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. -.. function:: message_from_binary_file(fp, _class=email.message.Message, *, \ - policy=policy.compat32) - Return a message object structure tree from an open binary :term:`file - object`. This is exactly equivalent to ``BytesParser().parse(fp)``. - *_class* and *policy* are interpreted as with the - :class:`~email.parser.Parser` class constructor. +.. function:: message_from_file(fp, _class=None, *, policy=policy.compat32) + + Return a message object structure tree from an open :term:`file object`. + This is equivalent to ``Parser().parse(fp)``. *_class* and *policy* are + interpreted as with the :class:`~email.parser.Parser` class constructor. - .. versionadded:: 3.2 .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. + .. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``. -Here's an example of how you might use this at an interactive Python prompt:: + +Here's an example of how you might use :func:`message_from_bytes` at an +interactive Python prompt:: >>> import email - >>> msg = email.message_from_string(myString) # doctest: +SKIP + >>> msg = email.message_from_bytes(myBytes) # doctest: +SKIP Additional notes @@ -278,35 +294,27 @@ Here are some notes on the parsing semantics: * Most non-\ :mimetype:`multipart` type messages are parsed as a single message object with a string payload. These objects will return ``False`` for - :meth:`~email.message.Message.is_multipart`. Their - :meth:`~email.message.Message.get_payload` method will return a string object. + :meth:`~email.message.EmailMessage.is_multipart`, and + :meth:`~email.message.EmailMessage.iter_parts` will yield an empty list. * All :mimetype:`multipart` type messages will be parsed as a container message object with a list of sub-message objects for their payload. The outer container message will return ``True`` for - :meth:`~email.message.Message.is_multipart` and their - :meth:`~email.message.Message.get_payload` method will return the list of - :class:`~email.message.Message` subparts. + :meth:`~email.message.EmailMessage.is_multipart`, and + :meth:`~email.message.EmailMessage.iter_parts` will yield a list of subparts. -* Most messages with a content type of :mimetype:`message/\*` (e.g. - :mimetype:`message/delivery-status` and :mimetype:`message/rfc822`) will also be - parsed as container object containing a list payload of length 1. Their - :meth:`~email.message.Message.is_multipart` method will return ``True``. - The single element in the list payload will be a sub-message object. +* Most messages with a content type of :mimetype:`message/\*` (such as + :mimetype:`message/delivery-status` and :mimetype:`message/rfc822`) will also + be parsed as container object containing a list payload of length 1. Their + :meth:`~email.message.EmailMessage.is_multipart` method will return ``True``. + The single element yielded by :meth:`~email.message.EmailMessage.iter_parts` + will be a sub-message object. -* Some non-standards compliant messages may not be internally consistent about +* Some non-standards-compliant messages may not be internally consistent about their :mimetype:`multipart`\ -edness. Such messages may have a :mailheader:`Content-Type` header of type :mimetype:`multipart`, but their - :meth:`~email.message.Message.is_multipart` method may return ``False``. + :meth:`~email.message.EmailMessage.is_multipart` method may return ``False``. If such messages were parsed with the :class:`~email.parser.FeedParser`, they will have an instance of the :class:`~email.errors.MultipartInvariantViolationDefect` class in their *defects* attribute list. See :mod:`email.errors` for details. - -.. rubric:: Footnotes - -.. [#] As of email package version 3.0, introduced in Python 2.4, the classic - :class:`~email.parser.Parser` was re-implemented in terms of the - :class:`~email.parser.FeedParser`, so the semantics and results are - identical between the two parsers. - diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index 2a6047d3a4..8a418778b8 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -18,9 +18,12 @@ described by the various email and MIME RFCs. However, the general format of email messages (a block of header fields each consisting of a name followed by a colon followed by a value, the whole block followed by a blank line and an arbitrary 'body'), is a format that has found utility outside of the realm of -email. Some of these uses conform fairly closely to the main RFCs, some do -not. And even when working with email, there are times when it is desirable to -break strict compliance with the RFCs. +email. Some of these uses conform fairly closely to the main email RFCs, some +do not. Even when working with email, there are times when it is desirable to +break strict compliance with the RFCs, such as generating emails that +interoperate with email servers that do not themselves follow the standards, or +that implement extensions you want to use in ways that violate the +standards. Policy objects give the email package the flexibility to handle all these disparate use cases. @@ -31,27 +34,40 @@ control the behavior of various components of the email package during use. email package to alter the default behavior. The settable values and their defaults are described below. -There is a default policy used by all classes in the email package. This -policy is named :class:`Compat32`, with a corresponding pre-defined instance -named :const:`compat32`. It provides for complete backward compatibility (in -some cases, including bug compatibility) with the pre-Python3.3 version of the -email package. +There is a default policy used by all classes in the email package. For all of +the :mod:`~email.parser` classes and the related convenience functions, and for +the :class:`~email.message.Message` class, this is the :class:`Compat32` +policy, via its corresponding pre-defined instance :const:`compat32`. This +policy provides for complete backward compatibility (in some cases, including +bug compatibility) with the pre-Python3.3 version of the email package. + +This default value for the *policy* keyword to +:class:`~email.message.EmailMessage` is the :class:`EmailPolicy` policy, via +its pre-defined instance :data:`~default`. + +When a :class:`~email.message.Message` or :class:`~email.message.EmailMessage` +object is created, it acquires a policy. If the message is created by a +:mod:`~email.parser`, a policy passed to the parser will be the policy used by +the message it creates. If the message is created by the program, then the +policy can be specified when it is created. When a message is passed to a +:mod:`~email.generator`, the generator uses the policy from the message by +default, but you can also pass a specific policy to the generator that will +override the one stored on the message object. + +The default value for the *policy* keyword for the :mod:`email.parser` classes +and the parser convenience functions **will be changing** in a future version of +Python. Therefore you should **always specify explicitly which policy you want +to use** when calling any of the classes and functions described in the +:mod:`~email.parser` module. The first part of this documentation covers the features of :class:`Policy`, an -:term:`abstract base class` that defines the features that are common to all +:term:`abstract base class` that defines the features that are common to all policy objects, including :const:`compat32`. This includes certain hook methods that are called internally by the email package, which a custom policy -could override to obtain different behavior. - -When a :class:`~email.message.Message` object is created, it acquires a policy. -By default this will be :const:`compat32`, but a different policy can be -specified. If the ``Message`` is created by a :mod:`~email.parser`, a policy -passed to the parser will be the policy used by the ``Message`` it creates. If -the ``Message`` is created by the program, then the policy can be specified -when it is created. When a ``Message`` is passed to a :mod:`~email.generator`, -the generator uses the policy from the ``Message`` by default, but you can also -pass a specific policy to the generator that will override the one stored on -the ``Message`` object. +could override to obtain different behavior. The second part describes the +concrete classes :class:`EmailPolicy` and :class:`Compat32`, which implement +the hooks that provide the standard behavior and the backward compatible +behavior and features, respectively. :class:`Policy` instances are immutable, but they can be cloned, accepting the same keyword arguments as the class constructor and returning a new @@ -147,6 +163,7 @@ added matters. To illustrate:: This class defines the following properties, and thus values for the following may be passed in the constructor of any policy class: + .. attribute:: max_line_length The maximum length of any line in the serialized output, not counting the @@ -154,12 +171,14 @@ added matters. To illustrate:: ``0`` or :const:`None` indicates that no line wrapping should be done at all. + .. attribute:: linesep The string to be used to terminate lines in serialized output. The default is ``\n`` because that's the internal end-of-line discipline used by Python, though ``\r\n`` is required by the RFCs. + .. attribute:: cte_type Controls the type of Content Transfer Encodings that may be or are @@ -174,8 +193,8 @@ added matters. To illustrate:: ``8bit`` data is not constrained to be 7 bit clean. Data in headers is still required to be ASCII-only and so will be encoded (see - 'binary_fold' below for an exception), but body parts may use - the ``8bit`` CTE. + :meth:`fold_binary` and :attr:`~EmailPolicy.utf8` below for + exceptions), but body parts may use the ``8bit`` CTE. ======== =============================================================== A ``cte_type`` value of ``8bit`` only works with ``BytesGenerator``, not @@ -183,6 +202,7 @@ added matters. To illustrate:: ``Generator`` is operating under a policy that specifies ``cte_type=8bit``, it will act as if ``cte_type`` is ``7bit``. + .. attribute:: raise_on_defect If :const:`True`, any defects encountered will be raised as errors. If @@ -190,7 +210,6 @@ added matters. To illustrate:: :meth:`register_defect` method. - .. attribute:: mangle_from\_ If :const:`True`, lines starting with *"From "* in the body are @@ -201,19 +220,31 @@ added matters. To illustrate:: .. versionadded:: 3.5 The *mangle_from_* parameter. + + .. attribute:: message_factory + + A factory function for constructing a new empty message object. Used + by the parser when building messages. Defaults to ``None``, in + which case :class:`~email.message.Message` is used. + + .. versionadded:: 3.6 + The following :class:`Policy` method is intended to be called by code using the email library to create policy instances with custom settings: + .. method:: clone(**kw) Return a new :class:`Policy` instance whose attributes have the same values as the current instance, except where those attributes are given new values by the keyword arguments. + The remaining :class:`Policy` methods are called by the email package code, and are not intended to be called by an application using the email package. A custom policy must implement all of these methods. + .. method:: handle_defect(obj, defect) Handle a *defect* found on *obj*. When the email package calls this @@ -224,6 +255,7 @@ added matters. To illustrate:: it is ``True``, *defect* is raised as an exception. If it is ``False`` (the default), *obj* and *defect* are passed to :meth:`register_defect`. + .. method:: register_defect(obj, defect) Register a *defect* on *obj*. In the email package, *defect* will always @@ -236,14 +268,16 @@ added matters. To illustrate:: custom ``Message`` objects) should also provide such an attribute, otherwise defects in parsed messages will raise unexpected errors. + .. method:: header_max_count(name) Return the maximum allowed number of headers named *name*. - Called when a header is added to a :class:`~email.message.Message` - object. If the returned value is not ``0`` or ``None``, and there are - already a number of headers with the name *name* equal to the value - returned, a :exc:`ValueError` is raised. + Called when a header is added to an :class:`~email.message.EmailMessage` + or :class:`~email.message.Message` object. If the returned value is not + ``0`` or ``None``, and there are already a number of headers with the + name *name* greather than or equal to the value returned, a + :exc:`ValueError` is raised. Because the default behavior of ``Message.__setitem__`` is to append the value to the list of headers, it is easy to create duplicate headers @@ -255,6 +289,7 @@ added matters. To illustrate:: The default implementation returns ``None`` for all header names. + .. method:: header_source_parse(sourcelines) The email package calls this method with a list of strings, each string @@ -274,6 +309,7 @@ added matters. To illustrate:: There is no default implementation + .. method:: header_store_parse(name, value) The email package calls this method with the name and value provided by @@ -289,6 +325,7 @@ added matters. To illustrate:: There is no default implementation + .. method:: header_fetch_parse(name, value) The email package calls this method with the *name* and *value* currently @@ -304,6 +341,7 @@ added matters. To illustrate:: There is no default implementation + .. method:: fold(name, value) The email package calls this method with the *name* and *value* currently @@ -316,6 +354,7 @@ added matters. To illustrate:: *value* may contain surrogateescaped binary data. There should be no surrogateescaped binary data in the string returned by the method. + .. method:: fold_binary(name, value) The same as :meth:`fold`, except that the returned value should be a @@ -325,73 +364,6 @@ added matters. To illustrate:: converted back into binary data in the returned bytes object. -.. class:: Compat32(**kw) - - This concrete :class:`Policy` is the backward compatibility policy. It - replicates the behavior of the email package in Python 3.2. The - :mod:`~email.policy` module also defines an instance of this class, - :const:`compat32`, that is used as the default policy. Thus the default - behavior of the email package is to maintain compatibility with Python 3.2. - - The following attributes have values that are different from the - :class:`Policy` default: - - .. attribute:: mangle_from_ - - The default is ``True``. - - The class provides the following concrete implementations of the - abstract methods of :class:`Policy`: - - .. method:: header_source_parse(sourcelines) - - The name is parsed as everything up to the '``:``' and returned - unmodified. The value is determined by stripping leading whitespace off - the remainder of the first line, joining all subsequent lines together, - and stripping any trailing carriage return or linefeed characters. - - .. method:: header_store_parse(name, value) - - The name and value are returned unmodified. - - .. method:: header_fetch_parse(name, value) - - If the value contains binary data, it is converted into a - :class:`~email.header.Header` object using the ``unknown-8bit`` charset. - Otherwise it is returned unmodified. - - .. method:: fold(name, value) - - Headers are folded using the :class:`~email.header.Header` folding - algorithm, which preserves existing line breaks in the value, and wraps - each resulting line to the ``max_line_length``. Non-ASCII binary data are - CTE encoded using the ``unknown-8bit`` charset. - - .. method:: fold_binary(name, value) - - Headers are folded using the :class:`~email.header.Header` folding - algorithm, which preserves existing line breaks in the value, and wraps - each resulting line to the ``max_line_length``. If ``cte_type`` is - ``7bit``, non-ascii binary data is CTE encoded using the ``unknown-8bit`` - charset. Otherwise the original source header is used, with its existing - line breaks and any (RFC invalid) binary data it may contain. - - -An instance of :class:`Compat32` is provided as a module constant: - -.. data:: compat32 - - An instance of :class:`Compat32`, providing backward compatibility with the - behavior of the email package in Python 3.2. - - -.. note:: - - The documentation below describes new policies that are included in the - standard library on a :term:`provisional basis <provisional package>`. - Backwards incompatible changes (up to and including removal of the feature) - may occur if deemed necessary by the core developers. - .. class:: EmailPolicy(**kw) @@ -404,9 +376,15 @@ An instance of :class:`Compat32` is provided as a module constant: on the type of the field. The parsing and folding algorithm fully implement :rfc:`2047` and :rfc:`5322`. + The default value for the :attr:`~email.policy.Policy.message_factory` + attribute is :class:`~email.message.EmailMessage`. + In addition to the settable attributes listed above that apply to all policies, this policy adds the following additional attributes: + .. versionadded:: 3.6 [1]_ + + .. attribute:: utf8 If ``False``, follow :rfc:`5322`, supporting non-ASCII characters in @@ -415,13 +393,14 @@ An instance of :class:`Compat32` is provided as a module constant: formatted in this way may be passed to SMTP servers that support the ``SMTPUTF8`` extension (:rfc:`6531`). + .. attribute:: refold_source If the value for a header in the ``Message`` object originated from a :mod:`~email.parser` (as opposed to being set by a program), this attribute indicates whether or not a generator should refold that value - when transforming the message back into stream form. The possible values - are: + when transforming the message back into serialized form. The possible + values are: ======== =============================================================== ``none`` all source values use original folding @@ -434,23 +413,24 @@ An instance of :class:`Compat32` is provided as a module constant: The default is ``long``. + .. attribute:: header_factory A callable that takes two arguments, ``name`` and ``value``, where ``name`` is a header field name and ``value`` is an unfolded header field value, and returns a string subclass that represents that header. A default ``header_factory`` (see :mod:`~email.headerregistry`) is provided - that understands some of the :RFC:`5322` header field types. (Currently - address fields and date fields have special treatment, while all other - fields are treated as unstructured. This list will be completed before - the extension is marked stable.) + that supports custom parsing for the various address and date :RFC:`5322` + header field types, and the major MIME header field stypes. Support for + additional custom parsing will be added in the future. + .. attribute:: content_manager An object with at least two methods: get_content and set_content. When - the :meth:`~email.message.Message.get_content` or - :meth:`~email.message.Message.set_content` method of a - :class:`~email.message.Message` object is called, it calls the + the :meth:`~email.message.EmailMessage.get_content` or + :meth:`~email.message.EmailMessage.set_content` method of an + :class:`~email.message.EmailMessage` object is called, it calls the corresponding method of this object, passing it the message object as its first argument, and any arguments or keywords that were passed to it as additional arguments. By default ``content_manager`` is set to @@ -462,16 +442,22 @@ An instance of :class:`Compat32` is provided as a module constant: The class provides the following concrete implementations of the abstract methods of :class:`Policy`: + .. method:: header_max_count(name) Returns the value of the :attr:`~email.headerregistry.BaseHeader.max_count` attribute of the specialized class used to represent the header with the given name. + .. method:: header_source_parse(sourcelines) - The implementation of this method is the same as that for the - :class:`Compat32` policy. + + The name is parsed as everything up to the '``:``' and returned + unmodified. The value is determined by stripping leading whitespace off + the remainder of the first line, joining all subsequent lines together, + and stripping any trailing carriage return or linefeed characters. + .. method:: header_store_parse(name, value) @@ -482,6 +468,7 @@ An instance of :class:`Compat32` is provided as a module constant: the value. In this case a ``ValueError`` is raised if the input value contains CR or LF characters. + .. method:: header_fetch_parse(name, value) If the value has a ``name`` attribute, it is returned to unmodified. @@ -490,6 +477,7 @@ An instance of :class:`Compat32` is provided as a module constant: header object is returned. Any surrogateescaped bytes get turned into the unicode unknown-character glyph. + .. method:: fold(name, value) Header folding is controlled by the :attr:`refold_source` policy setting. @@ -508,6 +496,7 @@ An instance of :class:`Compat32` is provided as a module constant: regardless of the ``refold_source`` setting, which causes the binary data to be CTE encoded using the ``unknown-8bit`` charset. + .. method:: fold_binary(name, value) The same as :meth:`fold` if :attr:`~Policy.cte_type` is ``7bit``, except @@ -519,23 +508,27 @@ An instance of :class:`Compat32` is provided as a module constant: ``refold_header`` setting, since there is no way to know whether the binary data consists of single byte characters or multibyte characters. + The following instances of :class:`EmailPolicy` provide defaults suitable for specific application domains. Note that in the future the behavior of these instances (in particular the ``HTTP`` instance) may be adjusted to conform even more closely to the RFCs relevant to their domains. + .. data:: default An instance of ``EmailPolicy`` with all defaults unchanged. This policy uses the standard Python ``\n`` line endings rather than the RFC-correct ``\r\n``. + .. data:: SMTP Suitable for serializing messages in conformance with the email RFCs. Like ``default``, but with ``linesep`` set to ``\r\n``, which is RFC compliant. + .. data:: SMTPUTF8 The same as ``SMTP`` except that :attr:`~EmailPolicy.utf8` is ``True``. @@ -544,11 +537,13 @@ more closely to the RFCs relevant to their domains. sender or recipient addresses have non-ASCII characters (the :meth:`smtplib.SMTP.send_message` method handles this automatically). + .. data:: HTTP Suitable for serializing headers with for use in HTTP traffic. Like ``SMTP`` except that ``max_line_length`` is set to ``None`` (unlimited). + .. data:: strict Convenience instance. The same as ``default`` except that @@ -557,6 +552,7 @@ more closely to the RFCs relevant to their domains. somepolicy + policy.strict + With all of these :class:`EmailPolicies <.EmailPolicy>`, the effective API of the email package is changed from the Python 3.2 API in the following ways: @@ -573,7 +569,7 @@ the email package is changed from the Python 3.2 API in the following ways: and allowed. From the application view, this means that any header obtained through the -:class:`~email.message.Message` is a header object with extra +:class:`~email.message.EmailMessage` is a header object with extra attributes, whose string value is the fully decoded unicode value of the header. Likewise, a header may be assigned a new value, or a new header created, using a unicode string, and the policy will take care of converting @@ -581,3 +577,75 @@ the unicode string into the correct RFC encoded form. The header objects and their attributes are described in :mod:`~email.headerregistry`. + + + +.. class:: Compat32(**kw) + + This concrete :class:`Policy` is the backward compatibility policy. It + replicates the behavior of the email package in Python 3.2. The + :mod:`~email.policy` module also defines an instance of this class, + :const:`compat32`, that is used as the default policy. Thus the default + behavior of the email package is to maintain compatibility with Python 3.2. + + The following attributes have values that are different from the + :class:`Policy` default: + + + .. attribute:: mangle_from_ + + The default is ``True``. + + + The class provides the following concrete implementations of the + abstract methods of :class:`Policy`: + + + .. method:: header_source_parse(sourcelines) + + The name is parsed as everything up to the '``:``' and returned + unmodified. The value is determined by stripping leading whitespace off + the remainder of the first line, joining all subsequent lines together, + and stripping any trailing carriage return or linefeed characters. + + + .. method:: header_store_parse(name, value) + + The name and value are returned unmodified. + + + .. method:: header_fetch_parse(name, value) + + If the value contains binary data, it is converted into a + :class:`~email.header.Header` object using the ``unknown-8bit`` charset. + Otherwise it is returned unmodified. + + + .. method:: fold(name, value) + + Headers are folded using the :class:`~email.header.Header` folding + algorithm, which preserves existing line breaks in the value, and wraps + each resulting line to the ``max_line_length``. Non-ASCII binary data are + CTE encoded using the ``unknown-8bit`` charset. + + + .. method:: fold_binary(name, value) + + Headers are folded using the :class:`~email.header.Header` folding + algorithm, which preserves existing line breaks in the value, and wraps + each resulting line to the ``max_line_length``. If ``cte_type`` is + ``7bit``, non-ascii binary data is CTE encoded using the ``unknown-8bit`` + charset. Otherwise the original source header is used, with its existing + line breaks and any (RFC invalid) binary data it may contain. + + +.. data:: compat32 + + An instance of :class:`Compat32`, providing backward compatibility with the + behavior of the email package in Python 3.2. + + +.. rubric:: Footnotes + +.. [1] Oringally added in 3.3 as a :term:`provisional feature <provisional + package>`. diff --git a/Doc/library/email.rst b/Doc/library/email.rst index e8bb02bff5..c4187dd009 100644 --- a/Doc/library/email.rst +++ b/Doc/library/email.rst @@ -3,50 +3,104 @@ .. module:: email :synopsis: Package supporting the parsing, manipulating, and generating - email messages, including MIME documents. - -.. moduleauthor:: Barry A. Warsaw <barry@python.org> -.. sectionauthor:: Barry A. Warsaw <barry@python.org> -.. Copyright (C) 2001-2010 Python Software Foundation + email messages. +.. moduleauthor:: Barry A. Warsaw <barry@python.org>, + R. David Murray <rdmurray@bitdance.com> +.. sectionauthor:: R. David Murray <rdmurray@bitdance.com> **Source code:** :source:`Lib/email/__init__.py` -------------- -The :mod:`email` package is a library for managing email messages, including -MIME and other :rfc:`2822`\ -based message documents. It is specifically *not* -designed to do any sending of email messages to SMTP (:rfc:`2821`), NNTP, or -other servers; those are functions of modules such as :mod:`smtplib` and -:mod:`nntplib`. The :mod:`email` package attempts to be as RFC-compliant as -possible, supporting in addition to :rfc:`2822`, such MIME-related RFCs as -:rfc:`2045`, :rfc:`2046`, :rfc:`2047`, and :rfc:`2231`. - -The primary distinguishing feature of the :mod:`email` package is that it splits -the parsing and generating of email messages from the internal *object model* -representation of email. Applications using the :mod:`email` package deal -primarily with objects; you can add sub-objects to messages, remove sub-objects -from messages, completely re-arrange the contents, etc. There is a separate -parser and a separate generator which handles the transformation from flat text -to the object model, and then back to flat text again. There are also handy -subclasses for some common MIME object types, and a few miscellaneous utilities -that help with such common tasks as extracting and parsing message field values, -creating RFC-compliant dates, etc. +The :mod:`email` package is a library for managing email messages. It is +specifically *not* designed to do any sending of email messages to SMTP +(:rfc:`2821`), NNTP, or other servers; those are functions of modules such as +:mod:`smtplib` and :mod:`nntplib`. The :mod:`email` package attempts to be as +RFC-compliant as possible, supporting :rfc:`5233` and :rfc:`6532`, as well as +such MIME-related RFCs as :rfc:`2045`, :rfc:`2046`, :rfc:`2047`, :rfc:`2183`, +and :rfc:`2231`. + +The overall structure of the email package can be divided into three major +components, plus a fourth component that controls the behavior of the other +components. + +The central component of the package is an "object model" that represents email +messages. An application interacts with the package primarily through the +object model interface defined in the :mod:`~email.message` sub-module. The +application can use this API to ask questions about an existing email, to +construct a new email, or to add or remove email subcomponents that themselves +use the same object model interface. That is, following the nature of email +messages and their MIME subcomponents, the email object model is a tree +structure of objects that all provide the :class:`~email.message.EmailMessage` +API. + +The other two major components of the package are the :mod:`~email.parser` and +the :mod:`~email.generator`. The parser takes the serialized version of an +email message (a stream of bytes) and converts it into a tree of +:class:`~email.message.EmailMessage` objects. The generator takes an +:class:`~email.message.EmailMessage` and turns it back into a serialized byte +stream. (The parser and generator also handle streams of text characters, but +this usage is discouraged as it is too easy to end up with messages that are +not valid in one way or another.) + +The control component is the :mod:`~email.policy` module. Every +:class:`~email.message.EmailMessage`, every :mod:`~email.generator`, and every +:mod:`~email.parser` has an associated :mod:`~email.policy` object that +controls its behavior. Usually an application only needs to specify the policy +when an :class:`~email.message.EmailMessage` is created, either by directly +instantiating an :class:`~email.message.EmailMessage` to create a new email, +or by parsing an input stream using a :mod:`~email.parser`. But the policy can +be changed when the message is serialized using a :mod:`~email.generator`. +This allows, for example, a generic email message to be parsed from disk, but +to serialize it using standard SMTP settings when sending it to an email +server. + +The email package does its best to hide the details of the various governing +RFCs from the application. Conceptually the application should be able to +treat the email message as a structured tree of unicode text and binary +attachments, without having to worry about how these are represented when +serialized. In practice, however, it is often necessary to be aware of at +least some of the rules governing MIME messages and their structure, +specifically the names and nature of the MIME "content types" and how they +identify multipart documents. For the most part this knowledge should only be +required for more complex applications, and even then it should only be the +high level structure in question, and not the details of how those structures +are represented. Since MIME content types are used widely in modern internet +software (not just email), this will be a familiar concept to many programmers. The following sections describe the functionality of the :mod:`email` package. -The ordering follows a progression that should be common in applications: an -email message is read as flat text from a file or other source, the text is -parsed to produce the object structure of the email message, this structure is -manipulated, and finally, the object tree is rendered back into flat text. - -It is perfectly feasible to create the object structure out of whole cloth --- -i.e. completely from scratch. From there, a similar progression can be taken as -above. - -Also included are detailed specifications of all the classes and modules that -the :mod:`email` package provides, the exception classes you might encounter -while using the :mod:`email` package, some auxiliary utilities, and a few -examples. For users of the older :mod:`mimelib` package, or previous versions -of the :mod:`email` package, a section on differences and porting is provided. +We start with the :mod:`~email.message` object model, which is the primary +interface an application will use, and follow that with the +:mod:`~email.parser` and :mod:`~email.generator` components. Then we cover the +:mod:`~email.policy` controls, which completes the treatment of the main +components of the library. + +The next three sections cover the exceptions the package may raise and the +defects (non-compliance with the RFCs) that the :mod:`~email.parser` may +detect. Then we cover the :mod:`~email.headerregistry` and the +:mod:`~email.contentmanager` sub-components, which provide tools for doing more +detailed manipulation of headers and payloads, respectively. Both of these +components contain features relevant to consuming and producing non-trivial +messages, but also document their extensibility APIs, which will be of interest +to advanced applications. + +Following those is a set of examples of using the fundamental parts of the APIs +covered in the preceding sections. + +The forgoing represent the modern (unicode friendly) API of the email package. +The remaining sections, starting with the :class:`~email.message.Message` +class, cover the legacy :data:`~email.policy.compat32` API that deals much more +directly with the details of how email messages are represented. The +:data:`~email.policy.compat32` API does *not* hide the details of the RFCs from +the application, but for applications that need to operate at that level, they +can be useful tools. This documentation is also relevant for applications that +are still using the :mod:`~email.policy.compat32` API for backward +compatibility reasons. + +.. versionchanged:: 3.6 + Docs reorganized and rewritten to promote the new + :class:`~email.message.EmailMessage`/:class:`~email.policy.EmailPolicy` + API. Contents of the :mod:`email` package documentation: @@ -56,335 +110,43 @@ Contents of the :mod:`email` package documentation: email.parser.rst email.generator.rst email.policy.rst + + email.errors.rst email.headerregistry.rst email.contentmanager.rst + + email.examples.rst + +Legacy API: + +.. toctree:: + + email.compat32-message.rst email.mime.rst email.header.rst email.charset.rst email.encoders.rst - email.errors.rst email.util.rst email.iterators.rst - email-examples.rst .. seealso:: Module :mod:`smtplib` - SMTP protocol client - - Module :mod:`nntplib` - NNTP protocol client - - -.. _email-pkg-history: - -Package History ---------------- - -This table describes the release history of the email package, corresponding to -the version of Python that the package was released with. For purposes of this -document, when you see a note about change or added versions, these refer to the -Python version the change was made in, *not* the email package version. This -table also describes the Python compatibility of each version of the package. - -+---------------+------------------------------+-----------------------+ -| email version | distributed with | compatible with | -+===============+==============================+=======================+ -| :const:`1.x` | Python 2.2.0 to Python 2.2.1 | *no longer supported* | -+---------------+------------------------------+-----------------------+ -| :const:`2.5` | Python 2.2.2+ and Python 2.3 | Python 2.1 to 2.5 | -+---------------+------------------------------+-----------------------+ -| :const:`3.0` | Python 2.4 and Python 2.5 | Python 2.3 to 2.6 | -+---------------+------------------------------+-----------------------+ -| :const:`4.0` | Python 2.5 to Python 2.7 | Python 2.3 to 2.7 | -+---------------+------------------------------+-----------------------+ -| :const:`5.0` | Python 3.0 and Python 3.1 | Python 3.0 to 3.2 | -+---------------+------------------------------+-----------------------+ -| :const:`5.1` | Python 3.2 | Python 3.2 | -+---------------+------------------------------+-----------------------+ - -After Version 5.1 (Python 3.2), the email package no longer has a version that -is separate from the Python version. (See the :ref:`whatsnew-index` documents -for the respective Python versions for details on changes.) - -Here are the major differences between :mod:`email` version 5.1 and -version 5.0: - -* It is once again possible to parse messages containing non-ASCII bytes, - and to reproduce such messages if the data containing the non-ASCII - bytes is not modified. - -* New functions :func:`message_from_bytes` and :func:`message_from_binary_file`, - and new classes :class:`~email.parser.BytesFeedParser` and - :class:`~email.parser.BytesParser` allow binary message data to be parsed - into model objects. - -* Given bytes input to the model, :meth:`~email.message.Message.get_payload` - will by default decode a message body that has a - :mailheader:`Content-Transfer-Encoding` of ``8bit`` using the charset - specified in the MIME headers and return the resulting string. - -* Given bytes input to the model, :class:`~email.generator.Generator` will - convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of - 8bit to instead have a 7bit Content-Transfer-Encoding. - -* New class :class:`~email.generator.BytesGenerator` produces bytes - as output, preserving any unchanged non-ASCII data that was - present in the input used to build the model, including message bodies - with a :mailheader:`Content-Transfer-Encoding` of 8bit. - -Here are the major differences between :mod:`email` version 5.0 and version 4: - -* All operations are on unicode strings. Text inputs must be strings, - text outputs are strings. Outputs are limited to the ASCII character - set and so can be encoded to ASCII for transmission. Inputs are also - limited to ASCII; this is an acknowledged limitation of email 5.0 and - means it can only be used to parse email that is 7bit clean. - -Here are the major differences between :mod:`email` version 4 and version 3: - -* All modules have been renamed according to :pep:`8` standards. For example, - the version 3 module :mod:`email.Message` was renamed to :mod:`email.message` in - version 4. - -* A new subpackage :mod:`email.mime` was added and all the version 3 - :mod:`email.MIME\*` modules were renamed and situated into the :mod:`email.mime` - subpackage. For example, the version 3 module :mod:`email.MIMEText` was renamed - to :mod:`email.mime.text`. - - *Note that the version 3 names will continue to work until Python 2.6*. - -* The :mod:`email.mime.application` module was added, which contains the - :class:`~email.mime.application.MIMEApplication` class. - -* Methods that were deprecated in version 3 have been removed. These include - :meth:`Generator.__call__`, :meth:`Message.get_type`, - :meth:`Message.get_main_type`, :meth:`Message.get_subtype`. - -* Fixes have been added for :rfc:`2231` support which can change some of the - return types for :func:`Message.get_param <email.message.Message.get_param>` - and friends. Under some - circumstances, values which used to return a 3-tuple now return simple strings - (specifically, if all extended parameter segments were unencoded, there is no - language and charset designation expected, so the return type is now a simple - string). Also, %-decoding used to be done for both encoded and unencoded - segments; this decoding is now done only for encoded segments. - -Here are the major differences between :mod:`email` version 3 and version 2: - -* The :class:`~email.parser.FeedParser` class was introduced, and the - :class:`~email.parser.Parser` class was implemented in terms of the - :class:`~email.parser.FeedParser`. All parsing therefore is - non-strict, and parsing will make a best effort never to raise an exception. - Problems found while parsing messages are stored in the message's *defect* - attribute. - -* All aspects of the API which raised :exc:`DeprecationWarning`\ s in version 2 - have been removed. These include the *_encoder* argument to the - :class:`~email.mime.text.MIMEText` constructor, the - :meth:`Message.add_payload` method, the :func:`Utils.dump_address_pair` - function, and the functions :func:`Utils.decode` and :func:`Utils.encode`. - -* New :exc:`DeprecationWarning`\ s have been added to: - :meth:`Generator.__call__`, :meth:`Message.get_type`, - :meth:`Message.get_main_type`, :meth:`Message.get_subtype`, and the *strict* - argument to the :class:`~email.parser.Parser` class. These are expected to - be removed in future versions. - -* Support for Pythons earlier than 2.3 has been removed. - -Here are the differences between :mod:`email` version 2 and version 1: - -* The :mod:`email.Header` and :mod:`email.Charset` modules have been added. - -* The pickle format for :class:`~email.message.Message` instances has changed. - Since this was never (and still isn't) formally defined, this isn't - considered a backward incompatibility. However if your application pickles - and unpickles :class:`~email.message.Message` instances, be aware that in - :mod:`email` version 2, :class:`~email.message.Message` instances now have - private variables *_charset* and *_default_type*. - -* Several methods in the :class:`~email.message.Message` class have been - deprecated, or their signatures changed. Also, many new methods have been - added. See the documentation for the :class:`~email.message.Message` class - for details. The changes should be completely backward compatible. + SMTP (Simple Mail Transport Protcol) client -* The object structure has changed in the face of :mimetype:`message/rfc822` - content types. In :mod:`email` version 1, such a type would be represented - by a scalar payload, i.e. the container message's - :meth:`~email.message.Message.is_multipart` returned false, - :meth:`~email.message.Message.get_payload` was not a list object, but a - single :class:`~email.message.Message` instance. + Module :mod:`poplib` + POP (Post Office Protocol) client - This structure was inconsistent with the rest of the package, so the object - representation for :mimetype:`message/rfc822` content types was changed. In - :mod:`email` version 2, the container *does* return ``True`` from - :meth:`~email.message.Message.is_multipart`, and - :meth:`~email.message.Message.get_payload` returns a list containing a single - :class:`~email.message.Message` item. + Module :mod:`imaplib` + IMAP (Internet Message Access Protocol) client - Note that this is one place that backward compatibility could not be - completely maintained. However, if you're already testing the return type of - :meth:`~email.message.Message.get_payload`, you should be fine. You just need - to make sure your code doesn't do a :meth:`~email.message.Message.set_payload` - with a :class:`~email.message.Message` instance on a container with a content - type of :mimetype:`message/rfc822`. - -* The :class:`~email.parser.Parser` constructor's *strict* argument was added, - and its :meth:`~email.parser.Parser.parse` and - :meth:`~email.parser.Parser.parsestr` methods grew a *headersonly* argument. - The *strict* flag was also added to functions :func:`email.message_from_file` - and :func:`email.message_from_string`. - -* :meth:`Generator.__call__` is deprecated; use :meth:`Generator.flatten - <email.generator.Generator.flatten>` instead. The - :class:`~email.generator.Generator` class has also grown the - :meth:`~email.generator.Generator.clone` method. - -* The :class:`~email.generator.DecodedGenerator` class in the - :mod:`email.generator` module was added. - -* The intermediate base classes - :class:`~email.mime.nonmultipart.MIMENonMultipart` and - :class:`~email.mime.multipart.MIMEMultipart` have been added, and interposed - in the class hierarchy for most of the other MIME-related derived classes. - -* The *_encoder* argument to the :class:`~email.mime.text.MIMEText` constructor - has been deprecated. Encoding now happens implicitly based on the - *_charset* argument. - -* The following functions in the :mod:`email.Utils` module have been deprecated: - :func:`dump_address_pairs`, :func:`decode`, and :func:`encode`. The following - functions have been added to the module: :func:`make_msgid`, - :func:`decode_rfc2231`, :func:`encode_rfc2231`, and :func:`decode_params`. - -* The non-public function :func:`email.Iterators._structure` was added. - - -Differences from :mod:`mimelib` -------------------------------- - -The :mod:`email` package was originally prototyped as a separate library called -`mimelib <http://mimelib.sourceforge.net/>`_. Changes have been made so that method names -are more consistent, and some methods or modules have either been added or -removed. The semantics of some of the methods have also changed. For the most -part, any functionality available in :mod:`mimelib` is still available in the -:mod:`email` package, albeit often in a different way. Backward compatibility -between the :mod:`mimelib` package and the :mod:`email` package was not a -priority. - -Here is a brief description of the differences between the :mod:`mimelib` and -the :mod:`email` packages, along with hints on how to port your applications. - -Of course, the most visible difference between the two packages is that the -package name has been changed to :mod:`email`. In addition, the top-level -package has the following differences: - -* :func:`messageFromString` has been renamed to :func:`message_from_string`. - -* :func:`messageFromFile` has been renamed to :func:`message_from_file`. - -The :class:`~email.message.Message` class has the following differences: - -* The method :meth:`asString` was renamed to - :meth:`~email.message.Message.as_string`. - -* The method :meth:`ismultipart` was renamed to - :meth:`~email.message.Message.is_multipart`. - -* The :meth:`~email.message.Message.get_payload` method has grown a *decode* - optional argument. - -* The method :meth:`getall` was renamed to - :meth:`~email.message.Message.get_all`. - -* The method :meth:`addheader` was renamed to - :meth:`~email.message.Message.add_header`. - -* The method :meth:`gettype` was renamed to :meth:`get_type`. - -* The method :meth:`getmaintype` was renamed to :meth:`get_main_type`. - -* The method :meth:`getsubtype` was renamed to :meth:`get_subtype`. - -* The method :meth:`getparams` was renamed to - :meth:`~email.message.Message.get_params`. Also, whereas :meth:`getparams` - returned a list of strings, :meth:`~email.message.Message.get_params` returns - a list of 2-tuples, effectively the key/value pairs of the parameters, split - on the ``'='`` sign. - -* The method :meth:`getparam` was renamed to - :meth:`~email.message.Message.get_param`. - -* The method :meth:`getcharsets` was renamed to - :meth:`~email.message.Message.get_charsets`. - -* The method :meth:`getfilename` was renamed to - :meth:`~email.message.Message.get_filename`. - -* The method :meth:`getboundary` was renamed to - :meth:`~email.message.Message.get_boundary`. - -* The method :meth:`setboundary` was renamed to - :meth:`~email.message.Message.set_boundary`. - -* The method :meth:`getdecodedpayload` was removed. To get similar - functionality, pass the value 1 to the *decode* flag of the - :meth:`~email.message.Message.get_payload` method. - -* The method :meth:`getpayloadastext` was removed. Similar functionality is - supported by the :class:`~email.generator.DecodedGenerator` class in the - :mod:`email.generator` module. - -* The method :meth:`getbodyastext` was removed. You can get similar - functionality by creating an iterator with - :func:`~email.iterators.typed_subpart_iterator` in the :mod:`email.iterators` - module. - -The :class:`~email.parser.Parser` class has no differences in its public -interface. It does have some additional smarts to recognize -:mimetype:`message/delivery-status` type messages, which it represents as a -:class:`~email.message.Message` instance containing separate -:class:`~email.message.Message` subparts for each header block in the delivery -status notification [#]_. - -The :class:`~email.generator.Generator` class has no differences in its public -interface. There is a new class in the :mod:`email.generator` module though, -called :class:`~email.generator.DecodedGenerator` which provides most of the -functionality previously available in the :meth:`Message.getpayloadastext` -method. - -The following modules and classes have been changed: - -* The :class:`~email.mime.base.MIMEBase` class constructor arguments *_major* - and *_minor* have changed to *_maintype* and *_subtype* respectively. - -* The ``Image`` class/module has been renamed to ``MIMEImage``. The *_minor* - argument has been renamed to *_subtype*. - -* The ``Text`` class/module has been renamed to ``MIMEText``. The *_minor* - argument has been renamed to *_subtype*. - -* The ``MessageRFC822`` class/module has been renamed to ``MIMEMessage``. Note - that an earlier version of :mod:`mimelib` called this class/module ``RFC822``, - but that clashed with the Python standard library module :mod:`rfc822` on some - case-insensitive file systems. - - Also, the :class:`~email.mime.message.MIMEMessage` class now represents any - kind of MIME message - with main type :mimetype:`message`. It takes an optional argument *_subtype* - which is used to set the MIME subtype. *_subtype* defaults to - :mimetype:`rfc822`. - -:mod:`mimelib` provided some utility functions in its :mod:`address` and -:mod:`date` modules. All of these functions have been moved to the -:mod:`email.utils` module. - -The ``MsgReader`` class/module has been removed. Its functionality is most -closely supported in the :func:`~email.iterators.body_line_iterator` function -in the :mod:`email.iterators` module. + Module :mod:`nntplib` + NNTP (Net News Transport Protocol) client -.. rubric:: Footnotes + Module :mod:`mailbox` + Tools for creating, reading, and managing collections of messages on disk + using a variety standard formats. -.. [#] Delivery Status Notifications (DSN) are defined in :rfc:`1894`. + Module :mod:`smtpd` + SMTP server framework (primarily useful for testing) diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst index 5cff7465df..63fae2ab84 100644 --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -8,7 +8,43 @@ -------------- -There are several useful utilities provided in the :mod:`email.utils` module: +There are a couple of useful utilities provided in the :mod:`email.utils` +module: + +.. function:: localtime(dt=None) + + Return local time as an aware datetime object. If called without + arguments, return current time. Otherwise *dt* argument should be a + :class:`~datetime.datetime` instance, and it is converted to the local time + zone according to the system time zone database. If *dt* is naive (that + is, ``dt.tzinfo`` is ``None``), it is assumed to be in local time. In this + case, a positive or zero value for *isdst* causes ``localtime`` to presume + initially that summer time (for example, Daylight Saving Time) is or is not + (respectively) in effect for the specified time. A negative value for + *isdst* causes the ``localtime`` to attempt to divine whether summer time + is in effect for the specified time. + + .. versionadded:: 3.3 + + +.. function:: make_msgid(idstring=None, domain=None) + + Returns a string suitable for an :rfc:`2822`\ -compliant + :mailheader:`Message-ID` header. Optional *idstring* if given, is a string + used to strengthen the uniqueness of the message id. Optional *domain* if + given provides the portion of the msgid after the '@'. The default is the + local hostname. It is not normally necessary to override this default, but + may be useful certain cases, such as a constructing distributed system that + uses a consistent domain name across multiple hosts. + + .. versionchanged:: 3.2 + Added the *domain* keyword. + + +The remaining functions are part of the legacy (``Compat32``) email API. There +is no need to directly use these with the new API, since the parsing and +formatting they provide is done automatically by the header parsing machinery +of the new API. .. function:: quote(str) @@ -141,36 +177,6 @@ There are several useful utilities provided in the :mod:`email.utils` module: .. versionadded:: 3.3 -.. function:: localtime(dt=None) - - Return local time as an aware datetime object. If called without - arguments, return current time. Otherwise *dt* argument should be a - :class:`~datetime.datetime` instance, and it is converted to the local time - zone according to the system time zone database. If *dt* is naive (that - is, ``dt.tzinfo`` is ``None``), it is assumed to be in local time. In this - case, a positive or zero value for *isdst* causes ``localtime`` to presume - initially that summer time (for example, Daylight Saving Time) is or is not - (respectively) in effect for the specified time. A negative value for - *isdst* causes the ``localtime`` to attempt to divine whether summer time - is in effect for the specified time. - - .. versionadded:: 3.3 - - -.. function:: make_msgid(idstring=None, domain=None) - - Returns a string suitable for an :rfc:`2822`\ -compliant - :mailheader:`Message-ID` header. Optional *idstring* if given, is a string - used to strengthen the uniqueness of the message id. Optional *domain* if - given provides the portion of the msgid after the '@'. The default is the - local hostname. It is not normally necessary to override this default, but - may be useful certain cases, such as a constructing distributed system that - uses a consistent domain name across multiple hosts. - - .. versionchanged:: 3.2 - Added the *domain* keyword. - - .. function:: decode_rfc2231(s) Decode the string *s* according to :rfc:`2231`. diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index a3d5afcb9e..87aa8b140b 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -23,9 +23,10 @@ by identity, and the enumeration itself can be iterated over. Module Contents --------------- -This module defines two enumeration classes that can be used to define unique -sets of names and values: :class:`Enum` and :class:`IntEnum`. It also defines -one decorator, :func:`unique`. +This module defines four enumeration classes that can be used to define unique +sets of names and values: :class:`Enum`, :class:`IntEnum`, and +:class:`IntFlags`. It also defines one decorator, :func:`unique`, and one +helper, :class:`auto`. .. class:: Enum @@ -37,10 +38,27 @@ one decorator, :func:`unique`. Base class for creating enumerated constants that are also subclasses of :class:`int`. +.. class:: IntFlag + + Base class for creating enumerated constants that can be combined using + the bitwise operators without losing their :class:`IntFlag` membership. + :class:`IntFlag` members are also subclasses of :class:`int`. + +.. class:: Flag + + Base class for creating enumerated constants that can be combined using + the bitwise operations without losing their :class:`Flag` membership. + .. function:: unique Enum class decorator that ensures only one name is bound to any one value. +.. class:: auto + + Instances are replaced with an appropriate value for Enum members. + +.. versionadded:: 3.6 ``Flag``, ``IntFlag``, ``auto`` + Creating an Enum ---------------- @@ -57,6 +75,13 @@ follows:: ... blue = 3 ... +.. note:: Enum member values + + Member values can be anything: :class:`int`, :class:`str`, etc.. If + the exact value is unimportant you may use :class:`auto` instances and an + appropriate value will be chosen for you. Care must be taken if you mix + :class:`auto` with other values. + .. note:: Nomenclature - The class :class:`Color` is an *enumeration* (or *enum*) @@ -212,6 +237,42 @@ found :exc:`ValueError` is raised with the details:: ValueError: duplicate values found in <enum 'Mistake'>: four -> three +Using automatic values +---------------------- + +If the exact value is unimportant you can use :class:`auto`:: + + >>> from enum import Enum, auto + >>> class Color(Enum): + ... red = auto() + ... blue = auto() + ... green = auto() + ... + >>> list(Color) + [<Color.red: 1>, <Color.blue: 2>, <Color.green: 3>] + +The values are chosen by :func:`_generate_next_value_`, which can be +overridden:: + + >>> class AutoName(Enum): + ... def _generate_next_value_(name, start, count, last_values): + ... return name + ... + >>> class Ordinal(AutoName): + ... north = auto() + ... south = auto() + ... east = auto() + ... west = auto() + ... + >>> list(Ordinal) + [<Ordinal.north: 'north'>, <Ordinal.south: 'south'>, <Ordinal.east: 'east'>, <Ordinal.west: 'west'>] + +.. note:: + + The goal of the default :meth:`_generate_next_value_` methods is to provide + the next :class:`int` in sequence with the last :class:`int` provided, but + the way it does this is an implementation detail and may change. + Iteration --------- @@ -257,7 +318,7 @@ members are not integers (but see `IntEnum`_ below):: >>> Color.red < Color.blue Traceback (most recent call last): File "<stdin>", line 1, in <module> - TypeError: unorderable types: Color() < Color() + TypeError: '<' not supported between instances of 'Color' and 'Color' Equality comparisons are defined though:: @@ -478,7 +539,7 @@ Derived Enumerations IntEnum ^^^^^^^ -A variation of :class:`Enum` is provided which is also a subclass of +The first variation of :class:`Enum` that is provided is also a subclass of :class:`int`. Members of an :class:`IntEnum` can be compared to integers; by extension, integer enumerations of different types can also be compared to each other:: @@ -521,13 +582,124 @@ However, they still can't be compared to standard :class:`Enum` enumerations:: >>> [i for i in range(Shape.square)] [0, 1] -For the vast majority of code, :class:`Enum` is strongly recommended, -since :class:`IntEnum` breaks some semantic promises of an enumeration (by -being comparable to integers, and thus by transitivity to other -unrelated enumerations). It should be used only in special cases where -there's no other choice; for example, when integer constants are -replaced with enumerations and backwards compatibility is required with code -that still expects integers. + +IntFlag +^^^^^^^ + +The next variation of :class:`Enum` provided, :class:`IntFlag`, is also based +on :class:`int`. The difference being :class:`IntFlag` members can be combined +using the bitwise operators (&, \|, ^, ~) and the result is still an +:class:`IntFlag` member. However, as the name implies, :class:`IntFlag` +members also subclass :class:`int` and can be used wherever an :class:`int` is +used. Any operation on an :class:`IntFlag` member besides the bit-wise +operations will lose the :class:`IntFlag` membership. + +.. versionadded:: 3.6 + +Sample :class:`IntFlag` class:: + + >>> from enum import IntFlag + >>> class Perm(IntFlag): + ... R = 4 + ... W = 2 + ... X = 1 + ... + >>> Perm.R | Perm.W + <Perm.R|W: 6> + >>> Perm.R + Perm.W + 6 + >>> RW = Perm.R | Perm.W + >>> Perm.R in RW + True + +It is also possible to name the combinations:: + + >>> class Perm(IntFlag): + ... R = 4 + ... W = 2 + ... X = 1 + ... RWX = 7 + >>> Perm.RWX + <Perm.RWX: 7> + >>> ~Perm.RWX + <Perm.-8: -8> + +Another important difference between :class:`IntFlag` and :class:`Enum` is that +if no flags are set (the value is 0), its boolean evaluation is :data:`False`:: + + >>> Perm.R & Perm.X + <Perm.0: 0> + >>> bool(Perm.R & Perm.X) + False + +Because :class:`IntFlag` members are also subclasses of :class:`int` they can +be combined with them:: + + >>> Perm.X | 8 + <Perm.8|X: 9> + + +Flag +^^^^ + +The last variation is :class:`Flag`. Like :class:`IntFlag`, :class:`Flag` +members can be combined using the bitwise operators (&, \|, ^, ~). Unlike +:class:`IntFlag`, they cannot be combined with, nor compared against, any +other :class:`Flag` enumeration, nor :class:`int`. While it is possible to +specify the values directly it is recommended to use :class:`auto` as the +value and let :class:`Flag` select an appropriate value. + +.. versionadded:: 3.6 + +Like :class:`IntFlag`, if a combination of :class:`Flag` members results in no +flags being set, the boolean evaluation is :data:`False`:: + + >>> from enum import Flag + >>> class Color(Flag): + ... red = auto() + ... blue = auto() + ... green = auto() + ... + >>> Color.red & Color.green + <Color.0: 0> + >>> bool(Color.red & Color.green) + False + +Individual flags should have values that are powers of two (1, 2, 4, 8, ...), +while combinations of flags won't:: + + >>> class Color(Flag): + ... red = auto() + ... blue = auto() + ... green = auto() + ... white = red | blue | green + ... + >>> Color.white + <Color.white: 7> + +Giving a name to the "no flags set" condition does not change its boolean +value:: + + >>> class Color(Flag): + ... black = 0 + ... red = auto() + ... blue = auto() + ... green = auto() + ... + >>> Color.black + <Color.black: 0> + >>> bool(Color.black) + False + +.. note:: + + For the majority of new code, :class:`Enum` and :class:`Flag` are strongly + recommended, since :class:`IntEnum` and :class:`IntFlag` break some + semantic promises of an enumeration (by being comparable to integers, and + thus by transitivity to other unrelated enumerations). :class:`IntEnum` + and :class:`IntFlag` should be used only in cases where :class:`Enum` and + :class:`Flag` will not do; for example, when integer constants are replaced + with enumerations, or for interoperability with other systems. Others @@ -558,7 +730,8 @@ Some rules: 4. %-style formatting: `%s` and `%r` call the :class:`Enum` class's :meth:`__str__` and :meth:`__repr__` respectively; other codes (such as `%i` or `%h` for IntEnum) treat the enum member as its mixed-in type. -5. :meth:`str.format` (or :func:`format`) will use the mixed-in +5. :ref:`Formatted string literals <f-strings>`, :meth:`str.format`, + and :func:`format` will use the mixed-in type's :meth:`__format__`. If the :class:`Enum` class's :func:`str` or :func:`repr` is desired, use the `!s` or `!r` format codes. @@ -566,18 +739,87 @@ Some rules: Interesting examples -------------------- -While :class:`Enum` and :class:`IntEnum` are expected to cover the majority of -use-cases, they cannot cover them all. Here are recipes for some different -types of enumerations that can be used directly, or as examples for creating -one's own. +While :class:`Enum`, :class:`IntEnum`, :class:`IntFlag`, and :class:`Flag` are +expected to cover the majority of use-cases, they cannot cover them all. Here +are recipes for some different types of enumerations that can be used directly, +or as examples for creating one's own. + + +Omitting values +^^^^^^^^^^^^^^^ +In many use-cases one doesn't care what the actual value of an enumeration +is. There are several ways to define this type of simple enumeration: -AutoNumber -^^^^^^^^^^ +- use instances of :class:`auto` for the value +- use instances of :class:`object` as the value +- use a descriptive string as the value +- use a tuple as the value and a custom :meth:`__new__` to replace the + tuple with an :class:`int` value -Avoids having to specify the value for each enumeration member:: +Using any of these methods signifies to the user that these values are not +important, and also enables one to add, remove, or reorder members without +having to renumber the remaining members. - >>> class AutoNumber(Enum): +Whichever method you choose, you should provide a :meth:`repr` that also hides +the (unimportant) value:: + + >>> class NoValue(Enum): + ... def __repr__(self): + ... return '<%s.%s>' % (self.__class__.__name__, self.name) + ... + + +Using :class:`auto` +""""""""""""""""""" + +Using :class:`object` would look like:: + + >>> class Color(NoValue): + ... red = auto() + ... blue = auto() + ... green = auto() + ... + >>> Color.green + <Color.green> + + +Using :class:`object` +""""""""""""""""""""" + +Using :class:`object` would look like:: + + >>> class Color(NoValue): + ... red = object() + ... green = object() + ... blue = object() + ... + >>> Color.green + <Color.green> + + +Using a descriptive string +"""""""""""""""""""""""""" + +Using a string as the value would look like:: + + >>> class Color(NoValue): + ... red = 'stop' + ... green = 'go' + ... blue = 'too fast!' + ... + >>> Color.green + <Color.green> + >>> Color.green.value + 'go' + + +Using a custom :meth:`__new__` +"""""""""""""""""""""""""""""" + +Using an auto-numbering :meth:`__new__` would look like:: + + >>> class AutoNumber(NoValue): ... def __new__(cls): ... value = len(cls.__members__) + 1 ... obj = object.__new__(cls) @@ -589,8 +831,11 @@ Avoids having to specify the value for each enumeration member:: ... green = () ... blue = () ... - >>> Color.green.value == 2 - True + >>> Color.green + <Color.green> + >>> Color.green.value + 2 + .. note:: @@ -730,10 +975,60 @@ member instances. Finer Points ^^^^^^^^^^^^ -:class:`Enum` members are instances of an :class:`Enum` class, and even -though they are accessible as `EnumClass.member`, they should not be accessed -directly from the member as that lookup may fail or, worse, return something -besides the :class:`Enum` member you looking for:: +Supported ``__dunder__`` names +"""""""""""""""""""""""""""""" + +:attr:`__members__` is an :class:`OrderedDict` of ``member_name``:``member`` +items. It is only available on the class. + +:meth:`__new__`, if specified, must create and return the enum members; it is +also a very good idea to set the member's :attr:`_value_` appropriately. Once +all the members are created it is no longer used. + + +Supported ``_sunder_`` names +"""""""""""""""""""""""""""" + +- ``_name_`` -- name of the member +- ``_value_`` -- value of the member; can be set / modified in ``__new__`` + +- ``_missing_`` -- a lookup function used when a value is not found; may be + overridden +- ``_order_`` -- used in Python 2/3 code to ensure member order is consistent + (class attribute, removed during class creation) +- ``_generate_next_value_`` -- used by the `Functional API`_ and by + :class:`auto` to get an appropriate value for an enum member; may be + overridden + +.. versionadded:: 3.6 ``_missing_``, ``_order_``, ``_generate_next_value_`` + +To help keep Python 2 / Python 3 code in sync an :attr:`_order_` attribute can +be provided. It will be checked against the actual order of the enumeration +and raise an error if the two do not match:: + + >>> class Color(Enum): + ... _order_ = 'red green blue' + ... red = 1 + ... blue = 3 + ... green = 2 + ... + Traceback (most recent call last): + ... + TypeError: member order does not match _order_ + +.. note:: + + In Python 2 code the :attr:`_order_` attribute is necessary as definition + order is lost before it can be recorded. + +``Enum`` member type +"""""""""""""""""""" + +:class:`Enum` members are instances of their :class:`Enum` class, and are +normally accessed as ``EnumClass.member``. Under certain circumstances they +can also be accessed as ``EnumClass.member.member``, but you should never do +this as that lookup may fail or, worse, return something besides the +:class:`Enum` member you are looking for:: >>> class FieldTypes(Enum): ... name = 0 @@ -747,7 +1042,24 @@ besides the :class:`Enum` member you looking for:: .. versionchanged:: 3.5 -The :attr:`__members__` attribute is only available on the class. + +Boolean value of ``Enum`` classes and members +""""""""""""""""""""""""""""""""""""""""""""" + +:class:`Enum` members that are mixed with non-:class:`Enum` types (such as +:class:`int`, :class:`str`, etc.) are evaluated according to the mixed-in +type's rules; otherwise, all members evaluate as :data:`True`. To make your +own Enum's boolean evaluation depend on the member's value add the following to +your class:: + + def __bool__(self): + return bool(self.value) + +:class:`Enum` classes always evaluate as :data:`True`. + + +``Enum`` classes with methods +""""""""""""""""""""""""""""" If you give your :class:`Enum` subclass extra methods, like the `Planet`_ class above, those methods will show up in a :func:`dir` of the member, @@ -758,11 +1070,23 @@ but not of the class:: >>> dir(Planet.EARTH) ['__class__', '__doc__', '__module__', 'name', 'surface_gravity', 'value'] -The :meth:`__new__` method will only be used for the creation of the -:class:`Enum` members -- after that it is replaced. Any custom :meth:`__new__` -method must create the object and set the :attr:`_value_` attribute -appropriately. -If you wish to change how :class:`Enum` members are looked up you should either -write a helper function or a :func:`classmethod` for the :class:`Enum` -subclass. +Combining members of ``Flag`` +""""""""""""""""""""""""""""" + +If a combination of Flag members is not named, the :func:`repr` will include +all named flags and all named combinations of flags that are in the value:: + + >>> class Color(Flag): + ... red = auto() + ... green = auto() + ... blue = auto() + ... magenta = red | blue + ... yellow = red | green + ... cyan = green | blue + ... + >>> Color(3) # named combination + <Color.yellow: 3> + >>> Color(7) # not named combination + <Color.cyan|magenta|blue|yellow|green|red: 7> + diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 5a7193393c..a428f5165f 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -170,8 +170,9 @@ The following exceptions are the exceptions that are usually raised. .. exception:: ImportError - Raised when an :keyword:`import` statement fails to find the module definition - or when a ``from ... import`` fails to find a name that is to be imported. + Raised when the :keyword:`import` statement has troubles trying to + load a module. Also raised when the "from list" in ``from ... import`` + has a name that cannot be found. The :attr:`name` and :attr:`path` attributes can be set using keyword-only arguments to the constructor. When set they represent the name of the module @@ -181,6 +182,14 @@ The following exceptions are the exceptions that are usually raised. .. versionchanged:: 3.3 Added the :attr:`name` and :attr:`path` attributes. +.. exception:: ModuleNotFoundError + + A subclass of :exc:`ImportError` which is raised by :keyword:`import` + when a module could not be located. It is also raised when ``None`` + is found in :data:`sys.modules`. + + .. versionadded:: 3.6 + .. exception:: IndexError @@ -228,9 +237,21 @@ The following exceptions are the exceptions that are usually raised. .. exception:: NotImplementedError This exception is derived from :exc:`RuntimeError`. In user defined base - classes, abstract methods should raise this exception when they require derived - classes to override the method. + classes, abstract methods should raise this exception when they require + derived classes to override the method, or while the class is being + developed to indicate that the real implementation still needs to be added. + + .. note:: + It should not be used to indicate that an operater or method is not + meant to be supported at all -- in that case either leave the operator / + method undefined or, if a subclass, set it to :data:`None`. + + .. note:: + + ``NotImplementedError`` and ``NotImplemented`` are not interchangeable, + even though they have similar names and purposes. See + :data:`NotImplemented` for details on when to use it. .. exception:: OSError([arg]) OSError(errno, strerror[, filename[, winerror[, filename2]]]) @@ -436,6 +457,15 @@ The following exceptions are the exceptions that are usually raised. Raised when an operation or function is applied to an object of inappropriate type. The associated value is a string giving details about the type mismatch. + This exception may be raised by user code to indicate that an attempted + operation on an object is not supported, and is not meant to be. If an object + is meant to support a given operation but has not yet provided an + implementation, :exc:`NotImplementedError` is the proper exception to raise. + + Passing arguments of the wrong type (e.g. passing a :class:`list` when an + :class:`int` is expected) should result in a :exc:`TypeError`, but passing + arguments with the wrong value (e.g. a number outside expected boundaries) + should result in a :exc:`ValueError`. .. exception:: UnboundLocalError diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst index deedea1f26..d0c4cd07c5 100644 --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -70,6 +70,9 @@ Fault handler state .. versionchanged:: 3.5 Added support for passing file descriptor to this function. + .. versionchanged:: 3.6 + On Windows, a handler for Windows exception is also installed. + .. function:: disable() Disable the fault handler: uninstall the signal handlers installed by diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index aa4c529b2a..5881fefe29 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -72,9 +72,8 @@ The following function is the primary interface of this module: .. versionchanged:: 3.2 Can be used as a context manager. - .. versionchanged:: 3.5.2 - The *bufsize* parameter is no longer used. - + .. deprecated-removed:: 3.6 3.8 + The *bufsize* parameter. The following functions use the global state created by :func:`fileinput.input`; if there is no active state, :exc:`RuntimeError` is raised. @@ -167,8 +166,8 @@ available for subclassing as well: .. deprecated:: 3.4 The ``'rU'`` and ``'U'`` modes. - .. versionchanged:: 3.5.2 - The *bufsize* parameter is no longer used. + .. deprecated-removed:: 3.6 3.8 + The *bufsize* parameter. **Optional in-place filtering:** if the keyword argument ``inplace=True`` is @@ -195,10 +194,14 @@ The two following opening hooks are provided by this module: Usage example: ``fi = fileinput.FileInput(openhook=fileinput.hook_compressed)`` -.. function:: hook_encoded(encoding) +.. function:: hook_encoded(encoding, errors=None) Returns a hook which opens each file with :func:`open`, using the given - *encoding* to read the file. + *encoding* and *errors* to read the file. Usage example: ``fi = - fileinput.FileInput(openhook=fileinput.hook_encoded("iso-8859-1"))`` + fileinput.FileInput(openhook=fileinput.hook_encoded("utf-8", + "surrogateescape"))`` + + .. versionchanged:: 3.6 + Added the optional *errors* parameter. diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst index 1e35f37f44..b8c1dcfef2 100644 --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -97,6 +97,13 @@ The module defines the following items: :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see :data:`ssl.HAS_SNI`). + .. deprecated:: 3.6 + + *keyfile* and *certfile* are deprecated in favor of *context*. + Please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let + :func:`ssl.create_default_context` select the system's trusted CA + certificates for you. + Here's a sample session using the :class:`FTP_TLS` class:: >>> ftps = FTP_TLS('ftp.pureftpd.org') diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 5af9fe4e79..c4fcd98252 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -271,6 +271,9 @@ are always available. They are listed here in alphabetical order. The complex type is described in :ref:`typesnumeric`. + .. versionchanged:: 3.6 + Grouping digits with underscores as in code literals is allowed. + .. function:: delattr(object, name) @@ -531,11 +534,14 @@ are always available. They are listed here in alphabetical order. The float type is described in :ref:`typesnumeric`. - .. index:: - single: __format__ - single: string; format() (built-in function) + .. versionchanged:: 3.6 + Grouping digits with underscores as in code literals is allowed. +.. index:: + single: __format__ + single: string; format() (built-in function) + .. function:: format(value[, format_spec]) Convert a *value* to a "formatted" representation, as controlled by @@ -702,6 +708,10 @@ are always available. They are listed here in alphabetical order. :meth:`base.__int__ <object.__int__>` instead of :meth:`base.__index__ <object.__index__>`. + .. versionchanged:: 3.6 + Grouping digits with underscores as in code literals is allowed. + + .. function:: isinstance(object, classinfo) Return true if the *object* argument is an instance of the *classinfo* @@ -878,11 +888,11 @@ are always available. They are listed here in alphabetical order. Open *file* and return a corresponding :term:`file object`. If the file cannot be opened, an :exc:`OSError` is raised. - *file* is either a string or bytes object giving the pathname (absolute or - relative to the current working directory) of the file to be opened or - an integer file descriptor of the file to be wrapped. (If a file descriptor - is given, it is closed when the returned I/O object is closed, unless - *closefd* is set to ``False``.) + *file* is a :term:`path-like object` giving the pathname (absolute or + relative to the current working directory) of the file to be opened or an + integer file descriptor of the file to be wrapped. (If a file descriptor is + given, it is closed when the returned I/O object is closed, unless *closefd* + is set to ``False``.) *mode* is an optional string that specifies the mode in which the file is opened. It defaults to ``'r'`` which means open for reading in text mode. @@ -1055,27 +1065,38 @@ are always available. They are listed here in alphabetical order. (where :func:`open` is declared), :mod:`os`, :mod:`os.path`, :mod:`tempfile`, and :mod:`shutil`. - .. versionchanged:: 3.3 - The *opener* parameter was added. - The ``'x'`` mode was added. - :exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`. - :exc:`FileExistsError` is now raised if the file opened in exclusive - creation mode (``'x'``) already exists. + .. versionchanged:: + 3.3 - .. versionchanged:: 3.4 - The file is now non-inheritable. + * The *opener* parameter was added. + * The ``'x'`` mode was added. + * :exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`. + * :exc:`FileExistsError` is now raised if the file opened in exclusive + * creation mode (``'x'``) already exists. + + .. versionchanged:: + 3.4 + + * The file is now non-inheritable. .. deprecated-removed:: 3.4 4.0 The ``'U'`` mode. - .. versionchanged:: 3.5 - If the system call is interrupted and the signal handler does not raise an - exception, the function now retries the system call instead of raising an - :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. versionchanged:: + 3.5 - .. versionchanged:: 3.5 - The ``'namereplace'`` error handler was added. + * If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + * The ``'namereplace'`` error handler was added. + + .. versionchanged:: + 3.6 + + * Support added to accept objects implementing :class:`os.PathLike`. + * On Windows, opening a console buffer may return a subclass of + :class:`io.RawIOBase` other than :class:`io.FileIO`. .. function:: ord(c) @@ -1460,6 +1481,9 @@ are always available. They are listed here in alphabetical order. See also :ref:`bltin-type-objects`. + .. versionchanged:: 3.6 + Subclasses of :class:`type` which don't override ``type.__new__`` may no + longer use the one-argument form to get the type of an object. .. function:: vars([object]) diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index 328eef30f1..a8a5a500cb 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -14,7 +14,7 @@ The :mod:`glob` module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell, although results are returned in arbitrary order. No tilde expansion is done, but ``*``, ``?``, and character ranges expressed with ``[]`` will be correctly matched. This is done by using -the :func:`os.listdir` and :func:`fnmatch.fnmatch` functions in concert, and +the :func:`os.scandir` and :func:`fnmatch.fnmatch` functions in concert, and not by actually invoking a subshell. Note that unlike :func:`fnmatch.fnmatch`, :mod:`glob` treats filenames beginning with a dot (``.``) as special cases. (For tilde and shell variable expansion, use :func:`os.path.expanduser` and diff --git a/Doc/library/grp.rst b/Doc/library/grp.rst index a30e6229db..74de3f9520 100644 --- a/Doc/library/grp.rst +++ b/Doc/library/grp.rst @@ -43,6 +43,9 @@ It defines the following items: Return the group database entry for the given numeric group ID. :exc:`KeyError` is raised if the entry asked for cannot be found. + .. deprecated:: 3.6 + Since Python 3.6 the support of non-integer arguments like floats or + strings in :func:`getgrgid` is deprecated. .. function:: getgrnam(name) diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index 792d57a8a3..9c6b722371 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -56,6 +56,8 @@ The module defines the following items: .. versionchanged:: 3.4 Added support for the ``'x'``, ``'xb'`` and ``'xt'`` modes. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. .. class:: GzipFile(filename=None, mode=None, compresslevel=9, fileobj=None, mtime=None) @@ -151,6 +153,9 @@ The module defines the following items: The :meth:`~io.BufferedIOBase.read` method now accepts an argument of ``None``. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: compress(data, compresslevel=9) diff --git a/Doc/library/hashlib-blake2-tree.png b/Doc/library/hashlib-blake2-tree.png Binary files differnew file mode 100644 index 0000000000..010dcbafe7 --- /dev/null +++ b/Doc/library/hashlib-blake2-tree.png diff --git a/Doc/library/hashlib-blake2.rst b/Doc/library/hashlib-blake2.rst new file mode 100644 index 0000000000..eb5e4414a1 --- /dev/null +++ b/Doc/library/hashlib-blake2.rst @@ -0,0 +1,444 @@ +.. _hashlib-blake2: + +:mod:`hashlib` --- BLAKE2 hash functions +======================================== + +.. module:: hashlib + :synopsis: BLAKE2 hash function for Python +.. sectionauthor:: Dmitry Chestnykh + +.. index:: + single: blake2b, blake2s + +BLAKE2_ is a cryptographic hash function defined in RFC-7693_ that comes in two +flavors: + +* **BLAKE2b**, optimized for 64-bit platforms and produces digests of any size + between 1 and 64 bytes, + +* **BLAKE2s**, optimized for 8- to 32-bit platforms and produces digests of any + size between 1 and 32 bytes. + +BLAKE2 supports **keyed mode** (a faster and simpler replacement for HMAC_), +**salted hashing**, **personalization**, and **tree hashing**. + +Hash objects from this module follow the API of standard library's +:mod:`hashlib` objects. + + +Module +====== + +Creating hash objects +--------------------- + +New hash objects are created by calling constructor functions: + + +.. function:: blake2b(data=b'', digest_size=64, key=b'', salt=b'', \ + person=b'', fanout=1, depth=1, leaf_size=0, node_offset=0, \ + node_depth=0, inner_size=0, last_node=False) + +.. function:: blake2s(data=b'', digest_size=32, key=b'', salt=b'', \ + person=b'', fanout=1, depth=1, leaf_size=0, node_offset=0, \ + node_depth=0, inner_size=0, last_node=False) + + +These functions return the corresponding hash objects for calculating +BLAKE2b or BLAKE2s. They optionally take these general parameters: + +* *data*: initial chunk of data to hash, which must be interpretable as buffer + of bytes. + +* *digest_size*: size of output digest in bytes. + +* *key*: key for keyed hashing (up to 64 bytes for BLAKE2b, up to 32 bytes for + BLAKE2s). + +* *salt*: salt for randomized hashing (up to 16 bytes for BLAKE2b, up to 8 + bytes for BLAKE2s). + +* *person*: personalization string (up to 16 bytes for BLAKE2b, up to 8 bytes + for BLAKE2s). + +The following table shows limits for general parameters (in bytes): + +======= =========== ======== ========= =========== +Hash digest_size len(key) len(salt) len(person) +======= =========== ======== ========= =========== +BLAKE2b 64 64 16 16 +BLAKE2s 32 32 8 8 +======= =========== ======== ========= =========== + +.. note:: + + BLAKE2 specification defines constant lengths for salt and personalization + parameters, however, for convenience, this implementation accepts byte + strings of any size up to the specified length. If the length of the + parameter is less than specified, it is padded with zeros, thus, for + example, ``b'salt'`` and ``b'salt\x00'`` is the same value. (This is not + the case for *key*.) + +These sizes are available as module `constants`_ described below. + +Constructor functions also accept the following tree hashing parameters: + +* *fanout*: fanout (0 to 255, 0 if unlimited, 1 in sequential mode). + +* *depth*: maximal depth of tree (1 to 255, 255 if unlimited, 1 in + sequential mode). + +* *leaf_size*: maximal byte length of leaf (0 to 2**32-1, 0 if unlimited or in + sequential mode). + +* *node_offset*: node offset (0 to 2**64-1 for BLAKE2b, 0 to 2**48-1 for + BLAKE2s, 0 for the first, leftmost, leaf, or in sequential mode). + +* *node_depth*: node depth (0 to 255, 0 for leaves, or in sequential mode). + +* *inner_size*: inner digest size (0 to 64 for BLAKE2b, 0 to 32 for + BLAKE2s, 0 in sequential mode). + +* *last_node*: boolean indicating whether the processed node is the last + one (`False` for sequential mode). + +.. figure:: hashlib-blake2-tree.png + :alt: Explanation of tree mode parameters. + +See section 2.10 in `BLAKE2 specification +<https://blake2.net/blake2_20130129.pdf>`_ for comprehensive review of tree +hashing. + + +Constants +--------- + +.. data:: blake2b.SALT_SIZE +.. data:: blake2s.SALT_SIZE + +Salt length (maximum length accepted by constructors). + + +.. data:: blake2b.PERSON_SIZE +.. data:: blake2s.PERSON_SIZE + +Personalization string length (maximum length accepted by constructors). + + +.. data:: blake2b.MAX_KEY_SIZE +.. data:: blake2s.MAX_KEY_SIZE + +Maximum key size. + + +.. data:: blake2b.MAX_DIGEST_SIZE +.. data:: blake2s.MAX_DIGEST_SIZE + +Maximum digest size that the hash function can output. + + +Examples +======== + +Simple hashing +-------------- + +To calculate hash of some data, you should first construct a hash object by +calling the appropriate constructor function (:func:`blake2b` or +:func:`blake2s`), then update it with the data by calling :meth:`update` on the +object, and, finally, get the digest out of the object by calling +:meth:`digest` (or :meth:`hexdigest` for hex-encoded string). + + >>> from hashlib import blake2b + >>> h = blake2b() + >>> h.update(b'Hello world') + >>> h.hexdigest() + '6ff843ba685842aa82031d3f53c48b66326df7639a63d128974c5c14f31a0f33343a8c65551134ed1ae0f2b0dd2bb495dc81039e3eeb0aa1bb0388bbeac29183' + + +As a shortcut, you can pass the first chunk of data to update directly to the +constructor as the first argument (or as *data* keyword argument): + + >>> from hashlib import blake2b + >>> blake2b(b'Hello world').hexdigest() + '6ff843ba685842aa82031d3f53c48b66326df7639a63d128974c5c14f31a0f33343a8c65551134ed1ae0f2b0dd2bb495dc81039e3eeb0aa1bb0388bbeac29183' + +You can call :meth:`hash.update` as many times as you need to iteratively +update the hash: + + >>> from hashlib import blake2b + >>> items = [b'Hello', b' ', b'world'] + >>> h = blake2b() + >>> for item in items: + ... h.update(item) + >>> h.hexdigest() + '6ff843ba685842aa82031d3f53c48b66326df7639a63d128974c5c14f31a0f33343a8c65551134ed1ae0f2b0dd2bb495dc81039e3eeb0aa1bb0388bbeac29183' + + +Using different digest sizes +---------------------------- + +BLAKE2 has configurable size of digests up to 64 bytes for BLAKE2b and up to 32 +bytes for BLAKE2s. For example, to replace SHA-1 with BLAKE2b without changing +the size of output, we can tell BLAKE2b to produce 20-byte digests: + + >>> from hashlib import blake2b + >>> h = blake2b(digest_size=20) + >>> h.update(b'Replacing SHA1 with the more secure function') + >>> h.hexdigest() + 'd24f26cf8de66472d58d4e1b1774b4c9158b1f4c' + >>> h.digest_size + 20 + >>> len(h.digest()) + 20 + +Hash objects with different digest sizes have completely different outputs +(shorter hashes are *not* prefixes of longer hashes); BLAKE2b and BLAKE2s +produce different outputs even if the output length is the same: + + >>> from hashlib import blake2b, blake2s + >>> blake2b(digest_size=10).hexdigest() + '6fa1d8fcfd719046d762' + >>> blake2b(digest_size=11).hexdigest() + 'eb6ec15daf9546254f0809' + >>> blake2s(digest_size=10).hexdigest() + '1bf21a98c78a1c376ae9' + >>> blake2s(digest_size=11).hexdigest() + '567004bf96e4a25773ebf4' + + +Keyed hashing +------------- + +Keyed hashing can be used for authentication as a faster and simpler +replacement for `Hash-based message authentication code +<http://en.wikipedia.org/wiki/Hash-based_message_authentication_code>`_ (HMAC). +BLAKE2 can be securely used in prefix-MAC mode thanks to the +indifferentiability property inherited from BLAKE. + +This example shows how to get a (hex-encoded) 128-bit authentication code for +message ``b'message data'`` with key ``b'pseudorandom key'``:: + + >>> from hashlib import blake2b + >>> h = blake2b(key=b'pseudorandom key', digest_size=16) + >>> h.update(b'message data') + >>> h.hexdigest() + '3d363ff7401e02026f4a4687d4863ced' + + +As a practical example, a web application can symmetrically sign cookies sent +to users and later verify them to make sure they weren't tampered with:: + + >>> from hashlib import blake2b + >>> from hmac import compare_digest + >>> + >>> SECRET_KEY = b'pseudorandomly generated server secret key' + >>> AUTH_SIZE = 16 + >>> + >>> def sign(cookie): + ... h = blake2b(data=cookie, digest_size=AUTH_SIZE, key=SECRET_KEY) + ... return h.hexdigest() + >>> + >>> cookie = b'user:vatrogasac' + >>> sig = sign(cookie) + >>> print("{0},{1}".format(cookie.decode('utf-8'), sig)) + user:vatrogasac,349cf904533767ed2d755279a8df84d0 + >>> compare_digest(cookie, sig) + True + >>> compare_digest(b'user:policajac', sig) + False + >>> compare_digesty(cookie, '0102030405060708090a0b0c0d0e0f00') + False + +Even though there's a native keyed hashing mode, BLAKE2 can, of course, be used +in HMAC construction with :mod:`hmac` module:: + + >>> import hmac, hashlib + >>> m = hmac.new(b'secret key', digestmod=hashlib.blake2s) + >>> m.update(b'message') + >>> m.hexdigest() + 'e3c8102868d28b5ff85fc35dda07329970d1a01e273c37481326fe0c861c8142' + + +Randomized hashing +------------------ + +By setting *salt* parameter users can introduce randomization to the hash +function. Randomized hashing is useful for protecting against collision attacks +on the hash function used in digital signatures. + + Randomized hashing is designed for situations where one party, the message + preparer, generates all or part of a message to be signed by a second + party, the message signer. If the message preparer is able to find + cryptographic hash function collisions (i.e., two messages producing the + same hash value), then she might prepare meaningful versions of the message + that would produce the same hash value and digital signature, but with + different results (e.g., transferring $1,000,000 to an account, rather than + $10). Cryptographic hash functions have been designed with collision + resistance as a major goal, but the current concentration on attacking + cryptographic hash functions may result in a given cryptographic hash + function providing less collision resistance than expected. Randomized + hashing offers the signer additional protection by reducing the likelihood + that a preparer can generate two or more messages that ultimately yield the + same hash value during the digital signature generation process – even if + it is practical to find collisions for the hash function. However, the use + of randomized hashing may reduce the amount of security provided by a + digital signature when all portions of the message are prepared + by the signer. + + (`NIST SP-800-106 "Randomized Hashing for Digital Signatures" + <http://csrc.nist.gov/publications/nistpubs/800-106/NIST-SP-800-106.pdf>`_) + +In BLAKE2 the salt is processed as a one-time input to the hash function during +initialization, rather than as an input to each compression function. + +.. warning:: + + *Salted hashing* (or just hashing) with BLAKE2 or any other general-purpose + cryptographic hash function, such as SHA-256, is not suitable for hashing + passwords. See `BLAKE2 FAQ <https://blake2.net/#qa>`_ for more + information. +.. + + >>> import os + >>> from hashlib import blake2b + >>> msg = b'some message' + >>> # Calculate the first hash with a random salt. + >>> salt1 = os.urandom(blake2b.SALT_SIZE) + >>> h1 = blake2b(salt=salt1) + >>> h1.update(msg) + >>> # Calculate the second hash with a different random salt. + >>> salt2 = os.urandom(blake2b.SALT_SIZE) + >>> h2 = blake2b(salt=salt2) + >>> h2.update(msg) + >>> # The digests are different. + >>> h1.digest() != h2.digest() + True + + +Personalization +--------------- + +Sometimes it is useful to force hash function to produce different digests for +the same input for different purposes. Quoting the authors of the Skein hash +function: + + We recommend that all application designers seriously consider doing this; + we have seen many protocols where a hash that is computed in one part of + the protocol can be used in an entirely different part because two hash + computations were done on similar or related data, and the attacker can + force the application to make the hash inputs the same. Personalizing each + hash function used in the protocol summarily stops this type of attack. + + (`The Skein Hash Function Family + <http://www.skein-hash.info/sites/default/files/skein1.3.pdf>`_, + p. 21) + +BLAKE2 can be personalized by passing bytes to the *person* argument:: + + >>> from hashlib import blake2b + >>> FILES_HASH_PERSON = b'MyApp Files Hash' + >>> BLOCK_HASH_PERSON = b'MyApp Block Hash' + >>> h = blake2b(digest_size=32, person=FILES_HASH_PERSON) + >>> h.update(b'the same content') + >>> h.hexdigest() + '20d9cd024d4fb086aae819a1432dd2466de12947831b75c5a30cf2676095d3b4' + >>> h = blake2b(digest_size=32, person=BLOCK_HASH_PERSON) + >>> h.update(b'the same content') + >>> h.hexdigest() + 'cf68fb5761b9c44e7878bfb2c4c9aea52264a80b75005e65619778de59f383a3' + +Personalization together with the keyed mode can also be used to derive different +keys from a single one. + + >>> from hashlib import blake2s + >>> from base64 import b64decode, b64encode + >>> orig_key = b64decode(b'Rm5EPJai72qcK3RGBpW3vPNfZy5OZothY+kHY6h21KM=') + >>> enc_key = blake2s(key=orig_key, person=b'kEncrypt').digest() + >>> mac_key = blake2s(key=orig_key, person=b'kMAC').digest() + >>> print(b64encode(enc_key).decode('utf-8')) + rbPb15S/Z9t+agffno5wuhB77VbRi6F9Iv2qIxU7WHw= + >>> print(b64encode(mac_key).decode('utf-8')) + G9GtHFE1YluXY1zWPlYk1e/nWfu0WSEb0KRcjhDeP/o= + +Tree mode +--------- + +Here's an example of hashing a minimal tree with two leaf nodes:: + + 10 + / \ + 00 01 + +This example uses 64-byte internal digests, and returns the 32-byte final +digest:: + + >>> from hashlib import blake2b + >>> + >>> FANOUT = 2 + >>> DEPTH = 2 + >>> LEAF_SIZE = 4096 + >>> INNER_SIZE = 64 + >>> + >>> buf = bytearray(6000) + >>> + >>> # Left leaf + ... h00 = blake2b(buf[0:LEAF_SIZE], fanout=FANOUT, depth=DEPTH, + ... leaf_size=LEAF_SIZE, inner_size=INNER_SIZE, + ... node_offset=0, node_depth=0, last_node=False) + >>> # Right leaf + ... h01 = blake2b(buf[LEAF_SIZE:], fanout=FANOUT, depth=DEPTH, + ... leaf_size=LEAF_SIZE, inner_size=INNER_SIZE, + ... node_offset=1, node_depth=0, last_node=True) + >>> # Root node + ... h10 = blake2b(digest_size=32, fanout=FANOUT, depth=DEPTH, + ... leaf_size=LEAF_SIZE, inner_size=INNER_SIZE, + ... node_offset=0, node_depth=1, last_node=True) + >>> h10.update(h00.digest()) + >>> h10.update(h01.digest()) + >>> h10.hexdigest() + '3ad2a9b37c6070e374c7a8c508fe20ca86b6ed54e286e93a0318e95e881db5aa' + +Credits +======= + +BLAKE2_ was designed by *Jean-Philippe Aumasson*, *Samuel Neves*, *Zooko +Wilcox-O'Hearn*, and *Christian Winnerlein* based on SHA-3_ finalist BLAKE_ +created by *Jean-Philippe Aumasson*, *Luca Henzen*, *Willi Meier*, and +*Raphael C.-W. Phan*. + +It uses core algorithm from ChaCha_ cipher designed by *Daniel J. Bernstein*. + +The stdlib implementation is based on pyblake2_ module. It was written by +*Dmitry Chestnykh* based on C implementation written by *Samuel Neves*. The +documentation was copied from pyblake2_ and written by *Dmitry Chestnykh*. + +The C code was partly rewritten for Python by *Christian Heimes*. + +The following public domain dedication applies for both C hash function +implementation, extension code, and this documentation: + + To the extent possible under law, the author(s) have dedicated all copyright + and related and neighboring rights to this software to the public domain + worldwide. This software is distributed without any warranty. + + You should have received a copy of the CC0 Public Domain Dedication along + with this software. If not, see + http://creativecommons.org/publicdomain/zero/1.0/. + +The following people have helped with development or contributed their changes +to the project and the public domain according to the Creative Commons Public +Domain Dedication 1.0 Universal: + +* *Alexandr Sokolovskiy* + +.. seealso:: Official BLAKE2 website: https://blake2.net + +.. _RFC-7693: https://tools.ietf.org/html/rfc7693 +.. _BLAKE2: https://blake2.net +.. _HMAC: https://en.wikipedia.org/wiki/Hash-based_message_authentication_code +.. _BLAKE: https://131002.net/blake/ +.. _SHA-3: https://en.wikipedia.org/wiki/NIST_hash_function_competition +.. _ChaCha: https://cr.yp.to/chacha.html +.. _pyblake2: https://pythonhosted.org/pyblake2/ + diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index a2e96caf76..2cb3c78f09 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -44,8 +44,8 @@ Hash algorithms --------------- There is one constructor method named for each type of :dfn:`hash`. All return -a hash object with the same simple interface. For example: use :func:`sha1` to -create a SHA1 hash object. You can now feed this object with :term:`bytes-like +a hash object with the same simple interface. For example: use :func:`sha256` to +create a SHA-256 hash object. You can now feed this object with :term:`bytes-like objects <bytes-like object>` (normally :class:`bytes`) using the :meth:`update` method. At any point you can ask it for the :dfn:`digest` of the concatenation of the data fed to it so far using the :meth:`digest` or @@ -64,21 +64,33 @@ concatenation of the data fed to it so far using the :meth:`digest` or .. index:: single: OpenSSL; (use in module hashlib) Constructors for hash algorithms that are always present in this module are -:func:`md5`, :func:`sha1`, :func:`sha224`, :func:`sha256`, :func:`sha384`, -and :func:`sha512`. Additional algorithms may also be available depending upon -the OpenSSL library that Python uses on your platform. +:func:`sha1`, :func:`sha224`, :func:`sha256`, :func:`sha384`, +:func:`sha512`, :func:`blake2b`, and :func:`blake2s`. +:func:`md5` is normally available as well, though it +may be missing if you are using a rare "FIPS compliant" build of Python. +Additional algorithms may also be available depending upon the OpenSSL +library that Python uses on your platform. On most platforms the +:func:`sha3_224`, :func:`sha3_256`, :func:`sha3_384`, :func:`sha3_512`, +:func:`shake_128`, :func:`shake_256` are also available. + +.. versionadded:: 3.6 + SHA3 (Keccak) and SHAKE constructors :func:`sha3_224`, :func:`sha3_256`, + :func:`sha3_384`, :func:`sha3_512`, :func:`shake_128`, :func:`shake_256`. + +.. versionadded:: 3.6 + :func:`blake2b` and :func:`blake2s` were added. For example, to obtain the digest of the byte string ``b'Nobody inspects the spammish repetition'``:: >>> import hashlib - >>> m = hashlib.md5() + >>> m = hashlib.sha256() >>> m.update(b"Nobody inspects") >>> m.update(b" the spammish repetition") >>> m.digest() - b'\xbbd\x9c\x83\xdd\x1e\xa5\xc9\xd9\xde\xc9\xa1\x8d\xf0\xff\xe9' + b'\x03\x1e\xdd}Ae\x15\x93\xc5\xfe\\\x00o\xa5u+7\xfd\xdf\xf7\xbcN\x84:\xa6\xaf\x0c\x95\x0fK\x94\x06' >>> m.digest_size - 16 + 32 >>> m.block_size 64 @@ -107,7 +119,9 @@ Hashlib provides the following constant attributes: .. data:: algorithms_guaranteed A set containing the names of the hash algorithms guaranteed to be supported - by this module on all platforms. + by this module on all platforms. Note that 'md5' is in this list despite + some upstream vendors offering an odd "FIPS compliant" Python build that + excludes it. .. versionadded:: 3.2 @@ -181,6 +195,28 @@ A hash object has the following methods: compute the digests of data sharing a common initial substring. +SHAKE variable length digests +----------------------------- + +The :func:`shake_128` and :func:`shake_256` algorithms provide variable +length digests with length_in_bits//2 up to 128 or 256 bits of security. +As such, their digest methods require a length. Maximum length is not limited +by the SHAKE algorithm. + +.. method:: shake.digest(length) + + Return the digest of the data passed to the :meth:`update` method so far. + This is a bytes object of size ``length`` which may contain bytes in + the whole range from 0 to 255. + + +.. method:: shake.hexdigest(length) + + Like :meth:`digest` except the digest is returned as a string object of + double length, containing only hexadecimal digits. This may be used to + exchange the value safely in email or other non-binary environments. + + Key derivation -------------- @@ -221,6 +257,29 @@ include a `salt <https://en.wikipedia.org/wiki/Salt_%28cryptography%29>`_. Python implementation uses an inline version of :mod:`hmac`. It is about three times slower and doesn't release the GIL. +.. function:: scrypt(password, *, salt, n, r, p, maxmem=0, dklen=64) + + The function provides scrypt password-based key derivation function as + defined in :rfc:`7914`. + + *password* and *salt* must be bytes-like objects. Applications and + libraries should limit *password* to a sensible length (e.g. 1024). *salt* + should be about 16 or more bytes from a proper source, e.g. :func:`os.urandom`. + + *n* is the CPU/Memory cost factor, *r* the block size, *p* parallelization + factor and *maxmem* limits memory (OpenSSL 1.1.0 defaults to 32 MB). + *dklen* is the length of the derived key. + + Availability: OpenSSL 1.1+ + + .. versionadded:: 3.6 + + +BLAKE2 +------ + +BLAKE2 takes additional arguments, see :ref:`hashlib-blake2`. + .. seealso:: @@ -230,6 +289,8 @@ include a `salt <https://en.wikipedia.org/wiki/Salt_%28cryptography%29>`_. Module :mod:`base64` Another way to encode binary hashes for non-binary environments. + See :ref:`hashlib-blake2`. + http://csrc.nist.gov/publications/fips/fips180-2/fips180-2.pdf The FIPS 180-2 publication on Secure Hash Algorithms. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index d863fe99f1..17f289d84f 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -69,13 +69,6 @@ The module provides the following classes: must be a :class:`ssl.SSLContext` instance describing the various SSL options. - *key_file* and *cert_file* are deprecated, please use - :meth:`ssl.SSLContext.load_cert_chain` instead, or let - :func:`ssl.create_default_context` select the system's trusted CA - certificates for you. The *check_hostname* parameter is also deprecated; the - :attr:`ssl.SSLContext.check_hostname` attribute of *context* should be used - instead. - Please read :ref:`ssl-security` for more information on best practices. .. versionchanged:: 3.2 @@ -95,6 +88,17 @@ The module provides the following classes: :func:`ssl._create_unverified_context` can be passed to the *context* parameter. + .. deprecated:: 3.6 + + *key_file* and *cert_file* are deprecated in favor of *context*. + Please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let + :func:`ssl.create_default_context` select the system's trusted CA + certificates for you. + + The *check_hostname* parameter is also deprecated; the + :attr:`ssl.SSLContext.check_hostname` attribute of *context* should + be used instead. + .. class:: HTTPResponse(sock, debuglevel=0, method=None, url=None) @@ -217,39 +221,61 @@ HTTPConnection Objects :class:`HTTPConnection` instances have the following methods: -.. method:: HTTPConnection.request(method, url, body=None, headers={}) +.. method:: HTTPConnection.request(method, url, body=None, headers={}, *, \ + encode_chunked=False) This will send a request to the server using the HTTP request method *method* and the selector *url*. If *body* is specified, the specified data is sent after the headers are - finished. It may be a string, a :term:`bytes-like object`, an open - :term:`file object`, or an iterable of :term:`bytes-like object`\s. If - *body* is a string, it is encoded as ISO-8859-1, the default for HTTP. If - it is a bytes-like object the bytes are sent as is. If it is a :term:`file - object`, the contents of the file is sent; this file object should support - at least the ``read()`` method. If the file object has a ``mode`` - attribute, the data returned by the ``read()`` method will be encoded as - ISO-8859-1 unless the ``mode`` attribute contains the substring ``b``, - otherwise the data returned by ``read()`` is sent as is. If *body* is an - iterable, the elements of the iterable are sent as is until the iterable is - exhausted. - - The *headers* argument should be a mapping of extra HTTP - headers to send with the request. - - If *headers* does not contain a Content-Length item, one is added - automatically if possible. If *body* is ``None``, the Content-Length header - is set to ``0`` for methods that expect a body (``PUT``, ``POST``, and - ``PATCH``). If *body* is a string or bytes object, the Content-Length - header is set to its length. If *body* is a :term:`file object` and it - works to call :func:`~os.fstat` on the result of its ``fileno()`` method, - then the Content-Length header is set to the ``st_size`` reported by the - ``fstat`` call. Otherwise no Content-Length header is added. + finished. It may be a :class:`str`, a :term:`bytes-like object`, an + open :term:`file object`, or an iterable of :class:`bytes`. If *body* + is a string, it is encoded as ISO-8859-1, the default for HTTP. If it + is a bytes-like object, the bytes are sent as is. If it is a :term:`file + object`, the contents of the file is sent; this file object should + support at least the ``read()`` method. If the file object is an + instance of :class:`io.TextIOBase`, the data returned by the ``read()`` + method will be encoded as ISO-8859-1, otherwise the data returned by + ``read()`` is sent as is. If *body* is an iterable, the elements of the + iterable are sent as is until the iterable is exhausted. + + The *headers* argument should be a mapping of extra HTTP headers to send + with the request. + + If *headers* contains neither Content-Length nor Transfer-Encoding, + but there is a request body, one of those + header fields will be added automatically. If + *body* is ``None``, the Content-Length header is set to ``0`` for + methods that expect a body (``PUT``, ``POST``, and ``PATCH``). If + *body* is a string or a bytes-like object that is not also a + :term:`file <file object>`, the Content-Length header is + set to its length. Any other type of *body* (files + and iterables in general) will be chunk-encoded, and the + Transfer-Encoding header will automatically be set instead of + Content-Length. + + The *encode_chunked* argument is only relevant if Transfer-Encoding is + specified in *headers*. If *encode_chunked* is ``False``, the + HTTPConnection object assumes that all encoding is handled by the + calling code. If it is ``True``, the body will be chunk-encoded. + + .. note:: + Chunked transfer encoding has been added to the HTTP protocol + version 1.1. Unless the HTTP server is known to handle HTTP 1.1, + the caller must either specify the Content-Length, or must pass a + :class:`str` or bytes-like object that is not also a file as the + body representation. .. versionadded:: 3.2 *body* can now be an iterable. + .. versionchanged:: 3.6 + If neither Content-Length nor Transfer-Encoding are set in + *headers*, file and iterable *body* objects are now chunk-encoded. + The *encode_chunked* argument was added. + No attempt is made to determine the Content-Length for file + objects. + .. method:: HTTPConnection.getresponse() Should be called after a request is sent to get the response from the server. @@ -335,13 +361,32 @@ also send your request step by step, by using the four functions below. an argument. -.. method:: HTTPConnection.endheaders(message_body=None) +.. method:: HTTPConnection.endheaders(message_body=None, *, encode_chunked=False) Send a blank line to the server, signalling the end of the headers. The optional *message_body* argument can be used to pass a message body - associated with the request. The message body will be sent in the same - packet as the message headers if it is string, otherwise it is sent in a - separate packet. + associated with the request. + + If *encode_chunked* is ``True``, the result of each iteration of + *message_body* will be chunk-encoded as specified in :rfc:`7230`, + Section 3.3.1. How the data is encoded is dependent on the type of + *message_body*. If *message_body* implements the :ref:`buffer interface + <bufferobjects>` the encoding will result in a single chunk. + If *message_body* is a :class:`collections.Iterable`, each iteration + of *message_body* will result in a chunk. If *message_body* is a + :term:`file object`, each call to ``.read()`` will result in a chunk. + The method automatically signals the end of the chunk-encoded data + immediately after *message_body*. + + .. note:: Due to the chunked encoding specification, empty chunks + yielded by an iterator body will be ignored by the chunk-encoder. + This is to avoid premature termination of the read of the request by + the target server due to malformed encoding. + + .. versionadded:: 3.6 + Chunked encoding support. The *encode_chunked* parameter was + added. + .. method:: HTTPConnection.send(data) diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index ae7fb970de..fb5c1df611 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -98,8 +98,8 @@ of which this module provides three different variants: .. attribute:: rfile - Contains an input stream, positioned at the start of the optional input - data. + An :class:`io.BufferedIOBase` input stream, ready to read from + the start of the optional input data. .. attribute:: wfile @@ -107,6 +107,9 @@ of which this module provides three different variants: client. Proper adherence to the HTTP protocol must be used when writing to this stream. + .. versionchanged:: 3.6 + This is an :class:`io.BufferedIOBase` stream. + :class:`BaseHTTPRequestHandler` has the following attributes: .. attribute:: server_version @@ -369,10 +372,9 @@ the current directory:: Handler = http.server.SimpleHTTPRequestHandler - httpd = socketserver.TCPServer(("", PORT), Handler) - - print("serving at port", PORT) - httpd.serve_forever() + with socketserver.TCPServer(("", PORT), Handler) as httpd: + print("serving at port", PORT) + httpd.serve_forever() .. _http-server-cli: diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index 771ca43783..1632eb7aed 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -103,6 +103,14 @@ There's also a subclass for secure connections: :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see :data:`ssl.HAS_SNI`). + .. deprecated:: 3.6 + + *keyfile* and *certfile* are deprecated in favor of *ssl_context*. + Please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let + :func:`ssl.create_default_context` select the system's trusted CA + certificates for you. + + The second subclass allows for connections created by a child process: @@ -500,6 +508,17 @@ An :class:`IMAP4` instance has the following methods: M.store(num, '+FLAGS', '\\Deleted') M.expunge() + .. note:: + + Creating flags containing ']' (for example: "[test]") violates + :rfc:`3501` (the IMAP protocol). However, imaplib has historically + allowed creation of such tags, and popular IMAP servers, such as Gmail, + accept and produce such flags. There are non-Python programs which also + create such tags. Although it is an RFC violation and IMAP clients and + servers are supposed to be strict, imaplib nonetheless continues to allow + such tags to be created for backward compatibility reasons, and as of + python 3.6, handles them if they are sent from the server, since this + improves real-world compatibility. .. method:: IMAP4.subscribe(mailbox) diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index f11f6dcf8e..800e919599 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -20,6 +20,9 @@ The :mod:`imghdr` module defines the following function: string describing the image type. If optional *h* is provided, the *filename* is ignored and *h* is assumed to contain the byte stream to test. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + The following image types are recognized, as listed below with the return value from :func:`what`: diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 9828ba6fe2..ccf5f92d16 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -85,7 +85,9 @@ This module provides an interface to the mechanisms used to implement the .. deprecated:: 3.3 Use :func:`importlib.util.find_spec` instead unless Python 3.3 compatibility is required, in which case use - :func:`importlib.find_loader`. + :func:`importlib.find_loader`. For example usage of the former case, + see the :ref:`importlib-examples` section of the :mod:`importlib` + documentation. .. function:: load_module(name, file, pathname, description) @@ -112,9 +114,12 @@ This module provides an interface to the mechanisms used to implement the If previously used in conjunction with :func:`imp.find_module` then consider using :func:`importlib.import_module`, otherwise use the loader returned by the replacement you chose for :func:`imp.find_module`. If you - called :func:`imp.load_module` and related functions directly then use the - classes in :mod:`importlib.machinery`, e.g. - ``importlib.machinery.SourceFileLoader(name, path).load_module()``. + called :func:`imp.load_module` and related functions directly with file + path arguments then use a combination of + :func:`importlib.util.spec_from_file_location` and + :func:`importlib.util.module_from_spec`. See the :ref:`importlib-examples` + section of the :mod:`importlib` documentation for details of the various + approaches. .. function:: new_module(name) @@ -123,7 +128,7 @@ This module provides an interface to the mechanisms used to implement the in ``sys.modules``. .. deprecated:: 3.4 - Use :class:`types.ModuleType` instead. + Use :func:`importlib.util.module_from_spec` instead. .. function:: reload(module) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index cb52f740ee..f63a18dd82 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -267,7 +267,7 @@ ABC hierarchy:: module and *path* will be the value of :attr:`__path__` from the parent package. If a spec cannot be found, ``None`` is returned. When passed in, ``target`` is a module object that the finder may - use to make a more educated about what spec to return. + use to make a more educated guess about what spec to return. .. versionadded:: 3.4 @@ -317,7 +317,7 @@ ABC hierarchy:: within the :term:`path entry` to which it is assigned. If a spec cannot be found, ``None`` is returned. When passed in, ``target`` is a module object that the finder may use to make a more educated - about what spec to return. + guess about what spec to return. .. versionadded:: 3.4 @@ -379,10 +379,14 @@ ABC hierarchy:: An abstract method that executes the module in its own namespace when a module is imported or reloaded. The module should already - be initialized when exec_module() is called. + be initialized when ``exec_module()`` is called. When this method exists, + :meth:`~importlib.abc.Loader.create_module` must be defined. .. versionadded:: 3.4 + .. versionchanged:: 3.6 + :meth:`~importlib.abc.Loader.create_module` must also be defined. + .. method:: load_module(fullname) A legacy method for loading a module. If the module cannot be @@ -936,6 +940,10 @@ find and load modules. Concrete implementation of :meth:`importlib.abc.Loader.load_module` where specifying the name of the module to load is optional. + .. deprecated:: 3.6 + + Use :meth:`importlib.abc.Loader.exec_module` instead. + .. class:: SourcelessFileLoader(fullname, path) @@ -975,6 +983,10 @@ find and load modules. Concrete implementation of :meth:`importlib.abc.Loader.load_module` where specifying the name of the module to load is optional. + .. deprecated:: 3.6 + + Use :meth:`importlib.abc.Loader.exec_module` instead. + .. class:: ExtensionFileLoader(fullname, path) @@ -1137,6 +1149,9 @@ an :term:`importer`. The *optimization* parameter was added and the *debug_override* parameter was deprecated. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: source_from_cache(path) @@ -1150,6 +1165,9 @@ an :term:`importer`. .. versionadded:: 3.4 + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: decode_source(source_bytes) Decode the given bytes representing source code and return it as a string @@ -1192,12 +1210,13 @@ an :term:`importer`. .. function:: module_from_spec(spec) - Create a new module based on **spec** and ``spec.loader.create_module()``. + Create a new module based on **spec** and + :meth:`spec.loader.create_module <importlib.abc.Loader.create_module>`. - If ``spec.loader.create_module()`` does not return ``None``, then any - pre-existing attributes will not be reset. Also, no :exc:`AttributeError` - will be raised if triggered while accessing **spec** or setting an attribute - on the module. + If :meth:`spec.loader.create_module <importlib.abc.Loader.create_module>` + does not return ``None``, then any pre-existing attributes will not be reset. + Also, no :exc:`AttributeError` will be raised if triggered while accessing + **spec** or setting an attribute on the module. This function is preferred over using :class:`types.ModuleType` to create a new module as **spec** is used to set as many import-controlled attributes on @@ -1259,7 +1278,8 @@ an :term:`importer`. .. decorator:: set_package - A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` to set the :attr:`__package__` attribute on the returned module. If :attr:`__package__` + A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` to set the + :attr:`__package__` attribute on the returned module. If :attr:`__package__` is set and has a value other than ``None`` it will not be changed. .. deprecated:: 3.4 @@ -1284,6 +1304,9 @@ an :term:`importer`. .. versionadded:: 3.4 + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. class:: LazyLoader(loader) A class which postpones the execution of the loader of a module until the @@ -1292,13 +1315,12 @@ an :term:`importer`. This class **only** works with loaders that define :meth:`~importlib.abc.Loader.exec_module` as control over what module type is used for the module is required. For those same reasons, the loader's - :meth:`~importlib.abc.Loader.create_module` method will be ignored (i.e., the - loader's method should only return ``None``; this excludes - :class:`BuiltinImporter` and :class:`ExtensionFileLoader`). Finally, - modules which substitute the object placed into :attr:`sys.modules` will - not work as there is no way to properly replace the module references - throughout the interpreter safely; :exc:`ValueError` is raised if such a - substitution is detected. + :meth:`~importlib.abc.Loader.create_module` method must return ``None`` or a + type for which its ``__class__`` attribute can be mutated along with not + using :term:`slots <__slots__>`. Finally, modules which substitute the object + placed into :attr:`sys.modules` will not work as there is no way to properly + replace the module references throughout the interpreter safely; + :exc:`ValueError` is raised if such a substitution is detected. .. note:: For projects where startup time is critical, this class allows for @@ -1309,6 +1331,11 @@ an :term:`importer`. .. versionadded:: 3.5 + .. versionchanged:: 3.6 + Began calling :meth:`~importlib.abc.Loader.create_module`, removing the + compatibility warning for :class:`importlib.machinery.BuiltinImporter` and + :class:`importlib.machinery.ExtensionFileLoader`. + .. classmethod:: factory(loader) A static method which returns a callable that creates a lazy loader. This @@ -1320,3 +1347,139 @@ an :term:`importer`. loader = importlib.machinery.SourceFileLoader lazy_loader = importlib.util.LazyLoader.factory(loader) finder = importlib.machinery.FileFinder(path, (lazy_loader, suffixes)) + +.. _importlib-examples: + +Examples +-------- + +Importing programmatically +'''''''''''''''''''''''''' + +To programmatically import a module, use :func:`importlib.import_module`. +:: + + import importlib + + itertools = importlib.import_module('itertools') + + +Checking if a module can be imported +'''''''''''''''''''''''''''''''''''' + +If you need to find out if a module can be imported without actually doing the +import, then you should use :func:`importlib.util.find_spec`. +:: + + import importlib.util + import sys + + # For illustrative purposes. + name = 'itertools' + + spec = importlib.util.find_spec(name) + if spec is None: + print("can't find the itertools module") + else: + # If you chose to perform the actual import ... + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + # Adding the module to sys.modules is optional. + sys.modules[name] = module + + +Importing a source file directly +'''''''''''''''''''''''''''''''' + +To import a Python source file directly, use the following recipe +(Python 3.4 and newer only):: + + import importlib.util + import sys + + # For illustrative purposes. + import tokenize + file_path = tokenize.__file__ + module_name = tokenize.__name__ + + spec = importlib.util.spec_from_file_location(module_name, file_path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + # Optional; only necessary if you want to be able to import the module + # by name later. + sys.modules[module_name] = module + + +Setting up an importer +'''''''''''''''''''''' + +For deep customizations of import, you typically want to implement an +:term:`importer`. This means managing both the :term:`finder` and :term:`loader` +side of things. For finders there are two flavours to choose from depending on +your needs: a :term:`meta path finder` or a :term:`path entry finder`. The +former is what you would put on :attr:`sys.meta_path` while the latter is what +you create using a :term:`path entry hook` on :attr:`sys.path_hooks` which works +with :attr:`sys.path` entries to potentially create a finder. This example will +show you how to register your own importers so that import will use them (for +creating an importer for yourself, read the documentation for the appropriate +classes defined within this package):: + + import importlib.machinery + import sys + + # For illustrative purposes only. + SpamMetaPathFinder = importlib.machinery.PathFinder + SpamPathEntryFinder = importlib.machinery.FileFinder + loader_details = (importlib.machinery.SourceFileLoader, + importlib.machinery.SOURCE_SUFFIXES) + + # Setting up a meta path finder. + # Make sure to put the finder in the proper location in the list in terms of + # priority. + sys.meta_path.append(SpamMetaPathFinder) + + # Setting up a path entry finder. + # Make sure to put the path hook in the proper location in the list in terms + # of priority. + sys.path_hooks.append(SpamPathEntryFinder.path_hook(loader_details)) + + +Approximating :func:`importlib.import_module` +''''''''''''''''''''''''''''''''''''''''''''' + +Import itself is implemented in Python code, making it possible to +expose most of the import machinery through importlib. The following +helps illustrate the various APIs that importlib exposes by providing an +approximate implementation of +:func:`importlib.import_module` (Python 3.4 and newer for the importlib usage, +Python 3.6 and newer for other parts of the code). +:: + + import importlib.util + import sys + + def import_module(name, package=None): + """An approximate implementation of import.""" + absolute_name = importlib.util.resolve_name(name, package) + try: + return sys.modules[absolute_name] + except KeyError: + pass + + path = None + if '.' in absolute_name: + parent_name, _, child_name = absolute_name.rpartition('.') + parent_module = import_module(parent_name) + path = parent_module.spec.submodule_search_locations + for finder in sys.meta_path: + spec = finder.find_spec(absolute_name, path) + if spec is not None: + break + else: + raise ImportError(f'No module named {absolute_name!r}') + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + sys.modules[absolute_name] = module + if path is not None: + setattr(parent_module, child_name, module) + return module diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 62a3988ae0..fc815a672c 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -235,24 +235,6 @@ attributes: listed in the metaclass' custom :meth:`__dir__`. -.. function:: getmoduleinfo(path) - - Returns a :term:`named tuple` ``ModuleInfo(name, suffix, mode, module_type)`` - of values that describe how Python will interpret the file identified by - *path* if it is a module, or ``None`` if it would not be identified as a - module. In that tuple, *name* is the name of the module without the name of - any enclosing package, *suffix* is the trailing part of the file name (which - may not be a dot-delimited extension), *mode* is the :func:`open` mode that - would be used (``'r'`` or ``'rb'``), and *module_type* is an integer giving - the type of the module. *module_type* will have a value which can be - compared to the constants defined in the :mod:`imp` module; see the - documentation for that module for more information on module types. - - .. deprecated:: 3.3 - You may check the file path's suffix against the supported suffixes - listed in :mod:`importlib.machinery` to infer the same information. - - .. function:: getmodulename(path) Return the name of the module named by the file *path*, without including the @@ -266,8 +248,7 @@ attributes: still return ``None``. .. versionchanged:: 3.3 - This function is now based directly on :mod:`importlib` rather than the - deprecated :func:`getmoduleinfo`. + The function is based directly on :mod:`importlib`. .. function:: ismodule(object) @@ -646,6 +627,16 @@ function. The name of the parameter as a string. The name must be a valid Python identifier. + .. impl-detail:: + + CPython generates implicit parameter names of the form ``.0`` on the + code objects used to implement comprehensions and generator + expressions. + + .. versionchanged:: 3.6 + These parameter names are exposed by this module as names like + ``implicit0``. + .. attribute:: Parameter.default The default value for the parameter. If the parameter has no default @@ -854,8 +845,6 @@ Classes and functions from kwonlyargs to defaults. *annotations* is a dictionary mapping argument names to annotations. - The first four items in the tuple correspond to :func:`getargspec`. - .. versionchanged:: 3.4 This function is now based on :func:`signature`, but still ignores ``__wrapped__`` attributes and includes the already bound first @@ -884,7 +873,7 @@ Classes and functions .. function:: formatargspec(args[, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations[, formatarg, formatvarargs, formatvarkw, formatvalue, formatreturns, formatannotations]]) Format a pretty argument spec from the values returned by - :func:`getargspec` or :func:`getfullargspec`. + :func:`getfullargspec`. The first seven arguments are (``args``, ``varargs``, ``varkw``, ``defaults``, ``kwonlyargs``, ``kwonlydefaults``, ``annotations``). @@ -1264,20 +1253,29 @@ the following flags: .. data:: CO_COROUTINE - The flag is set when the code object is a coroutine function, i.e. - a coroutine object is returned when the code object is executed. See - :pep:`492` for more details. + The flag is set when the code object is a coroutine function. + When the code object is executed it returns a coroutine object. + See :pep:`492` for more details. .. versionadded:: 3.5 .. data:: CO_ITERABLE_COROUTINE - Used to turn generators into generator-based coroutines. Generator - objects with this flag can be used in ``await`` expression, and can - ``yield from`` coroutine objects. See :pep:`492` for more details. + The flag is used to transform generators into generator-based + coroutines. Generator objects with this flag can be used in + ``await`` expression, and can ``yield from`` coroutine objects. + See :pep:`492` for more details. .. versionadded:: 3.5 +.. data:: CO_ASYNC_GENERATOR + + The flag is set when the code object is an asynchronous generator + function. When the code object is executed it returns an + asynchronous generator object. See :pep:`525` for more details. + + .. versionadded:: 3.6 + .. note:: The flags are specific to CPython, and may not be defined in other Python implementations. Furthermore, the flags are an implementation @@ -1286,7 +1284,6 @@ the following flags: for any introspection needs. - .. _inspect-module-cli: Command Line Interface diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index dfc1ddc2d7..b0d0a8c8c8 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -591,7 +591,13 @@ loops that truncate the stream. .. function:: tee(iterable, n=2) - Return *n* independent iterators from a single iterable. Roughly equivalent to:: + Return *n* independent iterators from a single iterable. + + The following Python code helps explain what *tee* does (although the actual + implementation is more complex and uses only a single underlying + :abbr:`FIFO (first-in, first-out)` queue). + + Roughly equivalent to:: def tee(iterable, n=2): it = iter(iterable) diff --git a/Doc/library/json.rst b/Doc/library/json.rst index c1a52a79ea..033e3fc41e 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -126,7 +126,7 @@ See :ref:`json-commandline` for detailed documentation. Basic Usage ----------- -.. function:: dump(obj, fp, skipkeys=False, ensure_ascii=True, \ +.. function:: dump(obj, fp, *, skipkeys=False, ensure_ascii=True, \ check_circular=True, allow_nan=True, cls=None, \ indent=None, separators=None, default=None, \ sort_keys=False, **kw) @@ -187,8 +187,11 @@ Basic Usage :meth:`default` method to serialize additional types), specify it with the *cls* kwarg; otherwise :class:`JSONEncoder` is used. + .. versionchanged:: 3.6 + All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`. -.. function:: dumps(obj, skipkeys=False, ensure_ascii=True, \ + +.. function:: dumps(obj, *, skipkeys=False, ensure_ascii=True, \ check_circular=True, allow_nan=True, cls=None, \ indent=None, separators=None, default=None, \ sort_keys=False, **kw) @@ -212,7 +215,7 @@ Basic Usage the original one. That is, ``loads(dumps(x)) != x`` if x has non-string keys. -.. function:: load(fp, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) +.. function:: load(fp, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) Deserialize *fp* (a ``.read()``-supporting :term:`file-like object` containing a JSON document) to a Python object using this :ref:`conversion @@ -260,10 +263,14 @@ Basic Usage If the data being deserialized is not a valid JSON document, a :exc:`JSONDecodeError` will be raised. -.. function:: loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) + .. versionchanged:: 3.6 + All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`. + +.. function:: loads(s, *, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) - Deserialize *s* (a :class:`str` instance containing a JSON document) to a - Python object using this :ref:`conversion table <json-to-py-table>`. + Deserialize *s* (a :class:`str`, :class:`bytes` or :class:`bytearray` + instance containing a JSON document) to a Python object using this + :ref:`conversion table <json-to-py-table>`. The other arguments have the same meaning as in :func:`load`, except *encoding* which is ignored and deprecated. @@ -274,7 +281,7 @@ Basic Usage Encoders and Decoders --------------------- -.. class:: JSONDecoder(object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None) +.. class:: JSONDecoder(*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None) Simple JSON decoder. @@ -344,6 +351,9 @@ Encoders and Decoders If the data being deserialized is not a valid JSON document, a :exc:`JSONDecodeError` will be raised. + .. versionchanged:: 3.6 + All parameters are now :ref:`keyword-only <keyword-only_parameter>`. + .. method:: decode(s) Return the Python representation of *s* (a :class:`str` instance @@ -362,7 +372,7 @@ Encoders and Decoders extraneous data at the end. -.. class:: JSONEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None) +.. class:: JSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None) Extensible JSON encoder for Python data structures. @@ -443,6 +453,9 @@ Encoders and Decoders the object or raise a :exc:`TypeError`. If not specified, :exc:`TypeError` is raised. + .. versionchanged:: 3.6 + All parameters are now :ref:`keyword-only <keyword-only_parameter>`. + .. method:: default(o) diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index 0c02d3d552..6be327908a 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -84,6 +84,9 @@ sends logging output to a disk file. It inherits the output functionality from with that encoding. If *delay* is true, then file opening is deferred until the first call to :meth:`emit`. By default, the file grows indefinitely. + .. versionchanged:: 3.6 + As well as string values, :class:`~pathlib.Path` objects are also accepted + for the *filename* argument. .. method:: close() @@ -160,12 +163,23 @@ for this value. with that encoding. If *delay* is true, then file opening is deferred until the first call to :meth:`emit`. By default, the file grows indefinitely. + .. versionchanged:: 3.6 + As well as string values, :class:`~pathlib.Path` objects are also accepted + for the *filename* argument. + + .. method:: reopenIfNeeded() + + Checks to see if the file has changed. If it has, the existing stream is + flushed and closed and the file opened again, typically as a precursor to + outputting the record to the file. + + .. versionadded:: 3.6 + .. method:: emit(record) - Outputs the record to the file, but first checks to see if the file has - changed. If it has, the existing stream is flushed and closed and the - file opened again, before outputting the record to the file. + Outputs the record to the file, but first calls :meth:`reopenIfNeeded` to + reopen the file if it has changed. .. _base-rotating-handler: @@ -279,6 +293,9 @@ module, supports rotation of disk log files. :file:`app.log.2`, etc. exist, then they are renamed to :file:`app.log.2`, :file:`app.log.3` etc. respectively. + .. versionchanged:: 3.6 + As well as string values, :class:`~pathlib.Path` objects are also accepted + for the *filename* argument. .. method:: doRollover() @@ -310,21 +327,24 @@ timed intervals. You can use the *when* to specify the type of *interval*. The list of possible values is below. Note that they are not case sensitive. - +----------------+-----------------------+ - | Value | Type of interval | - +================+=======================+ - | ``'S'`` | Seconds | - +----------------+-----------------------+ - | ``'M'`` | Minutes | - +----------------+-----------------------+ - | ``'H'`` | Hours | - +----------------+-----------------------+ - | ``'D'`` | Days | - +----------------+-----------------------+ - | ``'W0'-'W6'`` | Weekday (0=Monday) | - +----------------+-----------------------+ - | ``'midnight'`` | Roll over at midnight | - +----------------+-----------------------+ + +----------------+----------------------------+-------------------------+ + | Value | Type of interval | If/how *atTime* is used | + +================+============================+=========================+ + | ``'S'`` | Seconds | Ignored | + +----------------+----------------------------+-------------------------+ + | ``'M'`` | Minutes | Ignored | + +----------------+----------------------------+-------------------------+ + | ``'H'`` | Hours | Ignored | + +----------------+----------------------------+-------------------------+ + | ``'D'`` | Days | Ignored | + +----------------+----------------------------+-------------------------+ + | ``'W0'-'W6'`` | Weekday (0=Monday) | Used to compute initial | + | | | rollover time | + +----------------+----------------------------+-------------------------+ + | ``'midnight'`` | Roll over at midnight, if | Used to compute initial | + | | *atTime* not specified, | rollover time | + | | else at time *atTime* | | + +----------------+----------------------------+-------------------------+ When using weekday-based rotation, specify 'W0' for Monday, 'W1' for Tuesday, and so on up to 'W6' for Sunday. In this case, the value passed for @@ -352,16 +372,35 @@ timed intervals. If *atTime* is not ``None``, it must be a ``datetime.time`` instance which specifies the time of day when rollover occurs, for the cases where rollover - is set to happen "at midnight" or "on a particular weekday". + is set to happen "at midnight" or "on a particular weekday". Note that in + these cases, the *atTime* value is effectively used to compute the *initial* + rollover, and subsequent rollovers would be calculated via the normal + interval calculation. + + .. note:: Calculation of the initial rollover time is done when the handler + is initialised. Calculation of subsequent rollover times is done only + when rollover occurs, and rollover occurs only when emitting output. If + this is not kept in mind, it might lead to some confusion. For example, + if an interval of "every minute" is set, that does not mean you will + always see log files with times (in the filename) separated by a minute; + if, during application execution, logging output is generated more + frequently than once a minute, *then* you can expect to see log files + with times separated by a minute. If, on the other hand, logging messages + are only output once every five minutes (say), then there will be gaps in + the file times corresponding to the minutes where no output (and hence no + rollover) occurred. .. versionchanged:: 3.4 *atTime* parameter was added. + .. versionchanged:: 3.6 + As well as string values, :class:`~pathlib.Path` objects are also accepted + for the *filename* argument. + .. method:: doRollover() Does a rollover, as described above. - .. method:: emit(record) Outputs the record to the file, catering for rollover as described above. @@ -798,12 +837,18 @@ should, then :meth:`flush` is expected to do the flushing. overridden to implement custom flushing strategies. -.. class:: MemoryHandler(capacity, flushLevel=ERROR, target=None) +.. class:: MemoryHandler(capacity, flushLevel=ERROR, target=None, flushOnClose=True) Returns a new instance of the :class:`MemoryHandler` class. The instance is initialized with a buffer size of *capacity*. If *flushLevel* is not specified, :const:`ERROR` is used. If no *target* is specified, the target will need to be - set using :meth:`setTarget` before this handler does anything useful. + set using :meth:`setTarget` before this handler does anything useful. If + *flushOnClose* is specified as ``False``, then the buffer is *not* flushed when + the handler is closed. If not specified or specified as ``True``, the previous + behaviour of flushing the buffer will occur when the handler is closed. + + .. versionchanged:: 3.6 + The *flushOnClose* parameter was added. .. method:: close() diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst index 61b3ba3bb6..5edb23de83 100644 --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -39,8 +39,9 @@ Reading and writing compressed files object`. The *filename* argument can be either an actual file name (given as a - :class:`str` or :class:`bytes` object), in which case the named file is - opened, or it can be an existing file object to read from or write to. + :class:`str`, :class:`bytes` or :term:`path-like object` object), in + which case the named file is opened, or it can be an existing file object + to read from or write to. The *mode* argument can be any of ``"r"``, ``"rb"``, ``"w"``, ``"wb"``, ``"x"``, ``"xb"``, ``"a"`` or ``"ab"`` for binary mode, or ``"rt"``, @@ -64,6 +65,9 @@ Reading and writing compressed files .. versionchanged:: 3.4 Added support for the ``"x"``, ``"xb"`` and ``"xt"`` modes. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. class:: LZMAFile(filename=None, mode="r", \*, format=None, check=-1, preset=None, filters=None) @@ -71,9 +75,10 @@ Reading and writing compressed files An :class:`LZMAFile` can wrap an already-open :term:`file object`, or operate directly on a named file. The *filename* argument specifies either the file - object to wrap, or the name of the file to open (as a :class:`str` or - :class:`bytes` object). When wrapping an existing file object, the wrapped - file will not be closed when the :class:`LZMAFile` is closed. + object to wrap, or the name of the file to open (as a :class:`str`, + :class:`bytes` or :term:`path-like object` object). When wrapping an + existing file object, the wrapped file will not be closed when the + :class:`LZMAFile` is closed. The *mode* argument can be either ``"r"`` for reading (default), ``"w"`` for overwriting, ``"x"`` for exclusive creation, or ``"a"`` for appending. These @@ -118,6 +123,9 @@ Reading and writing compressed files The :meth:`~io.BufferedIOBase.read` method now accepts an argument of ``None``. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + Compressing and decompressing data in memory -------------------------------------------- diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 3fdea18cfd..da2b8cc586 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -426,6 +426,15 @@ Constants The mathematical constant e = 2.718281..., to available precision. +.. data:: tau + + The mathematical constant τ = 6.283185..., to available precision. + Tau is a circle constant equal to 2π, the ratio of a circle's circumference to + its radius. To learn more about Tau, check out Vi Hart's video `Pi is (still) + Wrong <https://www.youtube.com/watch?v=jG7vhMMXagQ>`_, and start celebrating + `Tau day <http://tauday.com/>`_ by eating twice as much pie! + + .. versionadded:: 3.6 .. data:: inf diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index f4a6f5365d..6e6e29020b 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -264,13 +264,18 @@ To map anonymous memory, -1 should be passed as the fileno along with the length .. method:: write(bytes) Write the bytes in *bytes* into memory at the current position of the - file pointer; the file position is updated to point after the bytes that - were written. If the mmap was created with :const:`ACCESS_READ`, then + file pointer and return the number of bytes written (never less than + ``len(bytes)``, since if the write fails, a :exc:`ValueError` will be + raised). The file position is updated to point after the bytes that + were written. If the mmap was created with :const:`ACCESS_READ`, then writing to it will raise a :exc:`TypeError` exception. .. versionchanged:: 3.5 Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.6 + The number of bytes written is now returned. + .. method:: write_byte(byte) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index d45bc2065d..96d1424126 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -650,8 +650,9 @@ primitives like locks. For passing messages one can use :func:`Pipe` (for a connection between two processes) or a queue (which allows multiple producers and consumers). -The :class:`Queue`, :class:`SimpleQueue` and :class:`JoinableQueue` types are multi-producer, -multi-consumer FIFO queues modelled on the :class:`queue.Queue` class in the +The :class:`Queue`, :class:`SimpleQueue` and :class:`JoinableQueue` types +are multi-producer, multi-consumer :abbr:`FIFO (first-in, first-out)` +queues modelled on the :class:`queue.Queue` class in the standard library. They differ in that :class:`Queue` lacks the :meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join` methods introduced into Python 2.5's :class:`queue.Queue` class. @@ -889,8 +890,13 @@ Miscellaneous .. function:: cpu_count() - Return the number of CPUs in the system. May raise - :exc:`NotImplementedError`. + Return the number of CPUs in the system. + + This number is not equivalent to the number of CPUs the current process can + use. The number of usable CPUs can be obtained with + ``len(os.sched_getaffinity(0))`` + + May raise :exc:`NotImplementedError`. .. seealso:: :func:`os.cpu_count` @@ -1679,7 +1685,9 @@ their parent process exits. The manager classes are defined in the of processes. Objects of this type are returned by :func:`multiprocessing.Manager`. - It also supports creation of shared lists and dictionaries. + Its methods create and return :ref:`multiprocessing-proxy_objects` for a + number of commonly used data types to be synchronized across processes. + This notably includes shared lists and dictionaries. .. method:: Barrier(parties[, action[, timeout]]) @@ -1742,31 +1750,17 @@ their parent process exits. The manager classes are defined in the dict(mapping) dict(sequence) - Create a shared ``dict`` object and return a proxy for it. + Create a shared :class:`dict` object and return a proxy for it. .. method:: list() list(sequence) - Create a shared ``list`` object and return a proxy for it. - - .. note:: - - Modifications to mutable values or items in dict and list proxies will not - be propagated through the manager, because the proxy has no way of knowing - when its values or items are modified. To modify such an item, you can - re-assign the modified object to the container proxy:: - - # create a list proxy and append a mutable object (a dictionary) - lproxy = manager.list() - lproxy.append({}) - # now mutate the dictionary - d = lproxy[0] - d['a'] = 1 - d['b'] = 2 - # at this point, the changes to d are not yet synced, but by - # reassigning the dictionary, the proxy is notified of the change - lproxy[0] = d + Create a shared :class:`list` object and return a proxy for it. + .. versionchanged:: 3.6 + Shared objects are capable of being nested. For example, a shared + container object such as a shared list can contain other shared objects + which will all be managed and synchronized by the :class:`SyncManager`. .. class:: Namespace @@ -1878,6 +1872,8 @@ client to access it remotely:: >>> s = m.get_server() >>> s.serve_forever() +.. _multiprocessing-proxy_objects: + Proxy Objects ~~~~~~~~~~~~~ @@ -1887,8 +1883,7 @@ proxy. Multiple proxy objects may have the same referent. A proxy object has methods which invoke corresponding methods of its referent (although not every method of the referent will necessarily be available through -the proxy). A proxy can usually be used in most of the same ways that its -referent can: +the proxy). In this way, a proxy can be used just like its referent can: .. doctest:: @@ -1909,9 +1904,9 @@ the referent, whereas applying :func:`repr` will return the representation of the proxy. An important feature of proxy objects is that they are picklable so they can be -passed between processes. Note, however, that if a proxy is sent to the -corresponding manager's process then unpickling it will produce the referent -itself. This means, for example, that one shared object can contain a second: +passed between processes. As such, a referent can contain +:ref:`multiprocessing-proxy_objects`. This permits nesting of these managed +lists, dicts, and other :ref:`multiprocessing-proxy_objects`: .. doctest:: @@ -1919,10 +1914,46 @@ itself. This means, for example, that one shared object can contain a second: >>> b = manager.list() >>> a.append(b) # referent of a now contains referent of b >>> print(a, b) - [[]] [] + [<ListProxy object, typeid 'list' at ...>] [] >>> b.append('hello') - >>> print(a, b) - [['hello']] ['hello'] + >>> print(a[0], b) + ['hello'] ['hello'] + +Similarly, dict and list proxies may be nested inside one another:: + + >>> l_outer = manager.list([ manager.dict() for i in range(2) ]) + >>> d_first_inner = l_outer[0] + >>> d_first_inner['a'] = 1 + >>> d_first_inner['b'] = 2 + >>> l_outer[1]['c'] = 3 + >>> l_outer[1]['z'] = 26 + >>> print(l_outer[0]) + {'a': 1, 'b': 2} + >>> print(l_outer[1]) + {'c': 3, 'z': 26} + +If standard (non-proxy) :class:`list` or :class:`dict` objects are contained +in a referent, modifications to those mutable values will not be propagated +through the manager because the proxy has no way of knowing when the values +contained within are modified. However, storing a value in a container proxy +(which triggers a ``__setitem__`` on the proxy object) does propagate through +the manager and so to effectively modify such an item, one could re-assign the +modified value to the container proxy:: + + # create a list proxy and append a mutable object (a dictionary) + lproxy = manager.list() + lproxy.append({}) + # now mutate the dictionary + d = lproxy[0] + d['a'] = 1 + d['b'] = 2 + # at this point, the changes to d are not yet synced, but by + # updating the dictionary, the proxy is notified of the change + lproxy[0] = d + +This approach is perhaps less convenient than employing nested +:ref:`multiprocessing-proxy_objects` for most use cases but also +demonstrates a level of control over the synchronization. .. note:: diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index bf0dface7e..406054e5d7 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -61,6 +61,9 @@ the :mod:`glob` module.) platforms, this is equivalent to calling the function :func:`normpath` as follows: ``normpath(join(os.getcwd(), path))``. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: basename(path) @@ -71,6 +74,9 @@ the :mod:`glob` module.) ``'/foo/bar/'`` returns ``'bar'``, the :func:`basename` function returns an empty string (``''``). + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: commonpath(paths) @@ -83,6 +89,9 @@ the :mod:`glob` module.) .. versionadded:: 3.5 + .. versionchanged:: 3.6 + Accepts a sequence of :term:`path-like objects <path-like object>`. + .. function:: commonprefix(list) @@ -104,12 +113,18 @@ the :mod:`glob` module.) >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) '/usr' + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: dirname(path) Return the directory name of pathname *path*. This is the first element of the pair returned by passing *path* to the function :func:`split`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: exists(path) @@ -123,6 +138,9 @@ the :mod:`glob` module.) *path* can now be an integer: ``True`` is returned if it is an open file descriptor, ``False`` otherwise. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: lexists(path) @@ -130,6 +148,9 @@ the :mod:`glob` module.) broken symbolic links. Equivalent to :func:`exists` on platforms lacking :func:`os.lstat`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: expanduser(path) @@ -151,6 +172,9 @@ the :mod:`glob` module.) If the expansion fails or if the path does not begin with a tilde, the path is returned unchanged. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: expandvars(path) @@ -162,6 +186,9 @@ the :mod:`glob` module.) On Windows, ``%name%`` expansions are supported in addition to ``$name`` and ``${name}``. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: getatime(path) @@ -182,6 +209,9 @@ the :mod:`glob` module.) If :func:`os.stat_float_times` returns ``True``, the result is a floating point number. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: getctime(path) @@ -191,12 +221,18 @@ the :mod:`glob` module.) the :mod:`time` module). Raise :exc:`OSError` if the file does not exist or is inaccessible. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: getsize(path) Return the size, in bytes, of *path*. Raise :exc:`OSError` if the file does not exist or is inaccessible. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: isabs(path) @@ -204,24 +240,36 @@ the :mod:`glob` module.) begins with a slash, on Windows that it begins with a (back)slash after chopping off a potential drive letter. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: isfile(path) Return ``True`` if *path* is an existing regular file. This follows symbolic links, so both :func:`islink` and :func:`isfile` can be true for the same path. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: isdir(path) Return ``True`` if *path* is an existing directory. This follows symbolic links, so both :func:`islink` and :func:`isdir` can be true for the same path. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: islink(path) Return ``True`` if *path* refers to a directory entry that is a symbolic link. Always ``False`` if symbolic links are not supported by the Python runtime. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: ismount(path) @@ -237,6 +285,9 @@ the :mod:`glob` module.) .. versionadded:: 3.4 Support for detecting non-root mount points on Windows. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: join(path, *paths) @@ -255,13 +306,20 @@ the :mod:`glob` module.) ``os.path.join("c:", "foo")`` represents a path relative to the current directory on drive :file:`C:` (:file:`c:foo`), not :file:`c:\\foo`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *path* and *paths*. + .. function:: normcase(path) Normalize the case of a pathname. On Unix and Mac OS X, this returns the path unchanged; on case-insensitive filesystems, it converts the path to lowercase. On Windows, it also converts forward slashes to backward slashes. - Raise a TypeError if the type of *path* is not ``str`` or ``bytes``. + Raise a TypeError if the type of *path* is not ``str`` or ``bytes`` (directly + or indirectly through the :class:`os.PathLike` interface). + + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. .. function:: normpath(path) @@ -272,12 +330,18 @@ the :mod:`glob` module.) that contains symbolic links. On Windows, it converts forward slashes to backward slashes. To normalize case, use :func:`normcase`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: realpath(path) Return the canonical path of the specified filename, eliminating any symbolic links encountered in the path (if they are supported by the operating system). + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: relpath(path, start=os.curdir) @@ -290,6 +354,9 @@ the :mod:`glob` module.) Availability: Unix, Windows. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: samefile(path1, path2) @@ -305,6 +372,9 @@ the :mod:`glob` module.) .. versionchanged:: 3.4 Windows now uses the same implementation as all other platforms. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: sameopenfile(fp1, fp2) @@ -315,6 +385,9 @@ the :mod:`glob` module.) .. versionchanged:: 3.2 Added Windows support. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: samestat(stat1, stat2) @@ -328,6 +401,9 @@ the :mod:`glob` module.) .. versionchanged:: 3.4 Added Windows support. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: split(path) @@ -341,6 +417,9 @@ the :mod:`glob` module.) (but the strings may differ). Also see the functions :func:`dirname` and :func:`basename`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: splitdrive(path) @@ -359,6 +438,9 @@ the :mod:`glob` module.) and share, up to but not including the fourth separator. e.g. ``splitdrive("//host/computer/dir")`` returns ``("//host/computer", "/dir")`` + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: splitext(path) @@ -367,6 +449,9 @@ the :mod:`glob` module.) period. Leading periods on the basename are ignored; ``splitext('.cshrc')`` returns ``('.cshrc', '')``. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: splitunc(path) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 5a67f93e3c..3260cdc093 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -57,7 +57,7 @@ Notes on the availability of these functions: The name of the operating system dependent module imported. The following names have currently been registered: ``'posix'``, ``'nt'``, - ``'ce'``, ``'java'``. + ``'java'``. .. seealso:: :attr:`sys.platform` has a finer granularity. :func:`os.uname` gives @@ -171,23 +171,60 @@ process and user. .. function:: fsencode(filename) - Encode *filename* to the filesystem encoding with ``'surrogateescape'`` - error handler, or ``'strict'`` on Windows; return :class:`bytes` unchanged. + Encode :term:`path-like <path-like object>` *filename* to the filesystem + encoding with ``'surrogateescape'`` error handler, or ``'strict'`` on + Windows; return :class:`bytes` unchanged. :func:`fsdecode` is the reverse function. .. versionadded:: 3.2 + .. versionchanged:: 3.6 + Support added to accept objects implementing the :class:`os.PathLike` + interface. + .. function:: fsdecode(filename) - Decode *filename* from the filesystem encoding with ``'surrogateescape'`` - error handler, or ``'strict'`` on Windows; return :class:`str` unchanged. + Decode the :term:`path-like <path-like object>` *filename* from the + filesystem encoding with ``'surrogateescape'`` error handler, or ``'strict'`` + on Windows; return :class:`str` unchanged. :func:`fsencode` is the reverse function. .. versionadded:: 3.2 + .. versionchanged:: 3.6 + Support added to accept objects implementing the :class:`os.PathLike` + interface. + + +.. function:: fspath(path) + + Return the file system representation of the path. + + If :class:`str` or :class:`bytes` is passed in, it is returned unchanged. + Otherwise :meth:`~os.PathLike.__fspath__` is called and its value is + returned as long as it is a :class:`str` or :class:`bytes` object. + In all other cases, :exc:`TypeError` is raised. + + .. versionadded:: 3.6 + + +.. class:: PathLike + + An :term:`abstract base class` for objects representing a file system path, + e.g. :class:`pathlib.PurePath`. + + .. versionadded:: 3.6 + + .. abstractmethod:: __fspath__() + + Return the file system path representation of the object. + + The method should only return a :class:`str` or :class:`bytes` object, + with the preference being for :class:`str`. + .. function:: getenv(key, default=None) @@ -896,6 +933,9 @@ as internal buffering of data. exception, the function now retries the system call instead of raising an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + The following constants are options for the *flags* parameter to the :func:`~os.open` function. They can be combined using the bitwise OR operator ``|``. Some of them are not available on all platforms. For descriptions of @@ -1392,6 +1432,9 @@ features: .. versionchanged:: 3.3 Added the *dir_fd*, *effective_ids*, and *follow_symlinks* parameters. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. data:: F_OK R_OK @@ -1416,6 +1459,9 @@ features: Added support for specifying *path* as a file descriptor on some platforms. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: chflags(path, flags, *, follow_symlinks=True) @@ -1442,6 +1488,9 @@ features: .. versionadded:: 3.3 The *follow_symlinks* argument. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: chmod(path, mode, *, dir_fd=None, follow_symlinks=True) @@ -1483,6 +1532,9 @@ features: Added support for specifying *path* as an open file descriptor, and the *dir_fd* and *follow_symlinks* arguments. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: chown(path, uid, gid, *, dir_fd=None, follow_symlinks=True) @@ -1502,6 +1554,9 @@ features: Added support for specifying an open file descriptor for *path*, and the *dir_fd* and *follow_symlinks* arguments. + .. versionchanged:: 3.6 + Supports a :term:`path-like object`. + .. function:: chroot(path) @@ -1509,6 +1564,9 @@ features: Availability: Unix. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: fchdir(fd) @@ -1537,6 +1595,9 @@ features: Availability: Unix. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: lchmod(path, mode) @@ -1547,6 +1608,8 @@ features: Availability: Unix. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. .. function:: lchown(path, uid, gid) @@ -1556,6 +1619,9 @@ features: Availability: Unix. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: link(src, dst, *, src_dir_fd=None, dst_dir_fd=None, follow_symlinks=True) @@ -1573,6 +1639,9 @@ features: .. versionadded:: 3.3 Added the *src_dir_fd*, *dst_dir_fd*, and *follow_symlinks* arguments. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *src* and *dst*. + .. function:: listdir(path='.') @@ -1580,8 +1649,9 @@ features: *path*. The list is in arbitrary order, and does not include the special entries ``'.'`` and ``'..'`` even if they are present in the directory. - *path* may be either of type ``str`` or of type ``bytes``. If *path* - is of type ``bytes``, the filenames returned will also be of type ``bytes``; + *path* may be a :term:`path-like object`. If *path* is of type ``bytes`` + (directly or indirectly through the :class:`PathLike` interface), + the filenames returned will also be of type ``bytes``; in all other circumstances, they will be of type ``str``. This function can also support :ref:`specifying a file descriptor @@ -1602,6 +1672,9 @@ features: .. versionadded:: 3.3 Added support for specifying an open file descriptor for *path*. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: lstat(path, \*, dir_fd=None) @@ -1628,6 +1701,9 @@ features: .. versionchanged:: 3.3 Added the *dir_fd* parameter. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *src* and *dst*. + .. function:: mkdir(path, mode=0o777, *, dir_fd=None) @@ -1652,6 +1728,9 @@ features: .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: makedirs(name, mode=0o777, exist_ok=False) @@ -1685,6 +1764,9 @@ features: mode of the existing directory. Since this behavior was impossible to implement safely, it was removed in Python 3.4.1. See :issue:`21082`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: mkfifo(path, mode=0o666, *, dir_fd=None) @@ -1705,6 +1787,9 @@ features: .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: mknod(path, mode=0o600, device=0, *, dir_fd=None) @@ -1724,6 +1809,9 @@ features: .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: major(device) @@ -1762,6 +1850,9 @@ features: Availability: Unix. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. data:: pathconf_names @@ -1779,9 +1870,10 @@ features: may be converted to an absolute pathname using ``os.path.join(os.path.dirname(path), result)``. - If the *path* is a string object, the result will also be a string object, + If the *path* is a string object (directly or indirectly through a + :class:`PathLike` interface), the result will also be a string object, and the call may raise a UnicodeDecodeError. If the *path* is a bytes - object, the result will be a bytes object. + object (direct or indirectly), the result will be a bytes object. This function can also support :ref:`paths relative to directory descriptors <dir_fd>`. @@ -1794,6 +1886,9 @@ features: .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: remove(path, *, dir_fd=None) @@ -1812,6 +1907,9 @@ features: .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: removedirs(name) @@ -1826,6 +1924,9 @@ features: they are empty. Raises :exc:`OSError` if the leaf directory could not be successfully removed. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: rename(src, dst, *, src_dir_fd=None, dst_dir_fd=None) @@ -1845,6 +1946,9 @@ features: .. versionadded:: 3.3 The *src_dir_fd* and *dst_dir_fd* arguments. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *src* and *dst*. + .. function:: renames(old, new) @@ -1858,6 +1962,9 @@ features: This function can fail with the new directory structure made if you lack permissions needed to remove the leaf directory or file. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *old* and *new*. + .. function:: replace(src, dst, *, src_dir_fd=None, dst_dir_fd=None) @@ -1872,6 +1979,9 @@ features: .. versionadded:: 3.3 + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *src* and *dst*. + .. function:: rmdir(path, *, dir_fd=None) @@ -1885,38 +1995,58 @@ features: .. versionadded:: 3.3 The *dir_fd* parameter. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: scandir(path='.') - Return an iterator of :class:`DirEntry` objects corresponding to the entries - in the directory given by *path*. The entries are yielded in arbitrary - order, and the special entries ``'.'`` and ``'..'`` are not included. + Return an iterator of :class:`os.DirEntry` objects corresponding to the + entries in the directory given by *path*. The entries are yielded in + arbitrary order, and the special entries ``'.'`` and ``'..'`` are not + included. Using :func:`scandir` instead of :func:`listdir` can significantly increase the performance of code that also needs file type or file - attribute information, because :class:`DirEntry` objects expose this + attribute information, because :class:`os.DirEntry` objects expose this information if the operating system provides it when scanning a directory. - All :class:`DirEntry` methods may perform a system call, but - :func:`~DirEntry.is_dir` and :func:`~DirEntry.is_file` usually only - require a system call for symbolic links; :func:`DirEntry.stat` + All :class:`os.DirEntry` methods may perform a system call, but + :func:`~os.DirEntry.is_dir` and :func:`~os.DirEntry.is_file` usually only + require a system call for symbolic links; :func:`os.DirEntry.stat` always requires a system call on Unix but only requires one for symbolic links on Windows. - On Unix, *path* can be of type :class:`str` or :class:`bytes` (use + On Unix, *path* can be of type :class:`str` or :class:`bytes` (either + directly or indirectly through the :class:`PathLike` interface; use :func:`~os.fsencode` and :func:`~os.fsdecode` to encode and decode :class:`bytes` paths). On Windows, *path* must be of type :class:`str`. - On both systems, the type of the :attr:`~DirEntry.name` and - :attr:`~DirEntry.path` attributes of each :class:`DirEntry` will be of + On both systems, the type of the :attr:`~os.DirEntry.name` and + :attr:`~os.DirEntry.path` attributes of each :class:`os.DirEntry` will be of the same type as *path*. + The :func:`scandir` iterator supports the :term:`context manager` protocol + and has the following method: + + .. method:: scandir.close() + + Close the iterator and free acquired resources. + + This is called automatically when the iterator is exhausted or garbage + collected, or when an error happens during iterating. However it + is advisable to call it explicitly or use the :keyword:`with` + statement. + + .. versionadded:: 3.6 + The following example shows a simple use of :func:`scandir` to display all the files (excluding directories) in the given *path* that don't start with ``'.'``. The ``entry.is_file()`` call will generally not make an additional system call:: - for entry in os.scandir(path): - if not entry.name.startswith('.') and entry.is_file(): - print(entry.name) + with os.scandir(path) as it: + for entry in it: + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) .. note:: @@ -1932,6 +2062,14 @@ features: .. versionadded:: 3.5 + .. versionadded:: 3.6 + Added support for the :term:`context manager` protocol and the + :func:`~scandir.close()` method. If a :func:`scandir` iterator is neither + exhausted nor explicitly closed a :exc:`ResourceWarning` will be emitted + in its destructor. + + The function accepts a :term:`path-like object`. + .. class:: DirEntry @@ -1940,19 +2078,22 @@ features: :func:`scandir` will provide as much of this information as possible without making additional system calls. When a ``stat()`` or ``lstat()`` system call - is made, the ``DirEntry`` object will cache the result. + is made, the ``os.DirEntry`` object will cache the result. - ``DirEntry`` instances are not intended to be stored in long-lived data + ``os.DirEntry`` instances are not intended to be stored in long-lived data structures; if you know the file metadata has changed or if a long time has elapsed since calling :func:`scandir`, call ``os.stat(entry.path)`` to fetch up-to-date information. - Because the ``DirEntry`` methods can make operating system calls, they may + Because the ``os.DirEntry`` methods can make operating system calls, they may also raise :exc:`OSError`. If you need very fine-grained control over errors, you can catch :exc:`OSError` when calling one of the - ``DirEntry`` methods and handle as appropriate. + ``os.DirEntry`` methods and handle as appropriate. - Attributes and methods on a ``DirEntry`` instance are as follows: + To be directly usable as a :term:`path-like object`, ``os.DirEntry`` + implements the :class:`PathLike` interface. + + Attributes and methods on a ``os.DirEntry`` instance are as follows: .. attribute:: name @@ -1978,8 +2119,9 @@ features: Return the inode number of the entry. - The result is cached on the ``DirEntry`` object. Use ``os.stat(entry.path, - follow_symlinks=False).st_ino`` to fetch up-to-date information. + The result is cached on the ``os.DirEntry`` object. Use + ``os.stat(entry.path, follow_symlinks=False).st_ino`` to fetch up-to-date + information. On the first, uncached call, a system call is required on Windows but not on Unix. @@ -1994,7 +2136,7 @@ features: is a directory (without following symlinks); return ``False`` if the entry is any other kind of file or if it doesn't exist anymore. - The result is cached on the ``DirEntry`` object, with a separate cache + The result is cached on the ``os.DirEntry`` object, with a separate cache for *follow_symlinks* ``True`` and ``False``. Call :func:`os.stat` along with :func:`stat.S_ISDIR` to fetch up-to-date information. @@ -2018,8 +2160,8 @@ features: is a file (without following symlinks); return ``False`` if the entry is a directory or other non-file entry, or if it doesn't exist anymore. - The result is cached on the ``DirEntry`` object. Caching, system calls - made, and exceptions raised are as per :func:`~DirEntry.is_dir`. + The result is cached on the ``os.DirEntry`` object. Caching, system calls + made, and exceptions raised are as per :func:`~os.DirEntry.is_dir`. .. method:: is_symlink() @@ -2027,7 +2169,7 @@ features: return ``False`` if the entry points to a directory or any kind of file, or if it doesn't exist anymore. - The result is cached on the ``DirEntry`` object. Call + The result is cached on the ``os.DirEntry`` object. Call :func:`os.path.islink` to fetch up-to-date information. On the first, uncached call, no system call is required in most cases. @@ -2052,23 +2194,28 @@ features: :class:`stat_result` are always set to zero. Call :func:`os.stat` to get these attributes. - The result is cached on the ``DirEntry`` object, with a separate cache + The result is cached on the ``os.DirEntry`` object, with a separate cache for *follow_symlinks* ``True`` and ``False``. Call :func:`os.stat` to fetch up-to-date information. Note that there is a nice correspondence between several attributes - and methods of ``DirEntry`` and of :class:`pathlib.Path`. In - particular, the ``name`` attribute has the same meaning, as do the - ``is_dir()``, ``is_file()``, ``is_symlink()`` and ``stat()`` methods. + and methods of ``os.DirEntry`` and of :class:`pathlib.Path`. In + particular, the ``name`` attribute has the same + meaning, as do the ``is_dir()``, ``is_file()``, ``is_symlink()`` + and ``stat()`` methods. .. versionadded:: 3.5 + .. versionchanged:: 3.6 + Added support for the :class:`~os.PathLike` interface. + .. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) Get the status of a file or a file descriptor. Perform the equivalent of a :c:func:`stat` system call on the given path. *path* may be specified as - either a string or as an open file descriptor. Return a :class:`stat_result` + either a string -- directly or indirectly through the :class:`PathLike` + interface -- or as an open file descriptor. Return a :class:`stat_result` object. This function normally follows symlinks; to stat a symlink add the argument @@ -2098,6 +2245,9 @@ features: Added the *dir_fd* and *follow_symlinks* arguments, specifying a file descriptor instead of a path. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. class:: stat_result @@ -2319,19 +2469,22 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. + Availability: Unix. + .. versionchanged:: 3.2 The :const:`ST_RDONLY` and :const:`ST_NOSUID` constants were added. + .. versionadded:: 3.3 + Added support for specifying an open file descriptor for *path*. + .. versionchanged:: 3.4 The :const:`ST_NODEV`, :const:`ST_NOEXEC`, :const:`ST_SYNCHRONOUS`, :const:`ST_MANDLOCK`, :const:`ST_WRITE`, :const:`ST_APPEND`, :const:`ST_IMMUTABLE`, :const:`ST_NOATIME`, :const:`ST_NODIRATIME`, and :const:`ST_RELATIME` constants were added. - Availability: Unix. - - .. versionadded:: 3.3 - Added support for specifying an open file descriptor for *path*. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. .. data:: supports_dir_fd @@ -2452,6 +2605,9 @@ features: Added the *dir_fd* argument, and now allow *target_is_directory* on non-Windows platforms. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *src* and *dst*. + .. function:: sync() @@ -2476,6 +2632,10 @@ features: .. versionchanged:: 3.5 Added support for Windows + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + + .. function:: unlink(path, *, dir_fd=None) Remove (delete) the file *path*. This function is semantically @@ -2486,6 +2646,9 @@ features: .. versionadded:: 3.3 The *dir_fd* parameter. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: utime(path, times=None, *[, ns], dir_fd=None, follow_symlinks=True) @@ -2523,6 +2686,9 @@ features: Added support for specifying an open file descriptor for *path*, and the *dir_fd*, *follow_symlinks*, and *ns* parameters. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: walk(top, topdown=True, onerror=None, followlinks=False) @@ -2613,6 +2779,9 @@ features: This function now calls :func:`os.scandir` instead of :func:`os.listdir`, making it faster by reducing the number of calls to :func:`os.stat`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: fwalk(top='.', topdown=True, onerror=None, *, follow_symlinks=False, dir_fd=None) @@ -2669,6 +2838,9 @@ features: .. versionadded:: 3.3 + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + Linux extended attributes ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2680,12 +2852,16 @@ These functions are all available on Linux only. .. function:: getxattr(path, attribute, *, follow_symlinks=True) Return the value of the extended filesystem attribute *attribute* for - *path*. *attribute* can be bytes or str. If it is str, it is encoded - with the filesystem encoding. + *path*. *attribute* can be bytes or str (directly or indirectly through the + :class:`PathLike` interface). If it is str, it is encoded with the filesystem + encoding. This function can support :ref:`specifying a file descriptor <path_fd>` and :ref:`not following symlinks <follow_symlinks>`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` fpr *path* and *attribute*. + .. function:: listxattr(path=None, *, follow_symlinks=True) @@ -2697,21 +2873,29 @@ These functions are all available on Linux only. This function can support :ref:`specifying a file descriptor <path_fd>` and :ref:`not following symlinks <follow_symlinks>`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: removexattr(path, attribute, *, follow_symlinks=True) Removes the extended filesystem attribute *attribute* from *path*. - *attribute* should be bytes or str. If it is a string, it is encoded + *attribute* should be bytes or str (directly or indirectly through the + :class:`PathLike` interface). If it is a string, it is encoded with the filesystem encoding. This function can support :ref:`specifying a file descriptor <path_fd>` and :ref:`not following symlinks <follow_symlinks>`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *path* and *attribute*. + .. function:: setxattr(path, attribute, value, flags=0, *, follow_symlinks=True) Set the extended filesystem attribute *attribute* on *path* to *value*. - *attribute* must be a bytes or str with no embedded NULs. If it is a str, + *attribute* must be a bytes or str with no embedded NULs (directly or + indirectly through the :class:`PathLike` interface). If it is a str, it is encoded with the filesystem encoding. *flags* may be :data:`XATTR_REPLACE` or :data:`XATTR_CREATE`. If :data:`XATTR_REPLACE` is given and the attribute does not exist, ``EEXISTS`` will be raised. @@ -2726,6 +2910,9 @@ These functions are all available on Linux only. A bug in Linux kernel versions less than 2.6.39 caused the flags argument to be ignored on some filesystems. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object` for *path* and *attribute*. + .. data:: XATTR_SIZE_MAX @@ -2827,6 +3014,9 @@ to be ignored. Added support for specifying an open file descriptor for *path* for :func:`execve`. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. function:: _exit(n) Exit the process with status *n*, without calling cleanup handlers, flushing @@ -3137,6 +3327,9 @@ written in Python, such as a mail server's external command delivery program. :func:`spawnve` are not thread-safe on Windows; we advise you to use the :mod:`subprocess` module instead. + .. versionchanged:: 3.6 + Accepts a :term:`path-like object`. + .. data:: P_NOWAIT P_NOWAITO @@ -3623,6 +3816,11 @@ Miscellaneous System Information Return the number of CPUs in the system. Returns ``None`` if undetermined. + This number is not equivalent to the number of CPUs the current process can + use. The number of usable CPUs can be obtained with + ``len(os.sched_getaffinity(0))`` + + .. versionadded:: 3.4 @@ -3738,35 +3936,89 @@ Higher-level operations on pathnames are defined in the :mod:`os.path` module. .. versionadded:: 3.3 -.. _os-miscfunc: -Miscellaneous Functions ------------------------ +Random numbers +-------------- + + +.. function:: getrandom(size, flags=0) + + Get up to *size* random bytes. The function can return less bytes than + requested. + + These bytes can be used to seed user-space random number generators or for + cryptographic purposes. + + ``getrandom()`` relies on entropy gathered from device drivers and other + sources of environmental noise. Unnecessarily reading large quantities of + data will have a negative impact on other users of the ``/dev/random`` and + ``/dev/urandom`` devices. + + The flags argument is a bit mask that can contain zero or more of the + following values ORed together: :py:data:`os.GRND_RANDOM` and + :py:data:`GRND_NONBLOCK`. + + See also the `Linux getrandom() manual page + <http://man7.org/linux/man-pages/man2/getrandom.2.html>`_. + + Availability: Linux 3.17 and newer. + .. versionadded:: 3.6 -.. function:: urandom(n) +.. function:: urandom(size) - Return a string of *n* random bytes suitable for cryptographic use. + Return a string of *size* random bytes suitable for cryptographic use. This function returns random bytes from an OS-specific randomness source. The returned data should be unpredictable enough for cryptographic applications, though its exact quality depends on the OS implementation. - On Linux, ``getrandom()`` syscall is used if available and the urandom - entropy pool is initialized (``getrandom()`` does not block). - On a Unix-like system this will query ``/dev/urandom``. On Windows, it - will use ``CryptGenRandom()``. If a randomness source is not found, - :exc:`NotImplementedError` will be raised. + On Linux, if the ``getrandom()`` syscall is available, it is used in + blocking mode: block until the system urandom entropy pool is initialized + (128 bits of entropy are collected by the kernel). See the :pep:`524` for + the rationale. On Linux, the :func:`getrandom` function can be used to get + random bytes in non-blocking mode (using the :data:`GRND_NONBLOCK` flag) or + to poll until the system urandom entropy pool is initialized. - For an easy-to-use interface to the random number generator - provided by your platform, please see :class:`random.SystemRandom`. + On a Unix-like system, random bytes are read from the ``/dev/urandom`` + device. If the ``/dev/urandom`` device is not available or not readable, the + :exc:`NotImplementedError` exception is raised. + + On Windows, it will use ``CryptGenRandom()``. + + .. seealso:: + The :mod:`secrets` module provides higher level functions. For an + easy-to-use interface to the random number generator provided by your + platform, please see :class:`random.SystemRandom`. + + .. versionchanged:: 3.6.0 + On Linux, ``getrandom()`` is now used in blocking mode to increase the + security. .. versionchanged:: 3.5.2 - On Linux, if ``getrandom()`` blocks (the urandom entropy pool is not - initialized yet), fall back on reading ``/dev/urandom``. + On Linux, if the ``getrandom()`` syscall blocks (the urandom entropy pool + is not initialized yet), fall back on reading ``/dev/urandom``. .. versionchanged:: 3.5 On Linux 3.17 and newer, the ``getrandom()`` syscall is now used when available. On OpenBSD 5.6 and newer, the C ``getentropy()`` function is now used. These functions avoid the usage of an internal file descriptor. + +.. data:: GRND_NONBLOCK + + By default, when reading from ``/dev/random``, :func:`getrandom` blocks if + no random bytes are available, and when reading from ``/dev/urandom``, it blocks + if the entropy pool has not yet been initialized. + + If the :py:data:`GRND_NONBLOCK` flag is set, then :func:`getrandom` does not + block in these cases, but instead immediately raises :exc:`BlockingIOError`. + + .. versionadded:: 3.6 + +.. data:: GRND_RANDOM + + If this bit is set, then random bytes are drawn from the + ``/dev/random`` pool instead of the ``/dev/urandom`` pool. + + .. versionadded:: 3.6 diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 7b90a1f3f9..5a819175cf 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -35,12 +35,6 @@ Pure paths are useful in some special cases; for example: accessing the OS. In this case, instantiating one of the pure classes may be useful since those simply don't have any OS-accessing operations. -.. note:: - This module has been included in the standard library on a - :term:`provisional basis <provisional package>`. Backwards incompatible - changes (up to and including removal of the package) may occur if deemed - necessary by the core developers. - .. seealso:: :pep:`428`: The pathlib module -- object-oriented filesystem paths. @@ -111,7 +105,8 @@ we also call *flavours*: PurePosixPath('setup.py') Each element of *pathsegments* can be either a string representing a - path segment, or another path object:: + path segment, an object implementing the :class:`os.PathLike` interface + which returns a string, or another path object:: >>> PurePath('foo', 'some/path', 'bar') PurePosixPath('foo/some/path/bar') @@ -152,6 +147,12 @@ we also call *flavours*: to ``PurePosixPath('bar')``, which is wrong if ``foo`` is a symbolic link to another directory) + Pure path objects implement the :class:`os.PathLike` interface, allowing them + to be used anywhere the interface is accepted. + + .. versionchanged:: 3.6 + Added support for the :class:`os.PathLike` interface. + .. class:: PurePosixPath(*pathsegments) A subclass of :class:`PurePath`, this path flavour represents non-Windows @@ -199,7 +200,7 @@ Paths of a different flavour compare unequal and cannot be ordered:: >>> PureWindowsPath('foo') < PurePosixPath('foo') Traceback (most recent call last): File "<stdin>", line 1, in <module> - TypeError: unorderable types: PureWindowsPath() < PurePosixPath() + TypeError: '<' not supported between instances of 'PureWindowsPath' and 'PurePosixPath' Operators @@ -216,6 +217,14 @@ The slash operator helps create child paths, similarly to :func:`os.path.join`:: >>> '/usr' / q PurePosixPath('/usr/bin') +A path object can be used anywhere an object implementing :class:`os.PathLike` +is accepted:: + + >>> import os + >>> p = PurePath('/etc') + >>> os.fspath(p) + '/etc' + The string representation of a path is the raw filesystem path itself (in native form, e.g. with backslashes under Windows), which you can pass to any function taking a file path as a string:: diff --git a/Doc/library/pdb.rst b/Doc/library/pdb.rst index d77de2e2bb..52bef6820d 100644 --- a/Doc/library/pdb.rst +++ b/Doc/library/pdb.rst @@ -144,7 +144,7 @@ The ``run*`` functions and :func:`set_trace` are aliases for instantiating the access further features, you have to do this yourself: .. class:: Pdb(completekey='tab', stdin=None, stdout=None, skip=None, \ - nosigint=False) + nosigint=False, readrc=True) :class:`Pdb` is the debugger class. @@ -160,6 +160,9 @@ access further features, you have to do this yourself: This allows you to break into the debugger again by pressing :kbd:`Ctrl-C`. If you want Pdb not to touch the SIGINT handler, set *nosigint* to true. + The *readrc* argument defaults to true and controls whether Pdb will load + .pdbrc files from the filesystem. + Example call to enable tracing with *skip*:: import pdb; pdb.Pdb(skip=['django.*']).set_trace() @@ -171,6 +174,9 @@ access further features, you have to do this yourself: The *nosigint* argument. Previously, a SIGINT handler was never set by Pdb. + .. versionchanged:: 3.6 + The *readrc* argument. + .. method:: run(statement, globals=None, locals=None) runeval(expression, globals=None, locals=None) runcall(function, *args, **kwds) diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 0d64191d79..6e8430fa8e 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -492,7 +492,7 @@ methods: .. method:: object.__getnewargs_ex__() - In protocols 4 and newer, classes that implements the + In protocols 2 and newer, classes that implements the :meth:`__getnewargs_ex__` method can dictate the values passed to the :meth:`__new__` method upon unpickling. The method must return a pair ``(args, kwargs)`` where *args* is a tuple of positional arguments @@ -504,15 +504,22 @@ methods: class requires keyword-only arguments. Otherwise, it is recommended for compatibility to implement :meth:`__getnewargs__`. + .. versionchanged:: 3.6 + :meth:`__getnewargs_ex__` is now used in protocols 2 and 3. + .. method:: object.__getnewargs__() - This method serve a similar purpose as :meth:`__getnewargs_ex__` but - for protocols 2 and newer. It must return a tuple of arguments ``args`` - which will be passed to the :meth:`__new__` method upon unpickling. + This method serve a similar purpose as :meth:`__getnewargs_ex__`, but + supports only positional arguments. It must return a tuple of arguments + ``args`` which will be passed to the :meth:`__new__` method upon unpickling. + + :meth:`__getnewargs__` will not be called if :meth:`__getnewargs_ex__` is + defined. - In protocols 4 and newer, :meth:`__getnewargs__` will not be called if - :meth:`__getnewargs_ex__` is defined. + .. versionchanged:: 3.6 + Before Python 3.6, :meth:`__getnewargs__` was called instead of + :meth:`__getnewargs_ex__` in protocols 2 and 3. .. method:: object.__getstate__() diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index abeac62952..90d69e4071 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -11,6 +11,11 @@ This module provides utilities for the import system, in particular package support. +.. class:: ModuleInfo(module_finder, name, ispkg) + + A namedtuple that holds a brief summary of a module's info. + + .. versionadded:: 3.6 .. function:: extend_path(path, name) @@ -139,7 +144,7 @@ support. .. function:: iter_modules(path=None, prefix='') - Yields ``(module_finder, name, ispkg)`` for all submodules on *path*, or, if + Yields :class:`ModuleInfo` for all submodules on *path*, or, if *path* is ``None``, all top-level modules on ``sys.path``. *path* should be either ``None`` or a list of paths to look for modules in. @@ -160,7 +165,7 @@ support. .. function:: walk_packages(path=None, prefix='', onerror=None) - Yields ``(module_finder, name, ispkg)`` for all modules recursively on + Yields :class:`ModuleInfo` for all modules recursively on *path*, or, if *path* is ``None``, all accessible modules. *path* should be either ``None`` or a list of paths to look for modules in. diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst index ffabc32b6e..d72b660d6e 100644 --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -62,6 +62,13 @@ The :mod:`poplib` module provides two classes: :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see :data:`ssl.HAS_SNI`). + .. deprecated:: 3.6 + + *keyfile* and *certfile* are deprecated in favor of *context*. + Please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let + :func:`ssl.create_default_context` select the system's trusted CA + certificates for you. + One exception is defined as an attribute of the :mod:`poplib` module: diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 924be5aadf..bd0fc2d8f3 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -16,8 +16,9 @@ availability of thread support in Python; see the :mod:`threading` module. The module implements three types of queue, which differ only in the order in -which the entries are retrieved. In a FIFO queue, the first tasks added are -the first retrieved. In a LIFO queue, the most recently added entry is +which the entries are retrieved. In a :abbr:`FIFO (first-in, first-out)` +queue, the first tasks added are the first retrieved. In a +:abbr:`LIFO (last-in, first-out)` queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the :mod:`heapq` module) and the lowest valued entry is retrieved first. @@ -29,14 +30,16 @@ The :mod:`queue` module defines the following classes and exceptions: .. class:: Queue(maxsize=0) - Constructor for a FIFO queue. *maxsize* is an integer that sets the upperbound + Constructor for a :abbr:`FIFO (first-in, first-out)` queue. *maxsize* is + an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If *maxsize* is less than or equal to zero, the queue size is infinite. .. class:: LifoQueue(maxsize=0) - Constructor for a LIFO queue. *maxsize* is an integer that sets the upperbound + Constructor for a :abbr:`LIFO (last-in, first-out)` queue. *maxsize* is + an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If *maxsize* is less than or equal to zero, the queue size is infinite. diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 22f18a0cd5..a47ed9ce3d 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -46,7 +46,8 @@ from sources provided by the operating system. .. warning:: The pseudo-random generators of this module should not be used for - security purposes. + security purposes. For security or cryptographic uses, see the + :mod:`secrets` module. Bookkeeping functions: @@ -123,6 +124,31 @@ Functions for sequences: Return a random element from the non-empty sequence *seq*. If *seq* is empty, raises :exc:`IndexError`. +.. function:: choices(population, weights=None, *, cum_weights=None, k=1) + + Return a *k* sized list of elements chosen from the *population* with replacement. + If the *population* is empty, raises :exc:`IndexError`. + + If a *weights* sequence is specified, selections are made according to the + relative weights. Alternatively, if a *cum_weights* sequence is given, the + selections are made according to the cumulative weights (perhaps computed + using :func:`itertools.accumulate`). For example, the relative weights + ``[10, 5, 30, 5]`` are equivalent to the cumulative weights + ``[10, 15, 45, 50]``. Internally, the relative weights are converted to + cumulative weights before making selections, so supplying the cumulative + weights saves work. + + If neither *weights* nor *cum_weights* are specified, selections are made + with equal probability. If a weights sequence is supplied, it must be + the same length as the *population* sequence. It is a :exc:`TypeError` + to specify both *weights* and *cum_weights*. + + The *weights* or *cum_weights* can use any numeric type that interoperates + with the :class:`float` values returned by :func:`random` (that includes + integers, floats, and fractions but excludes decimals). + + .. versionadded:: 3.6 + .. function:: shuffle(x[, random]) @@ -313,36 +339,28 @@ Basic usage:: >>> random.choice('abcdefghij') # Single random element 'c' - >>> items = [1, 2, 3, 4, 5, 6, 7] - >>> random.shuffle(items) - >>> items - [7, 3, 2, 5, 6, 4, 1] + >>> deck = ['jack', 'queen', 'king', 'ace'] + >>> shuffle(deck) + >>> deck + ['king', 'queen', 'ace', 'jack'] >>> random.sample([1, 2, 3, 4, 5], 3) # Three samples without replacement [4, 1, 5] -A common task is to make a :func:`random.choice` with weighted probabilities. - -If the weights are small integer ratios, a simple technique is to build a sample -population with repeats:: - - >>> weighted_choices = [('Red', 3), ('Blue', 2), ('Yellow', 1), ('Green', 4)] - >>> population = [val for val, cnt in weighted_choices for i in range(cnt)] - >>> population - ['Red', 'Red', 'Red', 'Blue', 'Blue', 'Yellow', 'Green', 'Green', 'Green', 'Green'] - - >>> random.choice(population) - 'Green' + >>> # Six weighted samples with replacement + >>> choices(['red', 'black', 'green'], [18, 18, 2], k=6) + ['red', 'green', 'black', 'black', 'red', 'black'] -A more general approach is to arrange the weights in a cumulative distribution -with :func:`itertools.accumulate`, and then locate the random value with -:func:`bisect.bisect`:: +Example of `statistical bootstrapping +<https://en.wikipedia.org/wiki/Bootstrapping_(statistics)>`_ using resampling +with replacement to estimate a confidence interval for the mean of a small +sample of size five:: - >>> choices, weights = zip(*weighted_choices) - >>> cumdist = list(itertools.accumulate(weights)) - >>> cumdist # [3, 3+2, 3+2+1, 3+2+1+4] - [3, 5, 6, 10] + # http://statistics.about.com/od/Applications/a/Example-Of-Bootstrapping.htm + from statistics import mean + from random import choices - >>> x = random.random() * cumdist[-1] - >>> choices[bisect.bisect(cumdist, x)] - 'Blue' + data = 1, 2, 4, 4, 10 + means = sorted(mean(choices(data, k=5)) for i in range(20)) + print('The sample mean of {:.1f} has a 90% confidence interval ' + 'from {:.1f} to {:.1f}'.format(mean(data), means[1], means[-2])) diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 1ca621eca6..c9f2263375 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -230,12 +230,8 @@ The special characters are: flags are described in :ref:`contents-of-module-re`.) This is useful if you wish to include the flags as part of the regular expression, instead of passing a *flag* argument to the - :func:`re.compile` function. - - Note that the ``(?x)`` flag changes how the expression is parsed. It should be - used first in the expression string, or after one or more whitespace characters. - If there are non-whitespace characters before the flag, the results are - undefined. + :func:`re.compile` function. Flags should be used first in the + expression string. ``(?:...)`` A non-capturing version of regular parentheses. Matches whatever regular @@ -243,6 +239,16 @@ The special characters are: *cannot* be retrieved after performing a match or referenced later in the pattern. +``(?imsx-imsx:...)`` + (Zero or more letters from the set ``'i'``, ``'m'``, ``'s'``, ``'x'``, + optionally followed by ``'-'`` followed by one or more letters from the + same set.) The letters set or removes the corresponding flags: + :const:`re.I` (ignore case), :const:`re.M` (multi-line), :const:`re.S` + (dot matches all), and :const:`re.X` (verbose), for the part of the + expression. (The flags are described in :ref:`contents-of-module-re`.) + + .. versionadded:: 3.6 + ``(?P<name>...)`` Similar to regular parentheses, but the substring matched by the group is accessible via the symbolic group name *name*. Group names must be valid @@ -327,8 +333,9 @@ The special characters are: The special sequences consist of ``'\'`` and a character from the list below. -If the ordinary character is not on the list, then the resulting RE will match -the second character. For example, ``\$`` matches the character ``'$'``. +If the ordinary character is not an ASCII digit or an ASCII letter, then the +resulting RE will match the second character. For example, ``\$`` matches the +character ``'$'``. ``\number`` Matches the contents of the group of the same number. Groups are numbered @@ -448,9 +455,8 @@ three digits in length. .. versionchanged:: 3.3 The ``'\u'`` and ``'\U'`` escape sequences have been added. -.. deprecated-removed:: 3.5 3.6 - Unknown escapes consisting of ``'\'`` and ASCII letter now raise a - deprecation warning and will be forbidden in Python 3.6. +.. versionchanged:: 3.6 + Unknown escapes consisting of ``'\'`` and an ASCII letter now are errors. .. seealso:: @@ -538,11 +544,11 @@ form. current locale. The use of this flag is discouraged as the locale mechanism is very unreliable, and it only handles one "culture" at a time anyway; you should use Unicode matching instead, which is the default in Python 3 - for Unicode (str) patterns. This flag makes sense only with bytes patterns. + for Unicode (str) patterns. This flag can be used only with bytes patterns. - .. deprecated-removed:: 3.5 3.6 - Deprecated the use of :const:`re.LOCALE` with string patterns or - :const:`re.ASCII`. + .. versionchanged:: 3.6 + :const:`re.LOCALE` can be used only with bytes patterns and is + not compatible with :const:`re.ASCII`. .. data:: M @@ -748,9 +754,8 @@ form. .. versionchanged:: 3.5 Unmatched groups are replaced with an empty string. - .. deprecated-removed:: 3.5 3.6 - Unknown escapes consist of ``'\'`` and ASCII letter now raise a - deprecation warning and will be forbidden in Python 3.6. + .. versionchanged:: 3.6 + Unknown escapes consisting of ``'\'`` and an ASCII letter now are errors. .. function:: subn(pattern, repl, string, count=0, flags=0) @@ -1016,6 +1021,22 @@ Match objects support the following methods and attributes: 'c3' +.. method:: match.__getitem__(g) + + This is identical to ``m.group(g)``. This allows easier access to + an individual group from a match: + + >>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist") + >>> m[0] # The entire match + 'Isaac Newton' + >>> m[1] # The first parenthesized subgroup. + 'Isaac' + >>> m[2] # The second parenthesized subgroup. + 'Newton' + + .. versionadded:: 3.6 + + .. method:: match.groups(default=None) Return a tuple containing all the subgroups of the match, from 1 up to however diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index 4d3c099ed2..54c54b461c 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -167,6 +167,20 @@ The following functions operate on a global history list: This calls :c:func:`add_history` in the underlying library. +.. function:: set_auto_history(enabled) + + Enable or disable automatic calls to :c:func:`add_history` when reading + input via readline. The *enabled* argument should be a Boolean value + that when true, enables auto history, and that when false, disables + auto history. + + .. versionadded:: 3.6 + + .. impl-detail:: + Auto history is enabled by default, and changes to this do not persist + across multiple sessions. + + Startup hooks ------------- diff --git a/Doc/library/secrets.rst b/Doc/library/secrets.rst new file mode 100644 index 0000000000..9bf848f911 --- /dev/null +++ b/Doc/library/secrets.rst @@ -0,0 +1,198 @@ +:mod:`secrets` --- Generate secure random numbers for managing secrets +====================================================================== + +.. module:: secrets + :synopsis: Generate secure random numbers for managing secrets. + +.. moduleauthor:: Steven D'Aprano <steve+python@pearwood.info> +.. sectionauthor:: Steven D'Aprano <steve+python@pearwood.info> +.. versionadded:: 3.6 + +.. testsetup:: + + from secrets import * + __name__ = '<doctest>' + +**Source code:** :source:`Lib/secrets.py` + +------------- + +The :mod:`secrets` module is used for generating cryptographically strong +random numbers suitable for managing data such as passwords, account +authentication, security tokens, and related secrets. + +In particularly, :mod:`secrets` should be used in preference to the +default pseudo-random number generator in the :mod:`random` module, which +is designed for modelling and simulation, not security or cryptography. + +.. seealso:: + + :pep:`506` + + +Random numbers +-------------- + +The :mod:`secrets` module provides access to the most secure source of +randomness that your operating system provides. + +.. class:: SystemRandom + + A class for generating random numbers using the highest-quality + sources provided by the operating system. See + :class:`random.SystemRandom` for additional details. + +.. function:: choice(sequence) + + Return a randomly-chosen element from a non-empty sequence. + +.. function:: randbelow(n) + + Return a random int in the range [0, *n*). + +.. function:: randbits(k) + + Return an int with *k* random bits. + + +Generating tokens +----------------- + +The :mod:`secrets` module provides functions for generating secure +tokens, suitable for applications such as password resets, +hard-to-guess URLs, and similar. + +.. function:: token_bytes([nbytes=None]) + + Return a random byte string containing *nbytes* number of bytes. + If *nbytes* is ``None`` or not supplied, a reasonable default is + used. + + .. doctest:: + + >>> token_bytes(16) #doctest:+SKIP + b'\xebr\x17D*t\xae\xd4\xe3S\xb6\xe2\xebP1\x8b' + + +.. function:: token_hex([nbytes=None]) + + Return a random text string, in hexadecimal. The string has *nbytes* + random bytes, each byte converted to two hex digits. If *nbytes* is + ``None`` or not supplied, a reasonable default is used. + + .. doctest:: + + >>> token_hex(16) #doctest:+SKIP + 'f9bf78b9a18ce6d46a0cd2b0b86df9da' + +.. function:: token_urlsafe([nbytes=None]) + + Return a random URL-safe text string, containing *nbytes* random + bytes. The text is Base64 encoded, so on average each byte results + in approximately 1.3 characters. If *nbytes* is ``None`` or not + supplied, a reasonable default is used. + + .. doctest:: + + >>> token_urlsafe(16) #doctest:+SKIP + 'Drmhze6EPcv0fN_81Bj-nA' + + +How many bytes should tokens use? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To be secure against +`brute-force attacks <https://en.wikipedia.org/wiki/Brute-force_attack>`_, +tokens need to have sufficient randomness. Unfortunately, what is +considered sufficient will necessarily increase as computers get more +powerful and able to make more guesses in a shorter period. As of 2015, +it is believed that 32 bytes (256 bits) of randomness is sufficient for +the typical use-case expected for the :mod:`secrets` module. + +For those who want to manage their own token length, you can explicitly +specify how much randomness is used for tokens by giving an :class:`int` +argument to the various ``token_*`` functions. That argument is taken +as the number of bytes of randomness to use. + +Otherwise, if no argument is provided, or if the argument is ``None``, +the ``token_*`` functions will use a reasonable default instead. + +.. note:: + + That default is subject to change at any time, including during + maintenance releases. + + +Other functions +--------------- + +.. function:: compare_digest(a, b) + + Return ``True`` if strings *a* and *b* are equal, otherwise ``False``, + in such a way as to reduce the risk of + `timing attacks <http://codahale.com/a-lesson-in-timing-attacks/>`_. + See :func:`hmac.compare_digest` for additional details. + + +Recipes and best practices +-------------------------- + +This section shows recipes and best practices for using :mod:`secrets` +to manage a basic level of security. + +Generate an eight-character alphanumeric password: + +.. testcode:: + + import string + alphabet = string.ascii_letters + string.digits + password = ''.join(choice(alphabet) for i in range(8)) + + +.. note:: + + Applications should not + `store passwords in a recoverable format <http://cwe.mitre.org/data/definitions/257.html>`_, + whether plain text or encrypted. They should be salted and hashed + using a cryptographically-strong one-way (irreversible) hash function. + + +Generate a ten-character alphanumeric password with at least one +lowercase character, at least one uppercase character, and at least +three digits: + +.. testcode:: + + import string + alphabet = string.ascii_letters + string.digits + while True: + password = ''.join(choice(alphabet) for i in range(10)) + if (any(c.islower() for c in password) + and any(c.isupper() for c in password) + and sum(c.isdigit() for c in password) >= 3): + break + + +Generate an `XKCD-style passphrase <http://xkcd.com/936/>`_: + +.. testcode:: + + # On standard Linux systems, use a convenient dictionary file. + # Other platforms may need to provide their own word-list. + with open('/usr/share/dict/words') as f: + words = [word.strip() for word in f] + password = ' '.join(choice(words) for i in range(4)) + + +Generate a hard-to-guess temporary URL containing a security token +suitable for password recovery applications: + +.. testcode:: + + url = 'https://mydomain.com/reset=' + token_urlsafe() + + + +.. + # This modeline must appear within the last ten lines of the file. + kate: indent-width 3; remove-trailing-space on; replace-tabs on; encoding utf-8; diff --git a/Doc/library/select.rst b/Doc/library/select.rst index a81386a4fb..f97118ebe0 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -268,35 +268,43 @@ Edge and Level Trigger Polling (epoll) Objects *eventmask* - +-----------------------+-----------------------------------------------+ - | Constant | Meaning | - +=======================+===============================================+ - | :const:`EPOLLIN` | Available for read | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLOUT` | Available for write | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLPRI` | Urgent data for read | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLERR` | Error condition happened on the assoc. fd | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLHUP` | Hang up happened on the assoc. fd | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLET` | Set Edge Trigger behavior, the default is | - | | Level Trigger behavior | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLONESHOT` | Set one-shot behavior. After one event is | - | | pulled out, the fd is internally disabled | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDNORM` | Equivalent to :const:`EPOLLIN` | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDBAND` | Priority data band can be read. | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRNORM` | Equivalent to :const:`EPOLLOUT` | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRBAND` | Priority data may be written. | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLMSG` | Ignored. | - +-----------------------+-----------------------------------------------+ + +-------------------------+-----------------------------------------------+ + | Constant | Meaning | + +=========================+===============================================+ + | :const:`EPOLLIN` | Available for read | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLOUT` | Available for write | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLPRI` | Urgent data for read | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLERR` | Error condition happened on the assoc. fd | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLHUP` | Hang up happened on the assoc. fd | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLET` | Set Edge Trigger behavior, the default is | + | | Level Trigger behavior | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLONESHOT` | Set one-shot behavior. After one event is | + | | pulled out, the fd is internally disabled | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLEXCLUSIVE` | Wake only one epoll object when the | + | | associated fd has an event. The default (if | + | | this flag is not set) is to wake all epoll | + | | objects polling on on a fd. | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLRDHUP` | Stream socket peer closed connection or shut | + | | down writing half of connection. | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLRDNORM` | Equivalent to :const:`EPOLLIN` | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLRDBAND` | Priority data band can be read. | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLWRNORM` | Equivalent to :const:`EPOLLOUT` | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLWRBAND` | Priority data may be written. | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLMSG` | Ignored. | + +-------------------------+-----------------------------------------------+ .. method:: epoll.close() @@ -385,6 +393,9 @@ linearly scanned again. :c:func:`select` is O(highest file descriptor), while +-------------------+------------------------------------------+ | :const:`POLLHUP` | Hung up | +-------------------+------------------------------------------+ + | :const:`POLLRDHUP`| Stream socket peer closed connection, or | + | | shut down writing half of connection | + +-------------------+------------------------------------------+ | :const:`POLLNVAL` | Invalid request: descriptor not open | +-------------------+------------------------------------------+ diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst index e81f9822bb..1a89bf6041 100644 --- a/Doc/library/shlex.rst +++ b/Doc/library/shlex.rst @@ -73,11 +73,11 @@ The :mod:`shlex` module defines the following functions: The :mod:`shlex` module defines the following class: -.. class:: shlex(instream=None, infile=None, posix=False) +.. class:: shlex(instream=None, infile=None, posix=False, punctuation_chars=False) A :class:`~shlex.shlex` instance or subclass instance is a lexical analyzer object. The initialization argument, if present, specifies where to read - characters from. It must be a file-/stream-like object with + characters from. It must be a file-/stream-like object with :meth:`~io.TextIOBase.read` and :meth:`~io.TextIOBase.readline` methods, or a string. If no argument is given, input will be taken from ``sys.stdin``. The second optional argument is a filename string, which sets the initial @@ -87,8 +87,19 @@ The :mod:`shlex` module defines the following class: when *posix* is not true (default), the :class:`~shlex.shlex` instance will operate in compatibility mode. When operating in POSIX mode, :class:`~shlex.shlex` will try to be as close as possible to the POSIX shell - parsing rules. - + parsing rules. The *punctuation_chars* argument provides a way to make the + behaviour even closer to how real shells parse. This can take a number of + values: the default value, ``False``, preserves the behaviour seen under + Python 3.5 and earlier. If set to ``True``, then parsing of the characters + ``();<>|&`` is changed: any run of these characters (considered punctuation + characters) is returned as a single token. If set to a non-empty string of + characters, those characters will be used as the punctuation characters. Any + characters in the :attr:`wordchars` attribute that appear in + *punctuation_chars* will be removed from :attr:`wordchars`. See + :ref:`improved-shell-compatibility` for more information. + + .. versionchanged:: 3.6 + The *punctuation_chars* parameter was added. .. seealso:: @@ -191,7 +202,13 @@ variables which either control lexical analysis or can be used for debugging: .. attribute:: shlex.wordchars The string of characters that will accumulate into multi-character tokens. By - default, includes all ASCII alphanumerics and underscore. + default, includes all ASCII alphanumerics and underscore. In POSIX mode, the + accented characters in the Latin-1 set are also included. If + :attr:`punctuation_chars` is not empty, the characters ``~-./*?=``, which can + appear in filename specifications and command line parameters, will also be + included in this attribute, and any characters which appear in + ``punctuation_chars`` will be removed from ``wordchars`` if they are present + there. .. attribute:: shlex.whitespace @@ -222,9 +239,13 @@ variables which either control lexical analysis or can be used for debugging: .. attribute:: shlex.whitespace_split - If ``True``, tokens will only be split in whitespaces. This is useful, for + If ``True``, tokens will only be split in whitespaces. This is useful, for example, for parsing command lines with :class:`~shlex.shlex`, getting - tokens in a similar way to shell arguments. + tokens in a similar way to shell arguments. If this attribute is ``True``, + :attr:`punctuation_chars` will have no effect, and splitting will happen + only on whitespaces. When using :attr:`punctuation_chars`, which is + intended to provide parsing closer to that implemented by shells, it is + advisable to leave ``whitespace_split`` as ``False`` (the default value). .. attribute:: shlex.infile @@ -245,10 +266,9 @@ variables which either control lexical analysis or can be used for debugging: This attribute is ``None`` by default. If you assign a string to it, that string will be recognized as a lexical-level inclusion request similar to the ``source`` keyword in various shells. That is, the immediately following token - will be opened as a filename and input will - be taken from that stream until EOF, at which - point the :meth:`~io.IOBase.close` method of that stream will be called and - the input source will again become the original input stream. Source + will be opened as a filename and input will be taken from that stream until + EOF, at which point the :meth:`~io.IOBase.close` method of that stream will be + called and the input source will again become the original input stream. Source requests may be stacked any number of levels deep. @@ -275,6 +295,16 @@ variables which either control lexical analysis or can be used for debugging: (``''``), in non-POSIX mode, and to ``None`` in POSIX mode. +.. attribute:: shlex.punctuation_chars + + Characters that will be considered punctuation. Runs of punctuation + characters will be returned as a single token. However, note that no + semantic validity checking will be performed: for example, '>>>' could be + returned as a token, even though it may not be recognised as such by shells. + + .. versionadded:: 3.6 + + .. _shlex-parsing-rules: Parsing Rules @@ -327,3 +357,62 @@ following parsing rules. * EOF is signaled with a :const:`None` value; * Quoted empty strings (``''``) are allowed. + +.. _improved-shell-compatibility: + +Improved Compatibility with Shells +---------------------------------- + +.. versionadded:: 3.6 + +The :class:`shlex` class provides compatibility with the parsing performed by +common Unix shells like ``bash``, ``dash``, and ``sh``. To take advantage of +this compatibility, specify the ``punctuation_chars`` argument in the +constructor. This defaults to ``False``, which preserves pre-3.6 behaviour. +However, if it is set to ``True``, then parsing of the characters ``();<>|&`` +is changed: any run of these characters is returned as a single token. While +this is short of a full parser for shells (which would be out of scope for the +standard library, given the multiplicity of shells out there), it does allow +you to perform processing of command lines more easily than you could +otherwise. To illustrate, you can see the difference in the following snippet:: + + import shlex + + for punct in (False, True): + if punct: + message = 'Old' + else: + message = 'New' + text = "a && b; c && d || e; f >'abc'; (def \"ghi\")" + s = shlex.shlex(text, punctuation_chars=punct) + print('%s: %s' % (message, list(s))) + +which prints out:: + + Old: ['a', '&', '&', 'b', ';', 'c', '&', '&', 'd', '|', '|', 'e', ';', 'f', '>', "'abc'", ';', '(', 'def', '"ghi"', ')'] + New: ['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', "'abc'", ';', '(', 'def', '"ghi"', ')'] + +Of course, tokens will be returned which are not valid for shells, and you'll +need to implement your own error checks on the returned tokens. + +Instead of passing ``True`` as the value for the punctuation_chars parameter, +you can pass a string with specific characters, which will be used to determine +which characters constitute punctuation. For example:: + + >>> import shlex + >>> s = shlex.shlex("a && b || c", punctuation_chars="|") + >>> list(s) + ['a', '&', '&', 'b', '||', 'c'] + +.. note:: When ``punctuation_chars`` is specified, the :attr:`~shlex.wordchars` + attribute is augmented with the characters ``~-./*?=``. That is because these + characters can appear in file names (including wildcards) and command-line + arguments (e.g. ``--color=auto``). Hence:: + + >>> import shlex + >>> s = shlex.shlex('~/a && b-c --color=auto || d *.py?', + ... punctuation_chars=True) + >>> list(s) + ['~/a', '&&', 'b-c', '--color=auto', '||', 'd', '*.py?'] + + diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index a1cf241327..fefd6abd72 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -425,7 +425,7 @@ Another example that uses the *ignore* argument to add a logging call:: import logging def _logpath(path, names): - logging.info('Working in %s' % path) + logging.info('Working in %s', path) return [] # nothing will be ignored copytree(source, destination, ignore=_logpath) diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 0a73f5abc6..43daf790b7 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -52,7 +52,8 @@ searched for site-packages; otherwise they won't. A path configuration file is a file whose name has the form :file:`{name}.pth` and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to ``sys.path``. Non-existing items -are never added to ``sys.path``. No item is added to ``sys.path`` more than +are never added to ``sys.path``, and no check is made that the item refers to a +directory rather than a file. No item is added to ``sys.path`` more than once. Blank lines and lines beginning with ``#`` are skipped. Lines starting with ``import`` (followed by space or tab) are executed. diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index a096de006e..e383201aab 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -29,7 +29,7 @@ SMTPServer Objects .. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432,\ - map=None, enable_SMTPUTF8=False, decode_data=True) + map=None, enable_SMTPUTF8=False, decode_data=False) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. Both @@ -46,20 +46,19 @@ SMTPServer Objects global socket map is used. *enable_SMTPUTF8* determines whether the ``SMTPUTF8`` extension (as defined - in :RFC:`6531`) should be enabled. The default is ``False``. If set to - ``True``, *decode_data* must be ``False`` (otherwise an error is raised). + in :RFC:`6531`) should be enabled. The default is ``False``. When ``True``, ``SMTPUTF8`` is accepted as a parameter to the ``MAIL`` command and when present is passed to :meth:`process_message` in the - ``kwargs['mail_options']`` list. + ``kwargs['mail_options']`` list. *decode_data* and *enable_SMTPUTF8* + cannot be set to ``True`` at the same time. *decode_data* specifies whether the data portion of the SMTP transaction - should be decoded using UTF-8. The default is ``True`` for backward - compatibility reasons, but will change to ``False`` in Python 3.6; specify - the keyword value explicitly to avoid the :exc:`DeprecationWarning`. When - *decode_data* is set to ``False`` the server advertises the ``8BITMIME`` + should be decoded using UTF-8. When *decode_data* is ``False`` (the + default), the server advertises the ``8BITMIME`` extension (:rfc:`6152`), accepts the ``BODY=8BITMIME`` parameter to the ``MAIL`` command, and when present passes it to :meth:`process_message` - in the ``kwargs['mail_options']`` list. + in the ``kwargs['mail_options']`` list. *decode_data* and *enable_SMTPUTF8* + cannot be set to ``True`` at the same time. .. method:: process_message(peer, mailfrom, rcpttos, data, **kwargs) @@ -76,9 +75,8 @@ SMTPServer Objects will be a bytes object. *kwargs* is a dictionary containing additional information. It is empty - unless at least one of ``decode_data=False`` or ``enable_SMTPUTF8=True`` - was given as an init parameter, in which case it contains the following - keys: + if ``decode_data=True`` was given as an init argument, otherwise + it contains the following keys: *mail_options*: a list of all received parameters to the ``MAIL`` @@ -109,9 +107,12 @@ SMTPServer Objects *localaddr* and *remoteaddr* may now contain IPv6 addresses. .. versionadded:: 3.5 - the *decode_data* and *enable_SMTPUTF8* constructor arguments, and the - *kwargs* argument to :meth:`process_message` when one or more of these is - specified. + The *decode_data* and *enable_SMTPUTF8* constructor parameters, and the + *kwargs* parameter to :meth:`process_message` when *decode_data* is + ``False``. + + .. versionchanged:: 3.6 + *decode_data* is now ``False`` by default. DebuggingServer Objects @@ -151,7 +152,7 @@ SMTPChannel Objects ------------------- .. class:: SMTPChannel(server, conn, addr, data_size_limit=33554432,\ - map=None, enable_SMTPUTF8=False, decode_data=True) + map=None, enable_SMTPUTF8=False, decode_data=False) Create a new :class:`SMTPChannel` object which manages the communication between the server and a single SMTP client. @@ -163,22 +164,25 @@ SMTPChannel Objects limit. *enable_SMTPUTF8* determines whether the ``SMTPUTF8`` extension (as defined - in :RFC:`6531`) should be enabled. The default is ``False``. A - :exc:`ValueError` is raised if both *enable_SMTPUTF8* and *decode_data* are - set to ``True`` at the same time. + in :RFC:`6531`) should be enabled. The default is ``False``. + *decode_data* and *enable_SMTPUTF8* cannot be set to ``True`` at the same + time. A dictionary can be specified in *map* to avoid using a global socket map. *decode_data* specifies whether the data portion of the SMTP transaction - should be decoded using UTF-8. The default is ``True`` for backward - compatibility reasons, but will change to ``False`` in Python 3.6. Specify - the keyword value explicitly to avoid the :exc:`DeprecationWarning`. + should be decoded using UTF-8. The default is ``False``. + *decode_data* and *enable_SMTPUTF8* cannot be set to ``True`` at the same + time. To use a custom SMTPChannel implementation you need to override the :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. .. versionchanged:: 3.5 - the *decode_data* and *enable_SMTPUTF8* arguments were added. + The *decode_data* and *enable_SMTPUTF8* parameters were added. + + .. versionchanged:: 3.6 + *decode_data* is now ``False`` by default. The :class:`SMTPChannel` has the following instance variables: diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 8b98ccf03c..86e769e6a1 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -95,6 +95,14 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see :data:`ssl.HAS_SNI`). + .. deprecated:: 3.6 + + *keyfile* and *certfile* are deprecated in favor of *context*. + Please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let + :func:`ssl.create_default_context` select the system's trusted CA + certificates for you. + + .. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, source_address=None) The LMTP protocol, which is very similar to ESMTP, is heavily based on the diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 79c7609e54..9671857fc3 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -133,6 +133,22 @@ created. Socket addresses are represented as follows: string format. (ex. ``b'12:23:34:45:56:67'``) This protocol is not supported under FreeBSD. +- :const:`AF_ALG` is a Linux-only socket based interface to Kernel + cryptography. An algorithm socket is configured with a tuple of two to four + elements ``(type, name [, feat [, mask]])``, where: + + - *type* is the algorithm type as string, e.g. ``aead``, ``hash``, + ``skcipher`` or ``rng``. + + - *name* is the algorithm name and operation mode as string, e.g. + ``sha256``, ``hmac(sha256)``, ``cbc(aes)`` or ``drbg_nopr_ctr_aes256``. + + - *feat* and *mask* are unsigned 32bit integers. + + Availability Linux 2.6.38, some algorithm types require more recent Kernels. + + .. versionadded:: 3.6 + - Certain other address families (:const:`AF_PACKET`, :const:`AF_CAN`) support specific representations. @@ -283,6 +299,10 @@ Constants in the Unix header files are defined; for a few symbols, default values are provided. + .. versionchanged:: 3.6 + ``SO_DOMAIN``, ``SO_PROTOCOL``, ``SO_PEERSEC``, ``SO_PASSSEC`` + were added. + .. data:: AF_CAN PF_CAN SOL_CAN_* @@ -331,18 +351,33 @@ Constants .. versionadded:: 3.3 -.. data:: SIO_* +.. data:: SIO_RCVALL + SIO_KEEPALIVE_VALS + SIO_LOOPBACK_FAST_PATH RCVALL_* Constants for Windows' WSAIoctl(). The constants are used as arguments to the :meth:`~socket.socket.ioctl` method of socket objects. + .. versionchanged:: 3.6 + ``SIO_LOOPBACK_FAST_PATH`` was added. + .. data:: TIPC_* TIPC related constants, matching the ones exported by the C socket API. See the TIPC documentation for more information. +.. data:: AF_ALG + SOL_ALG + ALG_* + + Constants for Linux Kernel cryptography. + + Availability: Linux >= 2.6.38. + + .. versionadded:: 3.6 + .. data:: AF_LINK Availability: BSD, OSX. @@ -874,6 +909,10 @@ to sockets. it is recommended to :meth:`close` them explicitly, or to use a :keyword:`with` statement around them. + .. versionchanged:: 3.6 + :exc:`OSError` is now raised if an error occurs when the underlying + :c:func:`close` call is made. + .. note:: :meth:`close()` releases the resource associated with a connection but @@ -994,6 +1033,12 @@ to sockets. On other platforms, the generic :func:`fcntl.fcntl` and :func:`fcntl.ioctl` functions may be used; they accept a socket object as their first argument. + Currently only the following control codes are supported: + ``SIO_RCVALL``, ``SIO_KEEPALIVE_VALS``, and ``SIO_LOOPBACK_FAST_PATH``. + + .. versionchanged:: 3.6 + ``SIO_LOOPBACK_FAST_PATH`` was added. + .. method:: socket.listen([backlog]) Enable a server to accept connections. If *backlog* is specified, it must @@ -1277,6 +1322,15 @@ to sockets. an exception, the method now retries the system call instead of raising an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). +.. method:: socket.sendmsg_afalg([msg], *, op[, iv[, assoclen[, flags]]]) + + Specialized version of :meth:`~socket.sendmsg` for :const:`AF_ALG` socket. + Set mode, IV, AEAD associated data length and flags for :const:`AF_ALG` socket. + + Availability: Linux >= 2.6.38 + + .. versionadded:: 3.6 + .. method:: socket.sendfile(file, offset=0, count=None) Send a file until EOF is reached by using high-performance @@ -1325,21 +1379,29 @@ to sockets. For further information, please consult the :ref:`notes on socket timeouts <socket-timeouts>`. -.. method:: socket.setsockopt(level, optname, value) +.. method:: socket.setsockopt(level, optname, value: int) +.. method:: socket.setsockopt(level, optname, value: buffer) +.. method:: socket.setsockopt(level, optname, None, optlen: int) .. index:: module: struct Set the value of the given socket option (see the Unix manual page :manpage:`setsockopt(2)`). The needed symbolic constants are defined in the - :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or - a :term:`bytes-like object` representing a buffer. In the latter case it is - up to the caller to - ensure that the bytestring contains the proper bits (see the optional built-in - module :mod:`struct` for a way to encode C structures as bytestrings). + :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer, + ``None`` or a :term:`bytes-like object` representing a buffer. In the later + case it is up to the caller to ensure that the bytestring contains the + proper bits (see the optional built-in module :mod:`struct` for a way to + encode C structures as bytestrings). When value is set to ``None``, + optlen argument is required. It's equivalent to call setsockopt C + function with optval=NULL and optlen=optlen. + .. versionchanged:: 3.5 Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.6 + setsockopt(level, optname, None, optlen: int) form added. + .. method:: socket.shutdown(how) diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 087f4e0d45..218a31c848 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -52,11 +52,12 @@ handler class by subclassing the :class:`BaseRequestHandler` class and overriding its :meth:`~BaseRequestHandler.handle` method; this method will process incoming requests. Second, you must instantiate one of the server classes, passing it -the server's address and the request handler class. Then call the +the server's address and the request handler class. It is recommended to use +the server in a :keyword:`with` statement. Then call the :meth:`~BaseServer.handle_request` or :meth:`~BaseServer.serve_forever` method of the server object to process one or many requests. Finally, call :meth:`~BaseServer.server_close` -to close the socket. +to close the socket (unless you used a :keyword:`with` statement). When inheriting from :class:`ThreadingMixIn` for threaded connection behavior, you should explicitly declare how you want your threads to behave on an abrupt @@ -111,6 +112,8 @@ server classes. :class:`UDPServer`. Setting the various attributes also changes the behavior of the underlying server mechanism. + :class:`ForkingMixIn` and the Forking classes mentioned below are + only available on POSIX platforms that support :func:`~os.fork`. .. class:: ForkingTCPServer ForkingUDPServer @@ -304,7 +307,11 @@ Server Objects This function is called if the :meth:`~BaseRequestHandler.handle` method of a :attr:`RequestHandlerClass` instance raises an exception. The default action is to print the traceback to - standard output and continue handling further requests. + standard error and continue handling further requests. + + .. versionchanged:: 3.6 + Now only called for exceptions derived from the :exc:`Exception` + class. .. method:: handle_timeout() @@ -349,6 +356,11 @@ Server Objects default implementation always returns :const:`True`. + .. versionchanged:: 3.6 + Support for the :term:`context manager` protocol was added. Exiting the + context manager is equivalent to calling :meth:`server_close`. + + Request Handler Objects ----------------------- @@ -397,6 +409,15 @@ Request Handler Objects read or written, respectively, to get the request data or return data to the client. + The :attr:`rfile` attributes of both classes support the + :class:`io.BufferedIOBase` readable interface, and + :attr:`DatagramRequestHandler.wfile` supports the + :class:`io.BufferedIOBase` writable interface. + + .. versionchanged:: 3.6 + :attr:`StreamRequestHandler.wfile` also supports the + :class:`io.BufferedIOBase` writable interface. + Examples -------- @@ -429,11 +450,10 @@ This is the server side:: HOST, PORT = "localhost", 9999 # Create the server, binding to localhost on port 9999 - server = socketserver.TCPServer((HOST, PORT), MyTCPHandler) - - # Activate the server; this will keep running until you - # interrupt the program with Ctrl-C - server.serve_forever() + with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server: + # Activate the server; this will keep running until you + # interrupt the program with Ctrl-C + server.serve_forever() An alternative request handler class that makes use of streams (file-like objects that simplify communication by providing the standard file interface):: @@ -525,8 +545,8 @@ This is the server side:: if __name__ == "__main__": HOST, PORT = "localhost", 9999 - server = socketserver.UDPServer((HOST, PORT), MyUDPHandler) - server.serve_forever() + with socketserver.UDPServer((HOST, PORT), MyUDPHandler) as server: + server.serve_forever() This is the client side:: @@ -585,22 +605,22 @@ An example for the :class:`ThreadingMixIn` class:: HOST, PORT = "localhost", 0 server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler) - ip, port = server.server_address + with server: + ip, port = server.server_address - # Start a thread with the server -- that thread will then start one - # more thread for each request - server_thread = threading.Thread(target=server.serve_forever) - # Exit the server thread when the main thread terminates - server_thread.daemon = True - server_thread.start() - print("Server loop running in thread:", server_thread.name) + # Start a thread with the server -- that thread will then start one + # more thread for each request + server_thread = threading.Thread(target=server.serve_forever) + # Exit the server thread when the main thread terminates + server_thread.daemon = True + server_thread.start() + print("Server loop running in thread:", server_thread.name) - client(ip, port, "Hello World 1") - client(ip, port, "Hello World 2") - client(ip, port, "Hello World 3") + client(ip, port, "Hello World 1") + client(ip, port, "Hello World 2") + client(ip, port, "Hello World 3") - server.shutdown() - server.server_close() + server.shutdown() The output of the example should look something like this: @@ -616,3 +636,5 @@ The output of the example should look something like this: The :class:`ForkingMixIn` class is used in the same way, except that the server will spawn a new process for each request. +Available only on POSIX platforms that support :func:`~os.fork`. + diff --git a/Doc/library/spwd.rst b/Doc/library/spwd.rst index fd3c9adee4..c6cad2a3c3 100644 --- a/Doc/library/spwd.rst +++ b/Doc/library/spwd.rst @@ -55,6 +55,9 @@ The following functions are defined: Return the shadow password database entry for the given user name. + .. versionchanged:: 3.6 + Raises a :exc:`PermissionError` instead of :exc:`KeyError` if the user + doesn't have privileges. .. function:: getspall() diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 84a478352f..5635577da0 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -629,9 +629,16 @@ Cursor Objects .. attribute:: lastrowid This read-only attribute provides the rowid of the last modified row. It is - only set if you issued an ``INSERT`` statement using the :meth:`execute` - method. For operations other than ``INSERT`` or when :meth:`executemany` is - called, :attr:`lastrowid` is set to :const:`None`. + only set if you issued an ``INSERT`` or a ``REPLACE`` statement using the + :meth:`execute` method. For operations other than ``INSERT`` or + ``REPLACE`` or when :meth:`executemany` is called, :attr:`lastrowid` is + set to :const:`None`. + + If the ``INSERT`` or ``REPLACE`` statement failed to insert the previous + successful rowid is returned. + + .. versionchanged:: 3.6 + Added support for the ``REPLACE`` statement. .. attribute:: description @@ -918,9 +925,7 @@ Controlling Transactions By default, the :mod:`sqlite3` module opens transactions implicitly before a Data Modification Language (DML) statement (i.e. -``INSERT``/``UPDATE``/``DELETE``/``REPLACE``), and commits transactions -implicitly before a non-DML, non-query statement (i. e. -anything other than ``SELECT`` or the aforementioned). +``INSERT``/``UPDATE``/``DELETE``/``REPLACE``). So if you are within a transaction and issue a command like ``CREATE TABLE ...``, ``VACUUM``, ``PRAGMA``, the :mod:`sqlite3` module will commit implicitly @@ -940,6 +945,9 @@ Otherwise leave it at its default, which will result in a plain "BEGIN" statement, or set it to one of SQLite's supported isolation levels: "DEFERRED", "IMMEDIATE" or "EXCLUSIVE". +.. versionchanged:: 3.6 + :mod:`sqlite3` used to implicitly commit an open transaction before DDL + statements. This is no longer the case. Using :mod:`sqlite3` efficiently diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index a2f008346b..eee4849331 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -49,6 +49,12 @@ For more sophisticated applications, the :class:`ssl.SSLContext` class helps manage settings and certificates, which can then be inherited by SSL sockets created through the :meth:`SSLContext.wrap_socket` method. +.. versionchanged:: 3.6 + + OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported. + In the future the ssl module will require at least OpenSSL 1.0.2 or + 1.1.0. + Functions, Constants, and Exceptions ------------------------------------ @@ -186,16 +192,20 @@ instead. .. table:: - ======================== ========= ========= ========== ========= =========== =========== - *client* / **server** **SSLv2** **SSLv3** **TLS** **TLSv1** **TLSv1.1** **TLSv1.2** - ------------------------ --------- --------- ---------- --------- ----------- ----------- - *SSLv2* yes no yes no no no - *SSLv3* no yes yes no no no - *TLS* (*SSLv23*) no yes yes yes yes yes - *TLSv1* no no yes yes no no - *TLSv1.1* no no yes no yes no - *TLSv1.2* no no yes no no yes - ======================== ========= ========= ========== ========= =========== =========== + ======================== ============ ============ ============= ========= =========== =========== + *client* / **server** **SSLv2** **SSLv3** **TLS** **TLSv1** **TLSv1.1** **TLSv1.2** + ------------------------ ------------ ------------ ------------- --------- ----------- ----------- + *SSLv2* yes no no [1]_ no no no + *SSLv3* no yes no [2]_ no no no + *TLS* (*SSLv23*) no [1]_ no [2]_ yes yes yes yes + *TLSv1* no no yes yes no no + *TLSv1.1* no no yes no yes no + *TLSv1.2* no no yes no no yes + ======================== ============ ============ ============= ========= =========== =========== + + .. rubric:: Footnotes + .. [1] :class:`SSLContext` disables SSLv2 with :data:`OP_NO_SSLv2` by default. + .. [2] :class:`SSLContext` disables SSLv3 with :data:`OP_NO_SSLv3` by default. .. note:: @@ -224,7 +234,6 @@ instead. .. versionchanged:: 3.2 New optional argument *ciphers*. - Context creation ^^^^^^^^^^^^^^^^ @@ -279,7 +288,7 @@ purposes. RC4 was dropped from the default cipher string. - .. versionchanged:: 3.5.3 + .. versionchanged:: 3.6 ChaCha20/Poly1305 was added to the default cipher string. @@ -322,7 +331,7 @@ Random generation .. versionadded:: 3.3 - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 OpenSSL has deprecated :func:`ssl.RAND_pseudo_bytes`, use :func:`ssl.RAND_bytes` instead. @@ -509,6 +518,10 @@ Certificate handling Constants ^^^^^^^^^ + All constants are now :class:`enum.IntEnum` or :class:`enum.IntFlag` collections. + + .. versionadded:: 3.6 + .. data:: CERT_NONE Possible value for :attr:`SSLContext.verify_mode`, or the ``cert_reqs`` @@ -542,6 +555,12 @@ Constants be passed, either to :meth:`SSLContext.load_verify_locations` or as a value of the ``ca_certs`` parameter to :func:`wrap_socket`. +.. class:: VerifyMode + + :class:`enum.IntEnum` collection of CERT_* constants. + + .. versionadded:: 3.6 + .. data:: VERIFY_DEFAULT Possible value for :attr:`SSLContext.verify_flags`. In this mode, certificate @@ -582,18 +601,40 @@ Constants .. versionadded:: 3.4.4 +.. class:: VerifyFlags + + :class:`enum.IntFlag` collection of VERIFY_* constants. + + .. versionadded:: 3.6 + .. data:: PROTOCOL_TLS Selects the highest protocol version that both the client and server support. Despite the name, this option can select "TLS" protocols as well as "SSL". - .. versionadded:: 3.5.3 + .. versionadded:: 3.6 + +.. data:: PROTOCOL_TLS_CLIENT + + Auto-negotiate the the highest protocol version like :data:`PROTOCOL_SSLv23`, + but only support client-side :class:`SSLSocket` connections. The protocol + enables :data:`CERT_REQUIRED` and :attr:`~SSLContext.check_hostname` by + default. + + .. versionadded:: 3.6 + +.. data:: PROTOCOL_TLS_SERVER + + Auto-negotiate the the highest protocol version like :data:`PROTOCOL_SSLv23`, + but only support server-side :class:`SSLSocket` connections. + + .. versionadded:: 3.6 .. data:: PROTOCOL_SSLv23 Alias for data:`PROTOCOL_TLS`. - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 Use data:`PROTOCOL_TLS` instead. @@ -608,7 +649,7 @@ Constants SSL version 2 is insecure. Its use is highly discouraged. - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 OpenSSL has removed support for SSLv2. @@ -623,7 +664,7 @@ Constants SSL version 3 is insecure. Its use is highly discouraged. - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 OpenSSL has deprecated all version specific protocols. Use the default protocol data:`PROTOCOL_TLS` with flags like data:`OP_NO_SSLv3` instead. @@ -632,7 +673,7 @@ Constants Selects TLS version 1.0 as the channel encryption protocol. - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 OpenSSL has deprecated all version specific protocols. Use the default protocol data:`PROTOCOL_TLS` with flags like data:`OP_NO_SSLv3` instead. @@ -644,7 +685,7 @@ Constants .. versionadded:: 3.4 - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 OpenSSL has deprecated all version specific protocols. Use the default protocol data:`PROTOCOL_TLS` with flags like data:`OP_NO_SSLv3` instead. @@ -657,7 +698,7 @@ Constants .. versionadded:: 3.4 - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 OpenSSL has deprecated all version specific protocols. Use the default protocol data:`PROTOCOL_TLS` with flags like data:`OP_NO_SSLv3` instead. @@ -678,7 +719,7 @@ Constants .. versionadded:: 3.2 - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 SSLv2 is deprecated @@ -691,7 +732,7 @@ Constants .. versionadded:: 3.2 - .. deprecated:: 3.5.3 + .. deprecated:: 3.6 SSLv3 is deprecated @@ -751,6 +792,16 @@ Constants .. versionadded:: 3.3 +.. class:: Options + + :class:`enum.IntFlag` collection of OP_* constants. + +.. data:: OP_NO_TICKET + + Prevent client side from requesting a session ticket. + + .. versionadded:: 3.6 + .. data:: HAS_ALPN Whether the OpenSSL library has built-in support for the *Application-Layer @@ -833,6 +884,12 @@ Constants .. versionadded:: 3.4 +.. class:: AlertDescription + + :class:`enum.IntEnum` collection of ALERT_DESCRIPTION_* constants. + + .. versionadded:: 3.6 + .. data:: Purpose.SERVER_AUTH Option for :func:`create_default_context` and @@ -851,6 +908,12 @@ Constants .. versionadded:: 3.4 +.. class:: SSLErrorNumber + + :class:`enum.IntEnum` collection of SSL_ERROR_* constants. + + .. versionadded:: 3.6 + SSL Sockets ----------- @@ -885,7 +948,7 @@ SSL Sockets :ref:`notes on non-blocking sockets <ssl-nonblocking>`. Usually, :class:`SSLSocket` are not created directly, but using the - :func:`wrap_socket` function or the :meth:`SSLContext.wrap_socket` method. + the :meth:`SSLContext.wrap_socket` method. .. versionchanged:: 3.5 The :meth:`sendfile` method was added. @@ -895,6 +958,10 @@ SSL Sockets are received or sent. The socket timeout is now to maximum total duration of the shutdown. + .. deprecated:: 3.6 + It is deprecated to create a :class:`SSLSocket` instance directly, use + :meth:`SSLContext.wrap_socket` to wrap a socket. + SSL sockets also have the following additional methods and attributes: @@ -915,6 +982,9 @@ SSL sockets also have the following additional methods and attributes: The socket timeout is now to maximum total duration to read up to *len* bytes. + .. deprecated:: 3.6 + Use :meth:`~SSLSocket.recv` instead of :meth:`~SSLSocket.read`. + .. method:: SSLSocket.write(buf) Write *buf* to the SSL socket and return the number of bytes written. The @@ -930,6 +1000,9 @@ SSL sockets also have the following additional methods and attributes: The socket timeout is no more reset each time bytes are received or sent. The socket timeout is now to maximum total duration to write *buf*. + .. deprecated:: 3.6 + Use :meth:`~SSLSocket.send` instead of :meth:`~SSLSocket.write`. + .. note:: The :meth:`~SSLSocket.read` and :meth:`~SSLSocket.write` methods are the @@ -1127,6 +1200,19 @@ SSL sockets also have the following additional methods and attributes: .. versionadded:: 3.2 +.. attribute:: SSLSocket.session + + The :class:`SSLSession` for this SSL connection. The session is available + for client and server side sockets after the TLS handshake has been + performed. For client sockets the session can be set before + :meth:`~SSLSocket.do_handshake` has been called to reuse a session. + + .. versionadded:: 3.6 + +.. attribute:: SSLSocket.session_reused + + .. versionadded:: 3.6 + SSL Contexts ------------ @@ -1149,9 +1235,16 @@ to speed up repeated connections from the same clients. :func:`create_default_context` lets the :mod:`ssl` module choose security settings for a given purpose. - .. versionchanged:: 3.5.3 + .. versionchanged:: 3.6 - :data:`PROTOCOL_TLS` is the default value. + The context is created with secure default values. The options + :data:`OP_NO_COMPRESSION`, :data:`OP_CIPHER_SERVER_PREFERENCE`, + :data:`OP_SINGLE_DH_USE`, :data:`OP_SINGLE_ECDH_USE`, + :data:`OP_NO_SSLv2` (except for :data:`PROTOCOL_SSLv2`), + and :data:`OP_NO_SSLv3` (except for :data:`PROTOCOL_SSLv3`) are + set by default. The initial cipher suite list contains only ``HIGH`` + ciphers, no ``NULL`` ciphers and no ``MD5`` ciphers (except for + :data:`PROTOCOL_SSLv2`). :class:`SSLContext` objects have the following methods and attributes: @@ -1259,6 +1352,62 @@ to speed up repeated connections from the same clients. .. versionadded:: 3.4 +.. method:: SSLContext.get_ciphers() + + Get a list of enabled ciphers. The list is in order of cipher priority. + See :meth:`SSLContext.set_ciphers`. + + Example:: + + >>> ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + >>> ctx.set_ciphers('ECDHE+AESGCM:!ECDSA') + >>> ctx.get_ciphers() # OpenSSL 1.0.x + [{'alg_bits': 256, + 'description': 'ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA ' + 'Enc=AESGCM(256) Mac=AEAD', + 'id': 50380848, + 'name': 'ECDHE-RSA-AES256-GCM-SHA384', + 'protocol': 'TLSv1/SSLv3', + 'strength_bits': 256}, + {'alg_bits': 128, + 'description': 'ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA ' + 'Enc=AESGCM(128) Mac=AEAD', + 'id': 50380847, + 'name': 'ECDHE-RSA-AES128-GCM-SHA256', + 'protocol': 'TLSv1/SSLv3', + 'strength_bits': 128}] + + On OpenSSL 1.1 and newer the cipher dict contains additional fields:: + >>> ctx.get_ciphers() # OpenSSL 1.1+ + [{'aead': True, + 'alg_bits': 256, + 'auth': 'auth-rsa', + 'description': 'ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA ' + 'Enc=AESGCM(256) Mac=AEAD', + 'digest': None, + 'id': 50380848, + 'kea': 'kx-ecdhe', + 'name': 'ECDHE-RSA-AES256-GCM-SHA384', + 'protocol': 'TLSv1.2', + 'strength_bits': 256, + 'symmetric': 'aes-256-gcm'}, + {'aead': True, + 'alg_bits': 128, + 'auth': 'auth-rsa', + 'description': 'ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA ' + 'Enc=AESGCM(128) Mac=AEAD', + 'digest': None, + 'id': 50380847, + 'kea': 'kx-ecdhe', + 'name': 'ECDHE-RSA-AES128-GCM-SHA256', + 'protocol': 'TLSv1.2', + 'strength_bits': 128, + 'symmetric': 'aes-128-gcm'}] + + Availability: OpenSSL 1.0.2+ + + .. versionadded:: 3.6 + .. method:: SSLContext.set_default_verify_paths() Load a set of default "certification authority" (CA) certificates from @@ -1397,7 +1546,7 @@ to speed up repeated connections from the same clients. .. method:: SSLContext.wrap_socket(sock, server_side=False, \ do_handshake_on_connect=True, suppress_ragged_eofs=True, \ - server_hostname=None) + server_hostname=None, session=None) Wrap an existing Python socket *sock* and return an :class:`SSLSocket` object. *sock* must be a :data:`~socket.SOCK_STREAM` socket; other socket @@ -1414,19 +1563,27 @@ to speed up repeated connections from the same clients. quite similarly to HTTP virtual hosts. Specifying *server_hostname* will raise a :exc:`ValueError` if *server_side* is true. + *session*, see :attr:`~SSLSocket.session`. + .. versionchanged:: 3.5 Always allow a server_hostname to be passed, even if OpenSSL does not have SNI. + .. versionchanged:: 3.6 + *session* argument was added. + .. method:: SSLContext.wrap_bio(incoming, outgoing, server_side=False, \ - server_hostname=None) + server_hostname=None, session=None) Create a new :class:`SSLObject` instance by wrapping the BIO objects *incoming* and *outgoing*. The SSL routines will read input data from the incoming BIO and write data to the outgoing BIO. - The *server_side* and *server_hostname* parameters have the same meaning as - in :meth:`SSLContext.wrap_socket`. + The *server_side*, *server_hostname* and *session* parameters have the + same meaning as in :meth:`SSLContext.wrap_socket`. + + .. versionchanged:: 3.6 + *session* argument was added. .. method:: SSLContext.session_stats() @@ -1478,6 +1635,12 @@ to speed up repeated connections from the same clients. to set options, not to clear them. Attempting to clear an option (by resetting the corresponding bits) will raise a ``ValueError``. + .. versionchanged:: 3.6 + :attr:`SSLContext.options` returns :class:`Options` flags: + + >>> ssl.create_default_context().options + <Options.OP_ALL|OP_NO_SSLv3|OP_NO_SSLv2|OP_NO_COMPRESSION: 2197947391> + .. attribute:: SSLContext.protocol The protocol version chosen when constructing the context. This attribute @@ -1492,12 +1655,23 @@ to speed up repeated connections from the same clients. .. versionadded:: 3.4 + .. versionchanged:: 3.6 + :attr:`SSLContext.verify_flags` returns :class:`VerifyFlags` flags: + + >>> ssl.create_default_context().verify_flags + <VerifyFlags.VERIFY_X509_TRUSTED_FIRST: 32768> + .. attribute:: SSLContext.verify_mode Whether to try to verify other peers' certificates and how to behave if verification fails. This attribute must be one of :data:`CERT_NONE`, :data:`CERT_OPTIONAL` or :data:`CERT_REQUIRED`. + .. versionchanged:: 3.6 + :attr:`SSLContext.verify_mode` returns :class:`VerifyMode` enum: + + >>> ssl.create_default_context().verify_mode + <VerifyMode.CERT_REQUIRED: 2> .. index:: single: certificates @@ -1916,6 +2090,8 @@ provided. - :attr:`~SSLSocket.context` - :attr:`~SSLSocket.server_side` - :attr:`~SSLSocket.server_hostname` + - :attr:`~SSLSocket.session` + - :attr:`~SSLSocket.session_reused` - :meth:`~SSLSocket.read` - :meth:`~SSLSocket.write` - :meth:`~SSLSocket.getpeercert` @@ -1997,6 +2173,22 @@ purpose. It wraps an OpenSSL memory BIO (Basic IO) object: become true after all data currently in the buffer has been read. +SSL session +----------- + +.. versionadded:: 3.6 + +.. class:: SSLSession + + Session object used by :attr:`~SSLSocket.session`. + + .. attribute:: id + .. attribute:: time + .. attribute:: timeout + .. attribute:: ticket_lifetime_hint + .. attribute:: has_ticket + + .. _ssl-security: Security considerations @@ -2063,18 +2255,20 @@ Protocol versions SSL versions 2 and 3 are considered insecure and are therefore dangerous to use. If you want maximum compatibility between clients and servers, it is -recommended to use :const:`PROTOCOL_TLS` as the protocol version and then -disable SSLv2 and SSLv3 explicitly using the :data:`SSLContext.options` -attribute:: +recommended to use :const:`PROTOCOL_TLS_CLIENT` or +:const:`PROTOCOL_TLS_SERVER` as the protocol version. SSLv2 and SSLv3 are +disabled by default. + + >>> client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + >>> client_context.options |= ssl.OP_NO_TLSv1 + >>> client_context.options |= ssl.OP_NO_TLSv1_1 - context = ssl.SSLContext(ssl.PROTOCOL_TLS) - context.options |= ssl.OP_NO_SSLv2 - context.options |= ssl.OP_NO_SSLv3 - context.options |= ssl.OP_NO_TLSv1 - context.options |= ssl.OP_NO_TLSv1_1 The SSL context created above will only allow TLSv1.2 and later (if -supported by your system) connections. +supported by your system) connections to a server. :const:`PROTOCOL_TLS_CLIENT` +implies certificate validation and hostname checks by default. You have to +load certificates into the context. + Cipher selection '''''''''''''''' @@ -2085,8 +2279,9 @@ enabled when negotiating a SSL session is possible through the ssl module disables certain weak ciphers by default, but you may want to further restrict the cipher choice. Be sure to read OpenSSL's documentation about the `cipher list format <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`_. -If you want to check which ciphers are enabled by a given cipher list, use the -``openssl ciphers`` command on your system. +If you want to check which ciphers are enabled by a given cipher list, use +:meth:`SSLContext.get_ciphers` or the ``openssl ciphers`` command on your +system. Multi-processing ^^^^^^^^^^^^^^^^ diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst index ea3d7dab0f..c020a34ff8 100644 --- a/Doc/library/statistics.rst +++ b/Doc/library/statistics.rst @@ -39,6 +39,7 @@ or sample. ======================= ============================================= :func:`mean` Arithmetic mean ("average") of data. +:func:`harmonic_mean` Harmonic mean of data. :func:`median` Median (middle value) of data. :func:`median_low` Low median of data. :func:`median_high` High median of data. @@ -111,6 +112,38 @@ However, for reading convenience, most of the examples show sorted sequences. ``mean(data)`` is equivalent to calculating the true population mean μ. +.. function:: harmonic_mean(data) + + Return the harmonic mean of *data*, a sequence or iterator of + real-valued numbers. + + The harmonic mean, sometimes called the subcontrary mean, is the + reciprocal of the arithmetic :func:`mean` of the reciprocals of the + data. For example, the harmonic mean of three values *a*, *b* and *c* + will be equivalent to ``3/(1/a + 1/b + 1/c)``. + + The harmonic mean is a type of average, a measure of the central + location of the data. It is often appropriate when averaging quantities + which are rates or ratios, for example speeds. For example: + + Suppose an investor purchases an equal value of shares in each of + three companies, with P/E (price/earning) ratios of 2.5, 3 and 10. + What is the average P/E ratio for the investor's portfolio? + + .. doctest:: + + >>> harmonic_mean([2.5, 3, 10]) # For an equal investment portfolio. + 3.6 + + Using the arithmetic mean would give an average of about 5.167, which + is too high. + + :exc:`StatisticsError` is raised if *data* is empty, or any element + is less than zero. + + .. versionadded:: 3.6 + + .. function:: median(data) Return the median (middle value) of numeric data, using the common "mean of diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 2587ac85b0..19a02ff033 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1363,6 +1363,11 @@ objects that compare equal might have different :attr:`~range.start`, The :attr:`~range.start`, :attr:`~range.stop` and :attr:`~range.step` attributes. +.. seealso:: + + * The `linspace recipe <http://code.activestate.com/recipes/579000/>`_ + shows how to implement a lazy version of range that suitable for floating + point applications. .. index:: single: string; text sequence type @@ -1453,8 +1458,8 @@ multiple fragments. For more information on the ``str`` class and its methods, see :ref:`textseq` and the :ref:`string-methods` section below. To output - formatted strings, see the :ref:`formatstrings` section. In addition, - see the :ref:`stringservices` section. + formatted strings, see the :ref:`f-strings` and :ref:`formatstrings` + sections. In addition, see the :ref:`stringservices` section. .. index:: @@ -2056,8 +2061,8 @@ expression support in the :mod:`re` module). .. index:: single: formatting, string (%) single: interpolation, string (%) - single: string; formatting - single: string; interpolation + single: string; formatting, printf + single: string; interpolation, printf single: printf-style formatting single: sprintf-style formatting single: % formatting @@ -2067,9 +2072,10 @@ expression support in the :mod:`re` module). The formatting operations described here exhibit a variety of quirks that lead to a number of common errors (such as failing to display tuples and - dictionaries correctly). Using the newer :meth:`str.format` interface - helps avoid these errors, and also provides a generally more powerful, - flexible and extensible approach to formatting text. + dictionaries correctly). Using the newer :ref:`formatted + string literals <f-strings>` or the :meth:`str.format` interface + helps avoid these errors. These alternatives also provide more powerful, + flexible and extensible approaches to formatting text. String objects have one unique built-in operation: the ``%`` operator (modulo). This is also known as the string *formatting* or *interpolation* operator. @@ -2630,8 +2636,8 @@ arbitrary binary data. The prefix(es) to search for may be any :term:`bytes-like object`. -.. method:: bytes.translate(table[, delete]) - bytearray.translate(table[, delete]) +.. method:: bytes.translate(table, delete=b'') + bytearray.translate(table, delete=b'') Return a copy of the bytes or bytearray object where all bytes occurring in the optional argument *delete* are removed, and the remaining bytes have @@ -2647,6 +2653,9 @@ arbitrary binary data. >>> b'read this short text'.translate(None, b'aeiou') b'rd ths shrt txt' + .. versionchanged:: 3.6 + *delete* is now supported as a keyword argument. + The following methods on bytes and bytearray objects have default behaviours that assume the use of ASCII compatible binary formats, but can still be used diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 05418a9bb7..a0977b6461 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -188,7 +188,9 @@ Format String Syntax The :meth:`str.format` method and the :class:`Formatter` class share the same syntax for format strings (although in the case of :class:`Formatter`, -subclasses can define their own format string syntax). +subclasses can define their own format string syntax). The syntax is +related to that of :ref:`formatted string literals <f-strings>`, but +there are differences. Format strings contain "replacement fields" surrounded by curly braces ``{}``. Anything that is not contained in braces is considered literal text, which is @@ -283,7 +285,8 @@ Format Specification Mini-Language "Format specifications" are used within replacement fields contained within a format string to define how individual values are presented (see -:ref:`formatstrings`). They can also be passed directly to the built-in +:ref:`formatstrings` and :ref:`f-strings`). +They can also be passed directly to the built-in :func:`format` function. Each formattable type may define how the format specification is to be interpreted. @@ -297,18 +300,20 @@ non-empty format string typically modifies the result. The general form of a *standard format specifier* is: .. productionlist:: sf - format_spec: [[`fill`]`align`][`sign`][#][0][`width`][,][.`precision`][`type`] + format_spec: [[`fill`]`align`][`sign`][#][0][`width`][`grouping_option`][.`precision`][`type`] fill: <any character> align: "<" | ">" | "=" | "^" sign: "+" | "-" | " " width: `integer` + grouping_option: "_" | "," precision: `integer` type: "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%" If a valid *align* value is specified, it can be preceded by a *fill* character that can be any character and defaults to a space if omitted. It is not possible to use a literal curly brace ("``{``" or "``}``") as -the *fill* character when using the :meth:`str.format` +the *fill* character in a :ref:`formatted string literal +<f-strings>` or when using the :meth:`str.format` method. However, it is possible to insert a curly brace with a nested replacement field. This limitation doesn't affect the :func:`format` function. @@ -374,6 +379,16 @@ instead. .. versionchanged:: 3.1 Added the ``','`` option (see also :pep:`378`). +The ``'_'`` option signals the use of an underscore for a thousands +separator for floating point presentation types and for integer +presentation type ``'d'``. For integer presentation types ``'b'``, +``'o'``, ``'x'``, and ``'X'``, underscores will be inserted every 4 +digits. For other presentation types, specifying this option is an +error. + +.. versionchanged:: 3.6 + Added the ``'_'`` option (see also :pep:`515`). + *width* is a decimal integer defining the minimum field width. If not specified, then the field width will be determined by the content. diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst index ae2e38fdc0..7e861fdeaf 100644 --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -216,6 +216,8 @@ platform-dependent. +--------+--------------------------+--------------------+----------------+------------+ | ``N`` | :c:type:`size_t` | integer | | \(4) | +--------+--------------------------+--------------------+----------------+------------+ +| ``e`` | \(7) | float | 2 | \(5) | ++--------+--------------------------+--------------------+----------------+------------+ | ``f`` | :c:type:`float` | float | 4 | \(5) | +--------+--------------------------+--------------------+----------------+------------+ | ``d`` | :c:type:`double` | float | 8 | \(5) | @@ -257,9 +259,10 @@ Notes: fits your application. (5) - For the ``'f'`` and ``'d'`` conversion codes, the packed representation uses - the IEEE 754 binary32 (for ``'f'``) or binary64 (for ``'d'``) format, - regardless of the floating-point format used by the platform. + For the ``'f'``, ``'d'`` and ``'e'`` conversion codes, the packed + representation uses the IEEE 754 binary32, binary64 or binary16 format (for + ``'f'``, ``'d'`` or ``'e'`` respectively), regardless of the floating-point + format used by the platform. (6) The ``'P'`` format character is only available for the native byte ordering @@ -268,6 +271,16 @@ Notes: on the host system. The struct module does not interpret this as native ordering, so the ``'P'`` format is not available. +(7) + The IEEE 754 binary16 "half precision" type was introduced in the 2008 + revision of the `IEEE 754 standard <ieee 754 standard_>`_. It has a sign + bit, a 5-bit exponent and 11-bit precision (with 10 bits explicitly stored), + and can represent numbers between approximately ``6.1e-05`` and ``6.5e+04`` + at full precision. This type is not widely supported by C compilers: on a + typical machine, an unsigned short can be used for storage, but not for math + operations. See the Wikipedia page on the `half-precision floating-point + format <half precision format_>`_ for more information. + A format character may be preceded by an integral repeat count. For example, the format string ``'4h'`` means exactly the same as ``'hhhh'``. @@ -430,3 +443,7 @@ The :mod:`struct` module also defines the following type: The calculated size of the struct (and hence of the bytes object produced by the :meth:`pack` method) corresponding to :attr:`format`. + +.. _half precision format: https://en.wikipedia.org/wiki/Half-precision_floating-point_format + +.. _ieee 754 standard: https://en.wikipedia.org/wiki/IEEE_floating_point#IEEE_754-2008 diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index c1b28f8fc3..ad2abe8245 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -38,7 +38,8 @@ compatibility with older versions, see the :ref:`call-function-trio` section. .. function:: run(args, *, stdin=None, input=None, stdout=None, stderr=None,\ - shell=False, timeout=None, check=False) + shell=False, timeout=None, check=False, \ + encoding=None, errors=None) Run the command described by *args*. Wait for command to complete, then return a :class:`CompletedProcess` instance. @@ -60,15 +61,20 @@ compatibility with older versions, see the :ref:`call-function-trio` section. The *input* argument is passed to :meth:`Popen.communicate` and thus to the subprocess's stdin. If used it must be a byte sequence, or a string if - ``universal_newlines=True``. When used, the internal :class:`Popen` object - is automatically created with ``stdin=PIPE``, and the *stdin* argument may - not be used as well. + *encoding* or *errors* is specified or *universal_newlines* is true. When + used, the internal :class:`Popen` object is automatically created with + ``stdin=PIPE``, and the *stdin* argument may not be used as well. If *check* is true, and the process exits with a non-zero exit code, a :exc:`CalledProcessError` exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured. + If *encoding* or *errors* are specified, or *universal_newlines* is true, + file objects for stdin, stdout and stderr are opened in text mode using the + specified *encoding* and *errors* or the :class:`io.TextIOWrapper` default. + Otherwise, file objects are opened in binary mode. + Examples:: >>> subprocess.run(["ls", "-l"]) # doesn't capture output @@ -85,6 +91,10 @@ compatibility with older versions, see the :ref:`call-function-trio` section. .. versionadded:: 3.5 + .. versionchanged:: 3.6 + + Added *encoding* and *errors* parameters + .. class:: CompletedProcess The return value from :func:`run`, representing a process that has finished. @@ -104,8 +114,8 @@ compatibility with older versions, see the :ref:`call-function-trio` section. .. attribute:: stdout Captured stdout from the child process. A bytes sequence, or a string if - :func:`run` was called with ``universal_newlines=True``. ``None`` if stdout - was not captured. + :func:`run` was called with an encoding or errors. ``None`` if stdout was not + captured. If you ran the process with ``stderr=subprocess.STDOUT``, stdout and stderr will be combined in this attribute, and :attr:`stderr` will be @@ -114,8 +124,8 @@ compatibility with older versions, see the :ref:`call-function-trio` section. .. attribute:: stderr Captured stderr from the child process. A bytes sequence, or a string if - :func:`run` was called with ``universal_newlines=True``. ``None`` if stderr - was not captured. + :func:`run` was called with an encoding or errors. ``None`` if stderr was not + captured. .. method:: check_returncode() @@ -249,19 +259,22 @@ default values. The arguments that are most commonly needed are: .. index:: single: universal newlines; subprocess module - If *universal_newlines* is ``False`` the file objects *stdin*, *stdout* and - *stderr* will be opened as binary streams, and no line ending conversion is - done. + If *encoding* or *errors* are specified, or *universal_newlines* is true, + the file objects *stdin*, *stdout* and *stderr* will be opened in text + mode using the *encoding* and *errors* specified in the call or the + defaults for :class:`io.TextIOWrapper`. + + For *stdin*, line ending characters ``'\n'`` in the input will be converted + to the default line separator :data:`os.linesep`. For *stdout* and *stderr*, + all line endings in the output will be converted to ``'\n'``. For more + information see the documentation of the :class:`io.TextIOWrapper` class + when the *newline* argument to its constructor is ``None``. - If *universal_newlines* is ``True``, these file objects - will be opened as text streams in :term:`universal newlines` mode - using the encoding returned by :func:`locale.getpreferredencoding(False) - <locale.getpreferredencoding>`. For *stdin*, line ending characters - ``'\n'`` in the input will be converted to the default line separator - :data:`os.linesep`. For *stdout* and *stderr*, all line endings in the - output will be converted to ``'\n'``. For more information see the - documentation of the :class:`io.TextIOWrapper` class when the *newline* - argument to its constructor is ``None``. + If text mode is not used, *stdin*, *stdout* and *stderr* will be opened as + binary streams. No encoding or line ending conversion is performed. + + .. versionadded:: 3.6 + Added *encoding* and *errors* parameters. .. note:: @@ -306,7 +319,8 @@ functions. stderr=None, preexec_fn=None, close_fds=True, shell=False, \ cwd=None, env=None, universal_newlines=False, \ startupinfo=None, creationflags=0, restore_signals=True, \ - start_new_session=False, pass_fds=()) + start_new_session=False, pass_fds=(), *, \ + encoding=None, errors=None) Execute a child program in a new process. On POSIX, the class uses :meth:`os.execvp`-like behavior to execute the child program. On Windows, @@ -482,10 +496,14 @@ functions. .. _side-by-side assembly: https://en.wikipedia.org/wiki/Side-by-Side_Assembly - If *universal_newlines* is ``True``, the file objects *stdin*, *stdout* - and *stderr* are opened as text streams in universal newlines mode, as - described above in :ref:`frequently-used-arguments`, otherwise they are - opened as binary streams. + If *encoding* or *errors* are specified, the file objects *stdin*, *stdout* + and *stderr* are opened in text mode with the specified encoding and + *errors*, as described above in :ref:`frequently-used-arguments`. If + *universal_newlines* is ``True``, they are opened in text mode with default + encoding. Otherwise, they are opened as binary streams. + + .. versionadded:: 3.6 + *encoding* and *errors* were added. If given, *startupinfo* will be a :class:`STARTUPINFO` object, which is passed to the underlying ``CreateProcess`` function. @@ -502,6 +520,10 @@ functions. .. versionchanged:: 3.2 Added context manager support. + .. versionchanged:: 3.6 + Popen destructor now emits a :exc:`ResourceWarning` warning if the child + process is still running. + Exceptions ^^^^^^^^^^ @@ -597,11 +619,12 @@ Instances of the :class:`Popen` class have the following methods: Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional *input* argument should be data to be sent to the child process, or - ``None``, if no data should be sent to the child. The type of *input* - must be bytes or, if *universal_newlines* was ``True``, a string. + ``None``, if no data should be sent to the child. If streams were opened in + text mode, *input* must be a string. Otherwise, it must be bytes. :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``. - The data will be bytes or, if *universal_newlines* was ``True``, strings. + The data will be strings if streams were opened in text mode; otherwise, + bytes. Note that if you want to send data to the process's stdin, you need to create the Popen object with ``stdin=PIPE``. Similarly, to get anything other than @@ -668,28 +691,30 @@ The following attributes are also available: .. attribute:: Popen.stdin If the *stdin* argument was :data:`PIPE`, this attribute is a writeable - stream object as returned by :func:`open`. If the *universal_newlines* - argument was ``True``, the stream is a text stream, otherwise it is a byte - stream. If the *stdin* argument was not :data:`PIPE`, this attribute is - ``None``. + stream object as returned by :func:`open`. If the *encoding* or *errors* + arguments were specified or the *universal_newlines* argument was ``True``, + the stream is a text stream, otherwise it is a byte stream. If the *stdin* + argument was not :data:`PIPE`, this attribute is ``None``. .. attribute:: Popen.stdout If the *stdout* argument was :data:`PIPE`, this attribute is a readable stream object as returned by :func:`open`. Reading from the stream provides - output from the child process. If the *universal_newlines* argument was - ``True``, the stream is a text stream, otherwise it is a byte stream. If the - *stdout* argument was not :data:`PIPE`, this attribute is ``None``. + output from the child process. If the *encoding* or *errors* arguments were + specified or the *universal_newlines* argument was ``True``, the stream is a + text stream, otherwise it is a byte stream. If the *stdout* argument was not + :data:`PIPE`, this attribute is ``None``. .. attribute:: Popen.stderr If the *stderr* argument was :data:`PIPE`, this attribute is a readable stream object as returned by :func:`open`. Reading from the stream provides - error output from the child process. If the *universal_newlines* argument was - ``True``, the stream is a text stream, otherwise it is a byte stream. If the - *stderr* argument was not :data:`PIPE`, this attribute is ``None``. + error output from the child process. If the *encoding* or *errors* arguments + were specified or the *universal_newlines* argument was ``True``, the stream + is a text stream, otherwise it is a byte stream. If the *stderr* argument was + not :data:`PIPE`, this attribute is ``None``. .. warning:: @@ -882,7 +907,9 @@ calls these functions. *timeout* was added. -.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) +.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, \ + encoding=None, errors=None, \ + universal_newlines=False, timeout=None) Run command with arguments and return its output. @@ -1138,7 +1165,7 @@ handling consistency are valid for these functions. Return ``(status, output)`` of executing *cmd* in a shell. Execute the string *cmd* in a shell with :meth:`Popen.check_output` and - return a 2-tuple ``(status, output)``. Universal newlines mode is used; + return a 2-tuple ``(status, output)``. The locale encoding is used; see the notes on :ref:`frequently-used-arguments` for more details. A trailing newline is stripped from the output. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 7aed75d483..2d14a1dd18 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -256,7 +256,7 @@ always available. (defaulting to zero), or another type of object. If it is an integer, zero is considered "successful termination" and any nonzero value is considered "abnormal termination" by shells and the like. Most systems require it to be - in the range 0-127, and produce undefined results otherwise. Some systems + in the range 0--127, and produce undefined results otherwise. Some systems have a convention for assigning specific meanings to specific exit codes, but these are generally underdeveloped; Unix programs generally use 2 for command line syntax errors and 1 for all other kind of errors. If another type of @@ -269,6 +269,11 @@ always available. the process when called from the main thread, and the exception is not intercepted. + .. versionchanged:: 3.6 + If an error occurs in the cleanup after the Python interpreter + has caught :exc:`SystemExit` (such as an error flushing buffered data + in the standard streams), the exit status is changed to 120. + .. data:: flags @@ -423,25 +428,42 @@ always available. .. function:: getfilesystemencoding() - Return the name of the encoding used to convert Unicode filenames into - system file names. The result value depends on the operating system: + Return the name of the encoding used to convert between Unicode + filenames and bytes filenames. For best compatibility, str should be + used for filenames in all cases, although representing filenames as bytes + is also supported. Functions accepting or returning filenames should support + either str or bytes and internally convert to the system's preferred + representation. - * On Mac OS X, the encoding is ``'utf-8'``. + This encoding is always ASCII-compatible. - * On Unix, the encoding is the user's preference according to the result of - nl_langinfo(CODESET). + :func:`os.fsencode` and :func:`os.fsdecode` should be used to ensure that + the correct encoding and errors mode are used. - * On Windows NT+, file names are Unicode natively, so no conversion is - performed. :func:`getfilesystemencoding` still returns ``'mbcs'``, as - this is the encoding that applications should use when they explicitly - want to convert Unicode strings to byte strings that are equivalent when - used as file names. + * On Mac OS X, the encoding is ``'utf-8'``. - * On Windows 9x, the encoding is ``'mbcs'``. + * On Unix, the encoding is the locale encoding. + + * On Windows, the encoding may be ``'utf-8'`` or ``'mbcs'``, depending + on user configuration. .. versionchanged:: 3.2 :func:`getfilesystemencoding` result cannot be ``None`` anymore. + .. versionchanged:: 3.6 + Windows is no longer guaranteed to return ``'mbcs'``. See :pep:`529` + and :func:`_enablelegacywindowsfsencoding` for more information. + +.. function:: getfilesystemencodeerrors() + + Return the name of the error mode used to convert between Unicode filenames + and bytes filenames. The encoding name is returned from + :func:`getfilesystemencoding`. + + :func:`os.fsencode` and :func:`os.fsdecode` should be used to ensure that + the correct encoding and errors mode are used. + + .. versionadded:: 3.6 .. function:: getrefcount(object) @@ -530,26 +552,15 @@ always available. Return a named tuple describing the Windows version currently running. The named elements are *major*, *minor*, *build*, *platform*, *service_pack*, *service_pack_minor*, - *service_pack_major*, *suite_mask*, and *product_type*. - *service_pack* contains a string while all other values are + *service_pack_major*, *suite_mask*, *product_type* and + *platform_version*. *service_pack* contains a string, + *platform_version* a 3-tuple and all other values are integers. The components can also be accessed by name, so ``sys.getwindowsversion()[0]`` is equivalent to ``sys.getwindowsversion().major``. For compatibility with prior versions, only the first 5 elements are retrievable by indexing. - *platform* may be one of the following values: - - +-----------------------------------------+-------------------------+ - | Constant | Platform | - +=========================================+=========================+ - | :const:`0 (VER_PLATFORM_WIN32s)` | Win32s on Windows 3.1 | - +-----------------------------------------+-------------------------+ - | :const:`1 (VER_PLATFORM_WIN32_WINDOWS)` | Windows 95/98/ME | - +-----------------------------------------+-------------------------+ - | :const:`2 (VER_PLATFORM_WIN32_NT)` | Windows NT/2000/XP/x64 | - +-----------------------------------------+-------------------------+ - | :const:`3 (VER_PLATFORM_WIN32_CE)` | Windows CE | - +-----------------------------------------+-------------------------+ + *platform* will be :const:`2 (VER_PLATFORM_WIN32_NT)`. *product_type* may be one of the following values: @@ -565,17 +576,23 @@ always available. | | a domain controller. | +---------------------------------------+---------------------------------+ - This function wraps the Win32 :c:func:`GetVersionEx` function; see the Microsoft documentation on :c:func:`OSVERSIONINFOEX` for more information about these fields. + *platform_version* returns the accurate major version, minor version and + build number of the current operating system, rather than the version that + is being emulated for the process. It is intended for use in logging rather + than for feature detection. + Availability: Windows. .. versionchanged:: 3.2 Changed to a named tuple and added *service_pack_minor*, *service_pack_major*, *suite_mask*, and *product_type*. + .. versionchanged:: 3.6 + Added *platform_version* .. function:: get_coroutine_wrapper() @@ -1082,18 +1099,6 @@ always available. thus may not be available in all Python implementations. -.. function:: settscdump(on_flag) - - Activate dumping of VM measurements using the Pentium timestamp counter, if - *on_flag* is true. Deactivate these dumps if *on_flag* is off. The function is - available only if Python was compiled with ``--with-tsc``. To understand - the output of this dump, read :file:`Python/ceval.c` in the Python sources. - - .. impl-detail:: - This function is intimately bound to CPython implementation details and - thus not likely to be implemented elsewhere. - - .. function:: set_coroutine_wrapper(wrapper) Allows intercepting creation of :term:`coroutine` objects (only ones that @@ -1133,6 +1138,18 @@ always available. This function has been added on a provisional basis (see :pep:`411` for details.) Use it only for debugging purposes. +.. function:: _enablelegacywindowsfsencoding() + + Changes the default filesystem encoding and errors mode to 'mbcs' and + 'replace' respectively, for consistency with versions of Python prior to 3.6. + + This is equivalent to defining the :envvar:`PYTHONLEGACYWINDOWSFSENCODING` + environment variable before launching Python. + + Availability: Windows + + .. versionadded:: 3.6 + See :pep:`529` for more details. .. data:: stdin stdout diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst index 02aaab3cf2..08b74a9aff 100644 --- a/Doc/library/sysconfig.rst +++ b/Doc/library/sysconfig.rst @@ -164,7 +164,7 @@ Other functions .. function:: get_python_version() Return the ``MAJOR.MINOR`` Python version number as a string. Similar to - ``sys.version[:3]``. + ``'%d.%d' % sys.version_info[:2]``. .. function:: get_platform() diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst index b950e41dc1..f9c5153e3f 100644 --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -43,6 +43,17 @@ Character), EL (Erase Line), GA (Go Ahead), SB (Subnegotiation Begin). :exc:`EOFError` when the end of the connection is read, because they can return an empty string for other reasons. See the individual descriptions below. + A :class:`Telnet` object is a context manager and can be used in a + :keyword:`with` statement. When the :keyword:`with` block ends, the + :meth:`close` method is called:: + + >>> from telnetlib import Telnet + >>> with Telnet('localhost', 23) as tn: + ... tn.interact() + ... + + .. versionchanged:: 3.6 Context manager support added + .. seealso:: diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 59577f0fe8..fab3e1fe4c 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -582,6 +582,48 @@ The :mod:`test.support` module defines the following functions: .. versionadded:: 3.5 +.. function:: check__all__(test_case, module, name_of_module=None, extra=(), blacklist=()) + + Assert that the ``__all__`` variable of *module* contains all public names. + + The module's public names (its API) are detected automatically + based on whether they match the public name convention and were defined in + *module*. + + The *name_of_module* argument can specify (as a string or tuple thereof) what + module(s) an API could be defined in in order to be detected as a public + API. One case for this is when *module* imports part of its public API from + other modules, possibly a C backend (like ``csv`` and its ``_csv``). + + The *extra* argument can be a set of names that wouldn't otherwise be automatically + detected as "public", like objects without a proper ``__module__`` + attribute. If provided, it will be added to the automatically detected ones. + + The *blacklist* argument can be a set of names that must not be treated as part of + the public API even though their names indicate otherwise. + + Example use:: + + import bar + import foo + import unittest + from test import support + + class MiscTestCase(unittest.TestCase): + def test__all__(self): + support.check__all__(self, foo) + + class OtherTestCase(unittest.TestCase): + def test__all__(self): + extra = {'BAR_CONST', 'FOO_CONST'} + blacklist = {'baz'} # Undocumented name. + # bar imports part of its API from _bar. + support.check__all__(self, bar, ('bar', '_bar'), + extra=extra, blacklist=blacklist) + + .. versionadded:: 3.6 + + The :mod:`test.support` module defines the following classes: .. class:: TransientResource(exc, **kwargs) diff --git a/Doc/library/time.rst b/Doc/library/time.rst index e6626f262d..ae17f6f4f0 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -83,6 +83,10 @@ An explanation of some terminology and conventions is in order. and :attr:`tm_zone` attributes when platform supports corresponding ``struct tm`` members. + .. versionchanged:: 3.6 + The :class:`struct_time` attributes :attr:`tm_gmtoff` and :attr:`tm_zone` + are now available on all platforms. + * Use the following functions to convert between time representations: +-------------------------+-------------------------+-------------------------+ @@ -566,10 +570,6 @@ The module defines the following functions and data items: :class:`struct_time`, or having elements of the wrong type, a :exc:`TypeError` is raised. - .. versionchanged:: 3.3 - :attr:`tm_gmtoff` and :attr:`tm_zone` attributes are available on platforms - with C library supporting the corresponding fields in ``struct tm``. - .. function:: time() Return the time in seconds since the epoch as a floating point number. @@ -637,11 +637,11 @@ The module defines the following functions and data items: it is possible to refer to February 29. :samp:`M{m}.{n}.{d}` - The *d*'th day (0 <= *d* <= 6) or week *n* of month *m* of the year (1 + The *d*'th day (0 <= *d* <= 6) of week *n* of month *m* of the year (1 <= *n* <= 5, 1 <= *m* <= 12, where week 5 means "the last *d* day in month *m*" which may occur in either the fourth or the fifth week). Week 1 is the first week in which the *d*'th day occurs. Day - zero is Sunday. + zero is a Sunday. ``time`` has the same format as ``offset`` except that no leading sign ('-' or '+') is allowed. The default, if time is not given, is 02:00:00. diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 57a4834c90..3b772765ac 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -100,8 +100,8 @@ The module defines three convenience functions and a public class: can be controlled by passing a namespace to *globals*. To measure the execution time of the first statement, use the :meth:`.timeit` - method. The :meth:`.repeat` method is a convenience to call :meth:`.timeit` - multiple times and return a list of results. + method. The :meth:`.repeat` and :meth:`.autorange` methods are convenience + methods to call :meth:`.timeit` multiple times. The execution time of *setup* is excluded from the overall timed execution run. @@ -134,6 +134,23 @@ The module defines three convenience functions and a public class: timeit.Timer('for i in range(10): oct(i)', 'gc.enable()').timeit() + .. method:: Timer.autorange(callback=None) + + Automatically determine how many times to call :meth:`.timeit`. + + This is a convenience function that calls :meth:`.timeit` repeatedly + so that the total time >= 0.2 second, returning the eventual + (number of loops, time taken for that number of loops). It calls + :meth:`.timeit` with *number* set to successive powers of ten (10, + 100, 1000, ...) up to a maximum of one billion, until the time taken + is at least 0.2 second, or the maximum is reached. + + If *callback* is given and is not ``None``, it will be called after + each trial with two arguments: ``callback(number, time_taken)``. + + .. versionadded:: 3.6 + + .. method:: Timer.repeat(repeat=3, number=1000000) Call :meth:`.timeit` a few times. diff --git a/Doc/library/tkinter.tix.rst b/Doc/library/tkinter.tix.rst index 41f20ddf4e..11ed755137 100644 --- a/Doc/library/tkinter.tix.rst +++ b/Doc/library/tkinter.tix.rst @@ -10,6 +10,10 @@ .. index:: single: Tix +.. deprecated:: 3.6 + This Tk extension is unmaintained and should not be used in new code. Use + :mod:`tkinter.ttk` instead. + -------------- The :mod:`tkinter.tix` (Tk Interface Extension) module provides an additional diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 3c1d9bb51d..066ee96fc0 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -291,6 +291,20 @@ capture data for later printing in a lightweight fashion. of tuples. Each tuple should be a 4-tuple with filename, lineno, name, line as the elements. + .. method:: format() + + Returns a list of strings ready for printing. Each string in the + resulting list corresponds to a single frame from the stack. + Each string ends in a newline; the strings may contain internal + newlines as well, for those items with source text lines. + + For long sequences of the same frame and line, the first few + repetitions are shown, followed by a summary line stating the exact + number of further repetitions. + + .. versionchanged:: 3.6 + Long sequences of repeated frames are now abbreviated. + :class:`FrameSummary` Objects ----------------------------- diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst index 3a0b1e0797..f56f27b626 100644 --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -359,10 +359,32 @@ Functions See also the :func:`get_object_traceback` function. +DomainFilter +^^^^^^^^^^^^ + +.. class:: DomainFilter(inclusive: bool, domain: int) + + Filter traces of memory blocks by their address space (domain). + + .. versionadded:: 3.6 + + .. attribute:: inclusive + + If *inclusive* is ``True`` (include), match memory blocks allocated + in the address space :attr:`domain`. + + If *inclusive* is ``False`` (exclude), match memory blocks not allocated + in the address space :attr:`domain`. + + .. attribute:: domain + + Address space of a memory block (``int``). Read-only property. + + Filter ^^^^^^ -.. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False) +.. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False, domain: int=None) Filter on traces of memory blocks. @@ -382,9 +404,17 @@ Filter .. versionchanged:: 3.5 The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. + .. versionchanged:: 3.6 + Added the :attr:`domain` attribute. + + + .. attribute:: domain + + Address space of a memory block (``int`` or ``None``). + .. attribute:: inclusive - If *inclusive* is ``True`` (include), only trace memory blocks allocated + If *inclusive* is ``True`` (include), only match memory blocks allocated in a file with a name matching :attr:`filename_pattern` at line number :attr:`lineno`. @@ -399,7 +429,7 @@ Filter .. attribute:: filename_pattern - Filename pattern of the filter (``str``). + Filename pattern of the filter (``str``). Read-only property. .. attribute:: all_frames @@ -462,14 +492,17 @@ Snapshot .. method:: filter_traces(filters) Create a new :class:`Snapshot` instance with a filtered :attr:`traces` - sequence, *filters* is a list of :class:`Filter` instances. If *filters* - is an empty list, return a new :class:`Snapshot` instance with a copy of - the traces. + sequence, *filters* is a list of :class:`DomainFilter` and + :class:`Filter` instances. If *filters* is an empty list, return a new + :class:`Snapshot` instance with a copy of the traces. All inclusive filters are applied at once, a trace is ignored if no inclusive filters match it. A trace is ignored if at least one exclusive filter matches it. + .. versionchanged:: 3.6 + :class:`DomainFilter` instances are now also accepted in *filters*. + .. classmethod:: load(filename) diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 118bc4c29d..898b95a940 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -55,6 +55,12 @@ Dynamic Type Creation .. versionadded:: 3.3 + .. versionchanged:: 3.6 + + The default value for the ``namespace`` element of the returned + tuple has changed. Now an insertion-order-preserving mapping is + used when the metaclass does not have a ``__prepare__`` method, + .. seealso:: :ref:`metaclasses` diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 9f7fa1a7cf..bbf5db6891 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -10,8 +10,8 @@ -------------- -This module supports type hints as specified by :pep:`484`. The most -fundamental support consists of the type :data:`Any`, :data:`Union`, +This module supports type hints as specified by :pep:`484` and :pep:`526`. +The most fundamental support consists of the type :data:`Any`, :data:`Union`, :data:`Tuple`, :data:`Callable`, :class:`TypeVar`, and :class:`Generic`. For full specification please see :pep:`484`. For a simplified introduction to type hints see :pep:`483`. @@ -507,7 +507,13 @@ The module defines the following classes, functions and decorators: An alias to :class:`collections.abc.Sized` -.. class:: AbstractSet(Sized, Iterable[T_co], Container[T_co]) +.. class:: Collection(Sized, Iterable[T_co], Container[T_co]) + + A generic version of :class:`collections.abc.Collection` + + .. versionadded:: 3.6 + +.. class:: AbstractSet(Sized, Collection[T_co]) A generic version of :class:`collections.abc.Set`. @@ -515,7 +521,7 @@ The module defines the following classes, functions and decorators: A generic version of :class:`collections.abc.MutableSet`. -.. class:: Mapping(Sized, Iterable[KT], Container[KT], Generic[VT_co]) +.. class:: Mapping(Sized, Collection[KT], Generic[VT_co]) A generic version of :class:`collections.abc.Mapping`. @@ -523,7 +529,7 @@ The module defines the following classes, functions and decorators: A generic version of :class:`collections.abc.MutableMapping`. -.. class:: Sequence(Sized, Iterable[T_co], Container[T_co]) +.. class:: Sequence(Reversible[T_co], Collection[T_co]) A generic version of :class:`collections.abc.Sequence`. @@ -590,6 +596,12 @@ The module defines the following classes, functions and decorators: A generic version of :class:`collections.abc.AsyncIterator`. +.. class:: ContextManager(Generic[T_co]) + + A generic version of :class:`contextlib.AbstractContextManager`. + + .. versionadded:: 3.6 + .. class:: Dict(dict, MutableMapping[KT, VT]) A generic version of :class:`dict`. @@ -702,7 +714,7 @@ The module defines the following classes, functions and decorators: .. function:: get_type_hints(obj) - Return type hints for a function or method object. + Return type hints for a class, module, function or method object. This is often the same as ``obj.__annotations__``, but it handles forward references encoded as string literals, and if necessary @@ -845,8 +857,8 @@ The module defines the following classes, functions and decorators: and should not be set on instances of that class. Usage:: class Starship: - stats = {} # type: ClassVar[Dict[str, int]] # class variable - damage = 10 # type: int # instance variable + stats: ClassVar[Dict[str, int]] = {} # class variable + damage: int = 10 # instance variable :data:`ClassVar` accepts only types and cannot be further subscribed. diff --git a/Doc/library/undoc.rst b/Doc/library/undoc.rst index 20830e2963..2444080d6b 100644 --- a/Doc/library/undoc.rst +++ b/Doc/library/undoc.rst @@ -20,7 +20,7 @@ These modules are used to implement the :mod:`os.path` module, and are not documented beyond this mention. There's little need to document these. :mod:`ntpath` - --- Implementation of :mod:`os.path` on Win32, Win64, and WinCE platforms. + --- Implementation of :mod:`os.path` on Win32 and Win64 platforms. :mod:`posixpath` --- Implementation of :mod:`os.path` on POSIX. diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index 6cd8132f8a..643180953f 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -17,8 +17,8 @@ This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in -this database is compiled from the `UCD version 8.0.0 -<http://www.unicode.org/Public/8.0.0/ucd>`_. +this database is compiled from the `UCD version 9.0.0 +<http://www.unicode.org/Public/9.0.0/ucd>`_. The module uses the same names and symbols as defined by Unicode Standard Annex #44, `"Unicode Character Database" @@ -168,6 +168,6 @@ Examples: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/8.0.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/9.0.0/ucd/NameAliases.txt -.. [#] http://www.unicode.org/Public/8.0.0/ucd/NamedSequences.txt +.. [#] http://www.unicode.org/Public/9.0.0/ucd/NamedSequences.txt diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index ef30c01b99..3cc22fdde7 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -262,6 +262,34 @@ the *new_callable* argument to :func:`patch`. used to set attributes on the mock after it is created. See the :meth:`configure_mock` method for details. + .. method:: assert_called(*args, **kwargs) + + Assert that the mock was called at least once. + + >>> mock = Mock() + >>> mock.method() + <Mock name='mock.method()' id='...'> + >>> mock.method.assert_called() + + .. versionadded:: 3.6 + + .. method:: assert_called_once(*args, **kwargs) + + Assert that the mock was called exactly once. + + >>> mock = Mock() + >>> mock.method() + <Mock name='mock.method()' id='...'> + >>> mock.method.assert_called_once() + >>> mock.method() + <Mock name='mock.method()' id='...'> + >>> mock.method.assert_called_once() + Traceback (most recent call last): + ... + AssertionError: Expected 'method' to have been called once. Called 2 times. + + .. versionadded:: 3.6 + .. method:: assert_called_with(*args, **kwargs) @@ -339,7 +367,7 @@ the *new_callable* argument to :func:`patch`. .. versionadded:: 3.5 - .. method:: reset_mock() + .. method:: reset_mock(*, return_value=False, side_effect=False) The reset_mock method resets all the call attributes on a mock object: @@ -351,12 +379,20 @@ the *new_callable* argument to :func:`patch`. >>> mock.called False + .. versionchanged:: 3.6 + Added two keyword only argument to the reset_mock function. + This can be useful where you want to make a series of assertions that reuse the same object. Note that :meth:`reset_mock` *doesn't* clear the return value, :attr:`side_effect` or any child attributes you have - set using normal assignment. Child mocks and the return value mock + set using normal assignment by default. In case you want to reset + *return_value* or :attr:`side_effect`, then pass the corresponding + parameter as ``True``. Child mocks and the return value mock (if any) are reset as well. + .. note:: *return_value*, and :attr:`side_effect` are keyword only + argument. + .. method:: mock_add_spec(spec, spec_set=False) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index b9b339b761..1b0d7a7881 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -1391,9 +1391,9 @@ Test cases Add a function to be called after :meth:`tearDown` to cleanup resources used during the test. Functions will be called in reverse order to the - order they are added (LIFO). They are called with any arguments and - keyword arguments passed into :meth:`addCleanup` when they are - added. + order they are added (:abbr:`LIFO (last-in, first-out)`). They + are called with any arguments and keyword arguments passed into + :meth:`addCleanup` when they are added. If :meth:`setUp` fails, meaning that :meth:`tearDown` is not called, then any cleanup functions added will still be called. diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index 499b2110c4..676321b46a 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -114,8 +114,9 @@ or on combining URL components into a URL string. | | | if present | | +------------------+-------+--------------------------+----------------------+ - See section :ref:`urlparse-result-object` for more information on the result - object. + Reading the :attr:`port` attribute will raise a :exc:`ValueError` if + an invalid port is specified in the URL. See section + :ref:`urlparse-result-object` for more information on the result object. .. versionchanged:: 3.2 Added IPv6 URL parsing capabilities. @@ -125,6 +126,10 @@ or on combining URL components into a URL string. false), in accordance with :rfc:`3986`. Previously, a whitelist of schemes that support fragments existed. + .. versionchanged:: 3.6 + Out-of-range port numbers now raise :exc:`ValueError`, instead of + returning :const:`None`. + .. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace') @@ -227,8 +232,13 @@ or on combining URL components into a URL string. | | | if present | | +------------------+-------+-------------------------+----------------------+ - See section :ref:`urlparse-result-object` for more information on the result - object. + Reading the :attr:`port` attribute will raise a :exc:`ValueError` if + an invalid port is specified in the URL. See section + :ref:`urlparse-result-object` for more information on the result object. + + .. versionchanged:: 3.6 + Out-of-range port numbers now raise :exc:`ValueError`, instead of + returning :const:`None`. .. function:: urlunsplit(parts) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 41b5eced38..c990fb1e7e 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -30,18 +30,9 @@ The :mod:`urllib.request` module defines the following functions: Open the URL *url*, which can be either a string or a :class:`Request` object. - *data* must be a bytes object specifying additional data to be sent to the - server, or ``None`` if no such data is needed. *data* may also be an - iterable object and in that case Content-Length value must be specified in - the headers. Currently HTTP requests are the only ones that use *data*; the - HTTP request will be a POST instead of a GET when the *data* parameter is - provided. - - *data* should be a buffer in the standard - :mimetype:`application/x-www-form-urlencoded` format. The - :func:`urllib.parse.urlencode` function takes a mapping or sequence of - 2-tuples and returns an ASCII text string in this format. It should - be encoded to bytes before being used as the *data* parameter. + *data* must be an object specifying additional data to be sent to the + server, or ``None`` if no such data is needed. See :class:`Request` + for details. urllib.request module uses HTTP/1.1 and includes ``Connection:close`` header in its HTTP requests. @@ -120,6 +111,12 @@ The :mod:`urllib.request` module defines the following functions: .. versionchanged:: 3.4.3 *context* was added. + .. deprecated:: 3.6 + + *cafile*, *capath* and *cadefault* are deprecated in favor of *context*. + Please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let + :func:`ssl.create_default_context` select the system's trusted CA + certificates for you. .. function:: install_opener(opener) @@ -192,14 +189,21 @@ The following classes are provided: *url* should be a string containing a valid URL. - *data* must be a bytes object specifying additional data to send to the - server, or ``None`` if no such data is needed. Currently HTTP requests are - the only ones that use *data*; the HTTP request will be a POST instead of a - GET when the *data* parameter is provided. *data* should be a buffer in the - standard :mimetype:`application/x-www-form-urlencoded` format. - The :func:`urllib.parse.urlencode` function takes a mapping or sequence of - 2-tuples and returns an ASCII string in this format. It should be - encoded to bytes before being used as the *data* parameter. + *data* must be an object specifying additional data to send to the + server, or ``None`` if no such data is needed. Currently HTTP + requests are the only ones that use *data*. The supported object + types include bytes, file-like objects, and iterables. If no + ``Content-Length`` nor ``Transfer-Encoding`` header field + has been provided, :class:`HTTPHandler` will set these headers according + to the type of *data*. ``Content-Length`` will be used to send + bytes objects, while ``Transfer-Encoding: chunked`` as specified in + :rfc:`7230`, Section 3.3.1 will be used to send files and other iterables. + + For an HTTP POST request method, *data* should be a buffer in the + standard :mimetype:`application/x-www-form-urlencoded` format. The + :func:`urllib.parse.urlencode` function takes a mapping or sequence + of 2-tuples and returns an ASCII string in this format. It should + be encoded to bytes before being used as the *data* parameter. *headers* should be a dictionary, and will be treated as if :meth:`add_header` was called with each key and value as arguments. @@ -211,8 +215,10 @@ The following classes are provided: :mod:`urllib`'s default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6). - An example of using ``Content-Type`` header with *data* argument would be - sending a dictionary like ``{"Content-Type": "application/x-www-form-urlencoded"}``. + An appropriate ``Content-Type`` header should be included if the *data* + argument is present. If this header has not been provided and *data* + is not None, ``Content-Type: application/x-www-form-urlencoded`` will + be added as a default. The final two arguments are only of interest for correct handling of third-party HTTP cookies: @@ -235,15 +241,28 @@ The following classes are provided: *method* should be a string that indicates the HTTP request method that will be used (e.g. ``'HEAD'``). If provided, its value is stored in the :attr:`~Request.method` attribute and is used by :meth:`get_method()`. - Subclasses may indicate a default method by setting the + The default is ``'GET'`` if *data* is ``None`` or ``'POST'`` otherwise. + Subclasses may indicate a different default method by setting the :attr:`~Request.method` attribute in the class itself. + .. note:: + The request will not work as expected if the data object is unable + to deliver its content more than once (e.g. a file or an iterable + that can produce the content only once) and the request is retried + for HTTP redirects or authentication. The *data* is sent to the + HTTP server right away after the headers. There is no support for + a 100-continue expectation in the library. + .. versionchanged:: 3.3 :attr:`Request.method` argument is added to the Request class. .. versionchanged:: 3.4 Default :attr:`Request.method` may be indicated at the class level. + .. versionchanged:: 3.6 + Do not raise an error if the ``Content-Length`` has not been + provided and *data* is neither ``None`` nor a bytes object. + Fall back to use chunked transfer encoding instead. .. class:: OpenerDirector() diff --git a/Doc/library/urllib.robotparser.rst b/Doc/library/urllib.robotparser.rst index ba701c3b68..7d31932f96 100644 --- a/Doc/library/urllib.robotparser.rst +++ b/Doc/library/urllib.robotparser.rst @@ -57,15 +57,41 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html. Sets the time the ``robots.txt`` file was last fetched to the current time. + .. method:: crawl_delay(useragent) -The following example demonstrates basic use of the RobotFileParser class. + Returns the value of the ``Crawl-delay`` parameter from ``robots.txt`` + for the *useragent* in question. If there is no such parameter or it + doesn't apply to the *useragent* specified or the ``robots.txt`` entry + for this parameter has invalid syntax, return ``None``. + + .. versionadded:: 3.6 + + .. method:: request_rate(useragent) + + Returns the contents of the ``Request-rate`` parameter from + ``robots.txt`` in the form of a :func:`~collections.namedtuple` + ``(requests, seconds)``. If there is no such parameter or it doesn't + apply to the *useragent* specified or the ``robots.txt`` entry for this + parameter has invalid syntax, return ``None``. + + .. versionadded:: 3.6 + + +The following example demonstrates basic use of the :class:`RobotFileParser` +class:: >>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.musi-cal.com/robots.txt") >>> rp.read() + >>> rrate = rp.request_rate("*") + >>> rrate.requests + 3 + >>> rrate.seconds + 20 + >>> rp.crawl_delay("*") + 6 >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco") False >>> rp.can_fetch("*", "http://www.musi-cal.com/") True - diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst index 62892d4027..a80d2d64ef 100644 --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -37,44 +37,50 @@ Creating virtual environments .. _venv-def: -.. note:: A virtual environment (also called a ``venv``) is a Python - environment such that the Python interpreter, libraries and scripts - installed into it are isolated from those installed in other virtual - environments, and (by default) any libraries installed in a "system" Python, - i.e. one which is installed as part of your operating system. +.. note:: A virtual environment is a Python environment such that the Python + interpreter, libraries and scripts installed into it are isolated from those + installed in other virtual environments, and (by default) any libraries + installed in a "system" Python, i.e., one which is installed as part of your + operating system. - A venv is a directory tree which contains Python executable files and - other files which indicate that it is a venv. + A virtual environment is a directory tree which contains Python executable + files and other files which indicate that it is a virtual environment. Common installation tools such as ``Setuptools`` and ``pip`` work as - expected with venvs - i.e. when a venv is active, they install Python - packages into the venv without needing to be told to do so explicitly. - - When a venv is active (i.e. the venv's Python interpreter is running), the - attributes :attr:`sys.prefix` and :attr:`sys.exec_prefix` point to the base - directory of the venv, whereas :attr:`sys.base_prefix` and - :attr:`sys.base_exec_prefix` point to the non-venv Python installation - which was used to create the venv. If a venv is not active, then - :attr:`sys.prefix` is the same as :attr:`sys.base_prefix` and - :attr:`sys.exec_prefix` is the same as :attr:`sys.base_exec_prefix` (they - all point to a non-venv Python installation). - - When a venv is active, any options that change the installation path will be - ignored from all distutils configuration files to prevent projects being - inadvertently installed outside of the virtual environment. - - When working in a command shell, users can make a venv active by running an - ``activate`` script in the venv's executables directory (the precise filename - is shell-dependent), which prepends the venv's directory for executables to - the ``PATH`` environment variable for the running shell. There should be no - need in other circumstances to activate a venv -- scripts installed into - venvs have a shebang line which points to the venv's Python interpreter. This - means that the script will run with that interpreter regardless of the value - of ``PATH``. On Windows, shebang line processing is supported if you have the - Python Launcher for Windows installed (this was added to Python in 3.3 - see - :pep:`397` for more details). Thus, double-clicking an installed script in - a Windows Explorer window should run the script with the correct interpreter - without there needing to be any reference to its venv in ``PATH``. + expected with virtual environments. In other words, when a virtual + environment is active, they install Python packages into the virtual + environment without needing to be told to do so explicitly. + + When a virtual environment is active (i.e., the virtual environment's Python + interpreter is running), the attributes :attr:`sys.prefix` and + :attr:`sys.exec_prefix` point to the base directory of the virtual + environment, whereas :attr:`sys.base_prefix` and + :attr:`sys.base_exec_prefix` point to the non-virtual environment Python + installation which was used to create the virtual environment. If a virtual + environment is not active, then :attr:`sys.prefix` is the same as + :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as + :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment + Python installation). + + When a virtual environment is active, any options that change the + installation path will be ignored from all distutils configuration files to + prevent projects being inadvertently installed outside of the virtual + environment. + + When working in a command shell, users can make a virtual environment active + by running an ``activate`` script in the virtual environment's executables + directory (the precise filename is shell-dependent), which prepends the + virtual environment's directory for executables to the ``PATH`` environment + variable for the running shell. There should be no need in other + circumstances to activate a virtual environment—scripts installed into + virtual environments have a "shebang" line which points to the virtual + environment's Python interpreter. This means that the script will run with + that interpreter regardless of the value of ``PATH``. On Windows, "shebang" + line processing is supported if you have the Python Launcher for Windows + installed (this was added to Python in 3.3 - see :pep:`397` for more + details). Thus, double-clicking an installed script in a Windows Explorer + window should run the script with the correct interpreter without there + needing to be any reference to its virtual environment in ``PATH``. .. _venv-api: @@ -89,7 +95,8 @@ mechanisms for third-party virtual environment creators to customize environment creation according to their needs, the :class:`EnvBuilder` class. .. class:: EnvBuilder(system_site_packages=False, clear=False, \ - symlinks=False, upgrade=False, with_pip=False) + symlinks=False, upgrade=False, with_pip=False, \ + prompt=None) The :class:`EnvBuilder` class accepts the following keyword arguments on instantiation: @@ -113,9 +120,16 @@ creation according to their needs, the :class:`EnvBuilder` class. installed in the virtual environment. This uses :mod:`ensurepip` with the ``--default-pip`` option. + * ``prompt`` -- a String to be used after virtual environment is activated + (defaults to ``None`` which means directory name of the environment would + be used). + .. versionchanged:: 3.4 Added the ``with_pip`` parameter + .. versionadded:: 3.6 + Added the ``prompt`` parameter + Creators of third-party virtual environment tools will be free to use the provided ``EnvBuilder`` class as a base class. @@ -225,7 +239,7 @@ An example of extending ``EnvBuilder`` -------------------------------------- The following script shows how to extend :class:`EnvBuilder` by implementing a -subclass which installs setuptools and pip into a created venv:: +subclass which installs setuptools and pip into a created virtual environment:: import os import os.path @@ -239,12 +253,12 @@ subclass which installs setuptools and pip into a created venv:: class ExtendedEnvBuilder(venv.EnvBuilder): """ This builder installs setuptools and pip so that you can pip or - easy_install other packages into the created environment. + easy_install other packages into the created virtual environment. :param nodist: If True, setuptools and pip are not installed into the - created environment. + created virtual environment. :param nopip: If True, pip is not installed into the created - environment. + virtual environment. :param progress: If setuptools or pip are installed, the progress of the installation can be monitored by passing a progress callable. If specified, it is called with two @@ -270,10 +284,10 @@ subclass which installs setuptools and pip into a created venv:: def post_setup(self, context): """ Set up any packages which need to be pre-installed into the - environment being created. + virtual environment being created. - :param context: The information for the environment creation request - being processed. + :param context: The information for the virtual environment + creation request being processed. """ os.environ['VIRTUAL_ENV'] = context.env_dir if not self.nodist: @@ -307,7 +321,7 @@ subclass which installs setuptools and pip into a created venv:: fn = os.path.split(path)[-1] binpath = context.bin_path distpath = os.path.join(binpath, fn) - # Download script into the env's binaries folder + # Download script into the virtual environment's binaries folder urlretrieve(url, distpath) progress = self.progress if self.verbose: @@ -319,7 +333,7 @@ subclass which installs setuptools and pip into a created venv:: else: sys.stderr.write('Installing %s ...%s' % (name, term)) sys.stderr.flush() - # Install in the env + # Install in the virtual environment args = [context.env_exe, fn] p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) @@ -338,10 +352,10 @@ subclass which installs setuptools and pip into a created venv:: def install_setuptools(self, context): """ - Install setuptools in the environment. + Install setuptools in the virtual environment. - :param context: The information for the environment creation request - being processed. + :param context: The information for the virtual environment + creation request being processed. """ url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py' self.install_script(context, 'setuptools', url) @@ -354,10 +368,10 @@ subclass which installs setuptools and pip into a created venv:: def install_pip(self, context): """ - Install pip in the environment. + Install pip in the virtual environment. - :param context: The information for the environment creation request - being processed. + :param context: The information for the virtual environment + creation request being processed. """ url = 'https://raw.github.com/pypa/pip/master/contrib/get-pip.py' self.install_script(context, 'pip', url) @@ -380,7 +394,8 @@ subclass which installs setuptools and pip into a created venv:: 'more target ' 'directories.') parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', - help='A directory to create the environment in.') + help='A directory in which to create the + 'virtual environment.') parser.add_argument('--no-setuptools', default=False, action='store_true', dest='nodist', help="Don't install setuptools or pip in the " @@ -404,14 +419,17 @@ subclass which installs setuptools and pip into a created venv:: 'the platform.') parser.add_argument('--clear', default=False, action='store_true', dest='clear', help='Delete the contents of the ' - 'environment directory if it ' - 'already exists, before ' + 'virtual environment ' + 'directory if it already ' + 'exists, before virtual ' 'environment creation.') parser.add_argument('--upgrade', default=False, action='store_true', - dest='upgrade', help='Upgrade the environment ' - 'directory to use this version ' - 'of Python, assuming Python ' - 'has been upgraded in-place.') + dest='upgrade', help='Upgrade the virtual ' + 'environment directory to ' + 'use this version of ' + 'Python, assuming Python ' + 'has been upgraded ' + 'in-place.') parser.add_argument('--verbose', default=False, action='store_true', dest='verbose', help='Display the output ' 'from the scripts which ' diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index 37f6874a02..5a42cc6bcc 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -301,7 +301,7 @@ Available Functions ------------------- -.. function:: warn(message, category=None, stacklevel=1) +.. function:: warn(message, category=None, stacklevel=1, source=None) Issue a warning, or maybe ignore it or raise an exception. The *category* argument, if given, must be a warning category class (see above); it defaults to @@ -319,8 +319,14 @@ Available Functions source of :func:`deprecation` itself (since the latter would defeat the purpose of the warning message). + *source*, if supplied, is the destroyed object which emitted a + :exc:`ResourceWarning`. -.. function:: warn_explicit(message, category, filename, lineno, module=None, registry=None, module_globals=None) + .. versionchanged:: 3.6 + Added *source* parameter. + + +.. function:: warn_explicit(message, category, filename, lineno, module=None, registry=None, module_globals=None, source=None) This is a low-level interface to the functionality of :func:`warn`, passing in explicitly the message, category, filename and line number, and optionally the @@ -336,6 +342,12 @@ Available Functions source for modules found in zipfiles or other non-filesystem import sources). + *source*, if supplied, is the destroyed object which emitted a + :exc:`ResourceWarning`. + + .. versionchanged:: 3.6 + Add the *source* parameter. + .. function:: showwarning(message, category, filename, lineno, file=None, line=None) diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index 52d591ad75..48bdf14bff 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -635,7 +635,7 @@ For more information, see `Registry Value Types .. data:: REG_DWORD_LITTLE_ENDIAN - A 32-bit number in little-endian format. + A 32-bit number in little-endian format. Equivalent to :const:`REG_DWORD`. .. data:: REG_DWORD_BIG_ENDIAN @@ -659,6 +659,18 @@ For more information, see `Registry Value Types No defined value type. +.. data:: REG_QWORD + + A 64-bit number. + + .. versionadded:: 3.6 + +.. data:: REG_QWORD_LITTLE_ENDIAN + + A 64-bit number in little-endian format. Equivalent to :const:`REG_QWORD`. + + .. versionadded:: 3.6 + .. data:: REG_RESOURCE_LIST A device-driver resource list. diff --git a/Doc/library/winsound.rst b/Doc/library/winsound.rst index d2c421047e..372f792a0f 100644 --- a/Doc/library/winsound.rst +++ b/Doc/library/winsound.rst @@ -25,7 +25,8 @@ provided by Windows platforms. It includes functions and several constants. .. function:: PlaySound(sound, flags) Call the underlying :c:func:`PlaySound` function from the Platform API. The - *sound* parameter may be a filename, audio data as a string, or ``None``. Its + *sound* parameter may be a filename, a system sound alias, audio data as a + :term:`bytes-like object`, or ``None``. Its interpretation depends on the value of *flags*, which can be a bitwise ORed combination of the constants described below. If the *sound* parameter is ``None``, any currently playing waveform sound is stopped. If the system @@ -39,7 +40,8 @@ provided by Windows platforms. It includes functions and several constants. sound to play; possible values are ``-1``, ``MB_ICONASTERISK``, ``MB_ICONEXCLAMATION``, ``MB_ICONHAND``, ``MB_ICONQUESTION``, and ``MB_OK``, all described below. The value ``-1`` produces a "simple beep"; this is the final - fallback if a sound cannot be played otherwise. + fallback if a sound cannot be played otherwise. If the system indicates an + error, :exc:`RuntimeError` is raised. .. data:: SND_FILENAME @@ -92,7 +94,7 @@ provided by Windows platforms. It includes functions and several constants. .. data:: SND_MEMORY The *sound* parameter to :func:`PlaySound` is a memory image of a WAV file, as a - string. + :term:`bytes-like object`. .. note:: diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index aad27a88c9..a1d446902b 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -133,9 +133,9 @@ parameter expect a WSGI-compliant dictionary to be supplied; please see for key, value in environ.items()] return ret - httpd = make_server('', 8000, simple_app) - print("Serving on port 8000...") - httpd.serve_forever() + with make_server('', 8000, simple_app) as httpd: + print("Serving on port 8000...") + httpd.serve_forever() In addition to the environment functions above, the :mod:`wsgiref.util` module @@ -285,14 +285,14 @@ request. (E.g., using the :func:`shift_path_info` function from from wsgiref.simple_server import make_server, demo_app - httpd = make_server('', 8000, demo_app) - print("Serving HTTP on port 8000...") + with make_server('', 8000, demo_app) as httpd: + print("Serving HTTP on port 8000...") - # Respond to requests until process is killed - httpd.serve_forever() + # Respond to requests until process is killed + httpd.serve_forever() - # Alternative: serve one request, then exit - httpd.handle_request() + # Alternative: serve one request, then exit + httpd.handle_request() .. function:: demo_app(environ, start_response) @@ -432,9 +432,9 @@ Paste" library. # This is the application wrapped in a validator validator_app = validator(simple_app) - httpd = make_server('', 8000, validator_app) - print("Listening on port 8000....") - httpd.serve_forever() + with make_server('', 8000, validator_app) as httpd: + print("Listening on port 8000....") + httpd.serve_forever() :mod:`wsgiref.handlers` -- server/gateway base classes @@ -774,8 +774,8 @@ This is a working "Hello World" WSGI application:: # The returned object is going to be printed return [b"Hello World"] - httpd = make_server('', 8000, hello_world_app) - print("Serving on port 8000...") + with make_server('', 8000, hello_world_app) as httpd: + print("Serving on port 8000...") - # Serve until process is killed - httpd.serve_forever() + # Serve until process is killed + httpd.serve_forever() diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index dad7a02082..390828ef3d 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -88,9 +88,13 @@ between conformable Python objects and XML on the wire. +======================+=======================================================+ | ``boolean`` | :class:`bool` | +----------------------+-------------------------------------------------------+ - | ``int`` or ``i4`` | :class:`int` in range from -2147483648 to 2147483647. | + | ``int``, ``i1``, | :class:`int` in range from -2147483648 to 2147483647. | + | ``i2``, ``i4``, | Values get the ``<int>`` tag. | + | ``i8`` or | | + | ``biginteger`` | | +----------------------+-------------------------------------------------------+ - | ``double`` | :class:`float` | + | ``double`` or | :class:`float`. Values get the ``<double>`` tag. | + | ``float`` | | +----------------------+-------------------------------------------------------+ | ``string`` | :class:`str` | +----------------------+-------------------------------------------------------+ @@ -114,6 +118,8 @@ between conformable Python objects and XML on the wire. | ``nil`` | The ``None`` constant. Passing is allowed only if | | | *allow_none* is true. | +----------------------+-------------------------------------------------------+ + | ``bigdecimal`` | :class:`decimal.Decimal`. Returned type only. | + +----------------------+-------------------------------------------------------+ This is the full set of data types supported by XML-RPC. Method calls may also raise a special :exc:`Fault` instance, used to signal XML-RPC server errors, or @@ -137,6 +143,13 @@ between conformable Python objects and XML on the wire. .. versionchanged:: 3.5 Added the *context* argument. + .. versionchanged:: 3.6 + Added support of type tags with prefixes (e.g. ``ex:nil``). + Added support of unmarsalling additional types used by Apache XML-RPC + implementation for numerics: ``i1``, ``i2``, ``i8``, ``biginteger``, + ``float`` and ``bigdecimal``. + See http://ws.apache.org/xmlrpc/types.html for a description. + .. seealso:: diff --git a/Doc/library/xmlrpc.server.rst b/Doc/library/xmlrpc.server.rst index 1c77e84c9b..0511ddf47e 100644 --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@ -148,29 +148,29 @@ Server code:: rpc_paths = ('/RPC2',) # Create server - server = SimpleXMLRPCServer(("localhost", 8000), - requestHandler=RequestHandler) - server.register_introspection_functions() + with SimpleXMLRPCServer(("localhost", 8000), + requestHandler=RequestHandler) as server: + server.register_introspection_functions() - # Register pow() function; this will use the value of - # pow.__name__ as the name, which is just 'pow'. - server.register_function(pow) + # Register pow() function; this will use the value of + # pow.__name__ as the name, which is just 'pow'. + server.register_function(pow) - # Register a function under a different name - def adder_function(x,y): - return x + y - server.register_function(adder_function, 'add') + # Register a function under a different name + def adder_function(x,y): + return x + y + server.register_function(adder_function, 'add') - # Register an instance; all the methods of the instance are - # published as XML-RPC methods (in this case, just 'mul'). - class MyFuncs: - def mul(self, x, y): - return x * y + # Register an instance; all the methods of the instance are + # published as XML-RPC methods (in this case, just 'mul'). + class MyFuncs: + def mul(self, x, y): + return x * y - server.register_instance(MyFuncs()) + server.register_instance(MyFuncs()) - # Run the server's main loop - server.serve_forever() + # Run the server's main loop + server.serve_forever() The following client code will call the methods made available by the preceding server:: @@ -207,18 +207,17 @@ a server allowing dotted names and registering a multicall function. def getCurrentTime(): return datetime.datetime.now() - server = SimpleXMLRPCServer(("localhost", 8000)) - server.register_function(pow) - server.register_function(lambda x,y: x+y, 'add') - server.register_instance(ExampleService(), allow_dotted_names=True) - server.register_multicall_functions() - print('Serving XML-RPC on localhost port 8000') - try: - server.serve_forever() - except KeyboardInterrupt: - print("\nKeyboard interrupt received, exiting.") - server.server_close() - sys.exit(0) + with SimpleXMLRPCServer(("localhost", 8000)) as server: + server.register_function(pow) + server.register_function(lambda x,y: x+y, 'add') + server.register_instance(ExampleService(), allow_dotted_names=True) + server.register_multicall_functions() + print('Serving XML-RPC on localhost port 8000') + try: + server.serve_forever() + except KeyboardInterrupt: + print("\nKeyboard interrupt received, exiting.") + sys.exit(0) This ExampleService demo can be invoked from the command line:: diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 1d6526ce95..47ba0b521e 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -147,10 +147,10 @@ ZipFile Objects *compression* is the ZIP compression method to use when writing the archive, and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized - values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED`, + values will cause :exc:`NotImplementedError` to be raised. If :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified but the corresponding module (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not available, :exc:`RuntimeError` - is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is + is raised. The default is :const:`ZIP_STORED`. If *allowZip64* is ``True`` (the default) zipfile will create ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GiB. If it is false :mod:`zipfile` will raise an exception when the ZIP file would require ZIP64 extensions. @@ -179,6 +179,10 @@ ZipFile Objects Added support for writing to unseekable streams. Added support for the ``'x'`` mode. + .. versionchanged:: 3.6 + Previously, a plain :exc:`RuntimeError` was raised for unrecognized + compression values. + .. method:: ZipFile.close() @@ -205,18 +209,12 @@ ZipFile Objects Return a list of archive members by name. -.. index:: - single: universal newlines; zipfile.ZipFile.open method - -.. method:: ZipFile.open(name, mode='r', pwd=None) +.. method:: ZipFile.open(name, mode='r', pwd=None, *, force_zip64=False) - Extract a member from the archive as a file-like object (ZipExtFile). *name* - is the name of the file in the archive, or a :class:`ZipInfo` object. The - *mode* parameter, if included, must be one of the following: ``'r'`` (the - default), ``'U'``, or ``'rU'``. Choosing ``'U'`` or ``'rU'`` will enable - :term:`universal newlines` support in the read-only object. *pwd* is the - password used for encrypted files. Calling :meth:`.open` on a closed - ZipFile will raise a :exc:`RuntimeError`. + Access a member of the archive as a binary file-like object. *name* + can be either the name of a file within the archive or a :class:`ZipInfo` + object. The *mode* parameter, if included, must be ``'r'`` (the default) + or ``'w'``. *pwd* is the password used to decrypt encrypted ZIP files. :meth:`~ZipFile.open` is also a context manager and therefore supports the :keyword:`with` statement:: @@ -225,17 +223,23 @@ ZipFile Objects with myzip.open('eggs.txt') as myfile: print(myfile.read()) - .. note:: + With *mode* ``'r'`` the file-like object + (``ZipExtFile``) is read-only and provides the following methods: + :meth:`~io.BufferedIOBase.read`, :meth:`~io.IOBase.readline`, + :meth:`~io.IOBase.readlines`, :meth:`__iter__`, + :meth:`~iterator.__next__`. These objects can operate independently of + the ZipFile. - The file-like object is read-only and provides the following methods: - :meth:`~io.BufferedIOBase.read`, :meth:`~io.IOBase.readline`, - :meth:`~io.IOBase.readlines`, :meth:`__iter__`, - :meth:`~iterator.__next__`. + With ``mode='w'``, a writable file handle is returned, which supports the + :meth:`~io.BufferedIOBase.write` method. While a writable file handle is open, + attempting to read or write other files in the ZIP file will raise a + :exc:`ValueError`. - .. note:: - - Objects returned by :meth:`.open` can operate independently of the - ZipFile. + When writing a file, if the file size is not known in advance but may exceed + 2 GiB, pass ``force_zip64=True`` to ensure that the header format is + capable of supporting large files. If the file size is known in advance, + construct a :class:`ZipInfo` object with :attr:`~ZipInfo.file_size` set, and + use that as the *name* parameter. .. note:: @@ -243,10 +247,19 @@ ZipFile Objects or a :class:`ZipInfo` object. You will appreciate this when trying to read a ZIP file that contains members with duplicate names. - .. deprecated-removed:: 3.4 3.6 - The ``'U'`` or ``'rU'`` mode. Use :class:`io.TextIOWrapper` for reading + .. versionchanged:: 3.6 + Removed support of ``mode='U'``. Use :class:`io.TextIOWrapper` for reading compressed text files in :term:`universal newlines` mode. + .. versionchanged:: 3.6 + :meth:`open` can now be used to write files into the archive with the + ``mode='w'`` option. + + .. versionchanged:: 3.6 + Calling :meth:`.open` on a closed ZipFile will raise a :exc:`ValueError`. + Previously, a :exc:`RuntimeError` was raised. + + .. method:: ZipFile.extract(member, path=None, pwd=None) Extract a member from the archive to the current working directory; *member* @@ -267,6 +280,10 @@ ZipFile Objects characters (``:``, ``<``, ``>``, ``|``, ``"``, ``?``, and ``*``) replaced by underscore (``_``). + .. versionchanged:: 3.6 + Calling :meth:`extract` on a closed ZipFile will raise a + :exc:`ValueError`. Previously, a :exc:`RuntimeError` was raised. + .. method:: ZipFile.extractall(path=None, members=None, pwd=None) @@ -283,6 +300,10 @@ ZipFile Objects dots ``".."``. This module attempts to prevent that. See :meth:`extract` note. + .. versionchanged:: 3.6 + Calling :meth:`extractall` on a closed ZipFile will raise a + :exc:`ValueError`. Previously, a :exc:`RuntimeError` was raised. + .. method:: ZipFile.printdir() @@ -300,18 +321,24 @@ ZipFile Objects file in the archive, or a :class:`ZipInfo` object. The archive must be open for read or append. *pwd* is the password used for encrypted files and, if specified, it will override the default password set with :meth:`setpassword`. Calling - :meth:`read` on a closed ZipFile will raise a :exc:`RuntimeError`. Calling :meth:`read` on a ZipFile that uses a compression method other than :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` will raise a :exc:`NotImplementedError`. An error will also be raised if the corresponding compression module is not available. + .. versionchanged:: 3.6 + Calling :meth:`read` on a closed ZipFile will raise a :exc:`ValueError`. + Previously, a :exc:`RuntimeError` was raised. + .. method:: ZipFile.testzip() Read all the files in the archive and check their CRC's and file headers. - Return the name of the first bad file, or else return ``None``. Calling - :meth:`testzip` on a closed ZipFile will raise a :exc:`RuntimeError`. + Return the name of the first bad file, or else return ``None``. + + .. versionchanged:: 3.6 + Calling :meth:`testfile` on a closed ZipFile will raise a + :exc:`ValueError`. Previously, a :exc:`RuntimeError` was raised. .. method:: ZipFile.write(filename, arcname=None, compress_type=None) @@ -321,10 +348,7 @@ ZipFile Objects letter and with leading path separators removed). If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for the new entry. - The archive must be open with mode ``'w'``, ``'x'`` or ``'a'`` -- calling - :meth:`write` on a ZipFile created with mode ``'r'`` will raise a - :exc:`RuntimeError`. Calling :meth:`write` on a closed ZipFile will raise a - :exc:`RuntimeError`. + The archive must be open with mode ``'w'``, ``'x'`` or ``'a'``. .. note:: @@ -343,16 +367,19 @@ ZipFile Objects If ``arcname`` (or ``filename``, if ``arcname`` is not given) contains a null byte, the name of the file in the archive will be truncated at the null byte. + .. versionchanged:: 3.6 + Calling :meth:`write` on a ZipFile created with mode ``'r'`` or + a closed ZipFile will raise a :exc:`ValueError`. Previously, + a :exc:`RuntimeError` was raised. + + .. method:: ZipFile.writestr(zinfo_or_arcname, data[, compress_type]) Write the string *data* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's an instance, at least the filename, date, and time must be given. If it's a name, the date and time is set to the current date and time. - The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'`` -- calling - :meth:`writestr` on a ZipFile created with mode ``'r'`` will raise a - :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will - raise a :exc:`RuntimeError`. + The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'``. If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for the new entry, or in the *zinfo_or_arcname* @@ -368,6 +395,12 @@ ZipFile Objects .. versionchanged:: 3.2 The *compress_type* argument. + .. versionchanged:: 3.6 + Calling :meth:`writestr` on a ZipFile created with mode ``'r'`` or + a closed ZipFile will raise a :exc:`ValueError`. Previously, + a :exc:`RuntimeError` was raised. + + The following data attributes are also available: @@ -465,7 +498,31 @@ Instances of the :class:`ZipInfo` class are returned by the :meth:`.getinfo` and :meth:`.infolist` methods of :class:`ZipFile` objects. Each object stores information about a single member of the ZIP archive. -Instances have the following attributes: +There is one classmethod to make a :class:`ZipInfo` instance for a filesystem +file: + +.. classmethod:: ZipInfo.from_file(filename, arcname=None) + + Construct a :class:`ZipInfo` instance for a file on the filesystem, in + preparation for adding it to a zip file. + + *filename* should be the path to a file or directory on the filesystem. + + If *arcname* is specified, it is used as the name within the archive. + If *arcname* is not specified, the name will be the same as *filename*, but + with any drive letter and leading path separators removed. + + .. versionadded:: 3.6 + +Instances have the following methods and attributes: + +.. method:: ZipInfo.is_dir() + + Return ``True`` if this archive member is a directory. + + This uses the entry's name: directories should always end with ``/``. + + .. versionadded:: 3.6 .. attribute:: ZipInfo.filename @@ -574,4 +631,5 @@ Instances have the following attributes: Size of the uncompressed file. + .. _PKZIP Application Note: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst index 1de7baee54..3d742ab35b 100644 --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -47,14 +47,19 @@ The available exception and functions in this module are: platforms, use ``adler32(data) & 0xffffffff``. -.. function:: compress(data[, level]) +.. function:: compress(data, level=-1) Compresses the bytes in *data*, returning a bytes object containing compressed data. - *level* is an integer from ``0`` to ``9`` controlling the level of compression; + *level* is an integer from ``0`` to ``9`` or ``-1`` controlling the level of compression; ``1`` is fastest and produces the least compression, ``9`` is slowest and - produces the most. ``0`` is no compression. The default value is ``6``. + produces the most. ``0`` is no compression. The default value is ``-1`` + (Z_DEFAULT_COMPRESSION). Z_DEFAULT_COMPRESSION represents a default + compromise between speed and compression (currently equivalent to level 6). Raises the :exc:`error` exception if any error occurs. + .. versionchanged:: 3.6 + *level* can now be used as a keyword parameter. + .. function:: compressobj(level=-1, method=DEFLATED, wbits=15, memLevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict]) @@ -124,7 +129,7 @@ The available exception and functions in this module are: platforms, use ``crc32(data) & 0xffffffff``. -.. function:: decompress(data[, wbits[, bufsize]]) +.. function:: decompress(data, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE) Decompresses the bytes in *data*, returning a bytes object containing the uncompressed data. The *wbits* parameter depends on @@ -159,14 +164,16 @@ The available exception and functions in this module are: When decompressing a stream, the window size must not be smaller than the size originally used to compress the stream; using a too-small value may result in an :exc:`error` exception. The default *wbits* value - is 15, which corresponds to the largest window size and requires a zlib - header and trailer to be included. + corresponds to the largest window size and requires a zlib header and + trailer to be included. *bufsize* is the initial size of the buffer used to hold decompressed data. If more space is required, the buffer size will be increased as needed, so you don't have to get this value exactly right; tuning it will only save a few calls - to :c:func:`malloc`. The default size is 16384. + to :c:func:`malloc`. + .. versionchanged:: 3.6 + *wbits* and *bufsize* can be used as keyword arguments. .. function:: decompressobj(wbits=15[, zdict]) @@ -252,7 +259,7 @@ Decompression objects support the following methods and attributes: .. versionadded:: 3.3 -.. method:: Decompress.decompress(data[, max_length]) +.. method:: Decompress.decompress(data, max_length=0) Decompress *data*, returning a bytes object containing the uncompressed data corresponding to at least part of the data in *string*. This data should be @@ -264,9 +271,11 @@ Decompression objects support the following methods and attributes: no longer than *max_length*. This may mean that not all of the compressed input can be processed; and unconsumed data will be stored in the attribute :attr:`unconsumed_tail`. This bytestring must be passed to a subsequent call to - :meth:`decompress` if decompression is to continue. If *max_length* is not - supplied then the whole input is decompressed, and :attr:`unconsumed_tail` is - empty. + :meth:`decompress` if decompression is to continue. If *max_length* is zero + then the whole input is decompressed, and :attr:`unconsumed_tail` is empty. + + .. versionchanged:: 3.6 + *max_length* can be used as a keyword argument. .. method:: Decompress.flush([length]) diff --git a/Doc/make.bat b/Doc/make.bat index 5ab80850a3..da1f8765a4 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -40,7 +40,7 @@ if "%1" == "clean" ( goto end
)
-%SPHINXBUILD% 2> nul
+%SPHINXBUILD% >nul 2> nul
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 88b94eaa95..4fc6af02f1 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -471,10 +471,10 @@ A function definition defines a user-defined function object (see section decorators: `decorator`+ decorator: "@" `dotted_name` ["(" [`argument_list` [","]] ")"] NEWLINE dotted_name: `identifier` ("." `identifier`)* - parameter_list: (`defparameter` ",")* - : | "*" [`parameter`] ("," `defparameter`)* ["," "**" `parameter`] - : | "**" `parameter` - : | `defparameter` [","] ) + parameter_list: `defparameter` ("," `defparameter`)* ["," [`parameter_list_starargs`]] + : | `parameter_list_starargs` + parameter_list_starargs: "*" [`parameter`] ("," `defparameter`)* ["," ["**" `parameter` [","]]] + : | "**" `parameter` [","] parameter: `identifier` [":" `expression`] defparameter: `parameter` ["=" `expression`] funcname: `identifier` @@ -546,11 +546,12 @@ Function call semantics are described in more detail in section :ref:`calls`. A function call always assigns values to all parameters mentioned in the parameter list, either from position arguments, from keyword arguments, or from default values. If the form "``*identifier``" is present, it is initialized to a tuple -receiving any excess positional parameters, defaulting to the empty tuple. If -the form "``**identifier``" is present, it is initialized to a new dictionary -receiving any excess keyword arguments, defaulting to a new empty dictionary. -Parameters after "``*``" or "``*identifier``" are keyword-only parameters and -may only be passed used keyword arguments. +receiving any excess positional parameters, defaulting to the empty tuple. +If the form "``**identifier``" is present, it is initialized to a new +ordered mapping receiving any excess keyword arguments, defaulting to a +new empty mapping of the same type. Parameters after "``*``" or +"``*identifier``" are keyword-only parameters and may only be passed +used keyword arguments. .. index:: pair: function; annotations @@ -632,6 +633,11 @@ list for the base classes and the saved local namespace for the attribute dictionary. The class name is bound to this class object in the original local namespace. +The order in which attributes are defined in the class body is preserved +in the new class's ``__dict__``. Note that this is reliable only right +after the class is created and only for classes that were defined using +the definition syntax. + Class creation can be customized heavily using :ref:`metaclasses <metaclasses>`. Classes can also be decorated: just like when decorating functions, :: diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index b763f10eeb..9619e53d4d 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -686,33 +686,36 @@ Modules Attribute assignment updates the module's namespace dictionary, e.g., ``m.x = 1`` is equivalent to ``m.__dict__["x"] = 1``. - .. index:: single: __dict__ (module attribute) - - Special read-only attribute: :attr:`~object.__dict__` is the module's namespace as a - dictionary object. - - .. impl-detail:: - - Because of the way CPython clears module dictionaries, the module - dictionary will be cleared when the module falls out of scope even if the - dictionary still has live references. To avoid this, copy the dictionary - or keep the module around while using its dictionary directly. - .. index:: single: __name__ (module attribute) single: __doc__ (module attribute) single: __file__ (module attribute) + single: __annotations__ (module attribute) pair: module; namespace Predefined (writable) attributes: :attr:`__name__` is the module's name; :attr:`__doc__` is the module's documentation string, or ``None`` if - unavailable; :attr:`__file__` is the pathname of the file from which the + unavailable; :attr:`__annotations__` (optional) is a dictionary containing + :term:`variable annotations <variable annotation>` collected during module + body execution; :attr:`__file__` is the pathname of the file from which the module was loaded, if it was loaded from a file. The :attr:`__file__` attribute may be missing for certain types of modules, such as C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file. + .. index:: single: __dict__ (module attribute) + + Special read-only attribute: :attr:`~object.__dict__` is the module's + namespace as a dictionary object. + + .. impl-detail:: + + Because of the way CPython clears module dictionaries, the module + dictionary will be cleared when the module falls out of scope even if the + dictionary still has live references. To avoid this, copy the dictionary + or keep the module around while using its dictionary directly. + Custom classes Custom class types are typically created by class definitions (see section :ref:`class`). A class has a namespace implemented by a dictionary object. @@ -761,13 +764,17 @@ Custom classes single: __dict__ (class attribute) single: __bases__ (class attribute) single: __doc__ (class attribute) + single: __annotations__ (class attribute) Special attributes: :attr:`~definition.__name__` is the class name; :attr:`__module__` is the module name in which the class was defined; :attr:`~object.__dict__` is the dictionary containing the class's namespace; :attr:`~class.__bases__` is a tuple (possibly empty or a singleton) containing the base classes, in the order of their occurrence in the base class list; :attr:`__doc__` is the - class's documentation string, or ``None`` if undefined. + class's documentation string, or ``None`` if undefined; + :attr:`__annotations__` (optional) is a dictionary containing + :term:`variable annotations <variable annotation>` collected during + class body execution. Class instances .. index:: @@ -1063,6 +1070,12 @@ to ``type(x).__getitem__(x, i)``. Except where mentioned, attempts to execute a operation raise an exception when no appropriate method is defined (typically :exc:`AttributeError` or :exc:`TypeError`). +Setting a special method to ``None`` indicates that the corresponding +operation is not available. For example, if a class sets +:meth:`__iter__` to ``None``, the class is not iterable, so calling +:func:`iter` on its instances will raise a :exc:`TypeError` (without +falling back to :meth:`__getitem__`). [#]_ + When implementing a class that emulates any built-in type, it is important that the emulation only be implemented to the degree that it makes sense for the object being modelled. For example, some sequences may work well with retrieval @@ -1233,8 +1246,9 @@ Basic customization .. method:: object.__format__(self, format_spec) - Called by the :func:`format` built-in function (and by extension, the - :meth:`str.format` method of class :class:`str`) to produce a "formatted" + Called by the :func:`format` built-in function, + and by extension, evaluation of :ref:`formatted string literals + <f-strings>` and the :meth:`str.format` method, to produce a "formatted" string representation of an object. The ``format_spec`` argument is a string that contains a description of the formatting options desired. The interpretation of the ``format_spec`` argument is up to the type @@ -1491,6 +1505,14 @@ class' :attr:`~object.__dict__`. Called to delete the attribute on an instance *instance* of the owner class. +.. method:: object.__set_name__(self, owner, name) + + Called at the time the owning class *owner* is created. The + descriptor has been assigned to *name*. + + .. versionadded:: 3.6 + + The attribute :attr:`__objclass__` is interpreted by the :mod:`inspect` module as specifying the class where this object was defined (setting this appropriately can assist in runtime introspection of dynamic class attributes). @@ -1628,11 +1650,56 @@ Notes on using *__slots__* * *__class__* assignment works only if both classes have the same *__slots__*. -.. _metaclasses: +.. _class-customization: Customizing class creation -------------------------- +Whenever a class inherits from another class, *__init_subclass__* is +called on that class. This way, it is possible to write classes which +change the behavior of subclasses. This is closely related to class +decorators, but where class decorators only affect the specific class they're +applied to, ``__init_subclass__`` solely applies to future subclasses of the +class defining the method. + +.. classmethod:: object.__init_subclass__(cls) + + This method is called whenever the containing class is subclassed. + *cls* is then the new subclass. If defined as a normal instance method, + this method is implicitly converted to a class method. + + Keyword arguments which are given to a new class are passed to + the parent's class ``__init_subclass__``. For compatibility with + other classes using ``__init_subclass__``, one should take out the + needed keyword arguments and pass the others over to the base + class, as in:: + + class Philosopher: + def __init_subclass__(cls, default_name, **kwargs): + super().__init_subclass__(**kwargs) + cls.default_name = default_name + + class AustralianPhilosopher(Philosopher, default_name="Bruce"): + pass + + The default implementation ``object.__init_subclass__`` does + nothing, but raises an error if it is called with any arguments. + + .. note:: + + The metaclass hint ``metaclass`` is consumed by the rest of the type + machinery, and is never passed to ``__init_subclass__`` implementations. + The actual metaclass (rather than the explicit hint) can be accessed as + ``type(cls)``. + + .. versionadded:: 3.6 + + +.. _metaclasses: + +Metaclasses +^^^^^^^^^^^ + By default, classes are constructed using :func:`type`. The class body is executed in a new namespace and the class name is bound locally to the result of ``type(name, bases, namespace)``. @@ -1690,7 +1757,7 @@ as ``namespace = metaclass.__prepare__(name, bases, **kwds)`` (where the additional keyword arguments, if any, come from the class definition). If the metaclass has no ``__prepare__`` attribute, then the class namespace -is initialised as an empty :func:`dict` instance. +is initialised as an empty ordered mapping. .. seealso:: @@ -1734,9 +1801,9 @@ included in the class definition (if any) and the resulting object is bound in the local namespace as the defined class. When a new class is created by ``type.__new__``, the object provided as the -namespace parameter is copied to a standard Python dictionary and the original -object is discarded. The new copy becomes the :attr:`~object.__dict__` attribute -of the class object. +namespace parameter is copied to a new ordered mapping and the original +object is discarded. The new copy is wrapped in a read-only proxy, which +becomes the :attr:`~object.__dict__` attribute of the class object. .. seealso:: @@ -2059,7 +2126,7 @@ left undefined. (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected (swapped) operands. These functions are only called if the left operand does - not support the corresponding operation and the operands are of different + not support the corresponding operation [#]_ and the operands are of different types. [#]_ For instance, to evaluate the expression ``x - y``, where *y* is an instance of a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` is called if ``x.__sub__(y)`` returns *NotImplemented*. @@ -2475,6 +2542,17 @@ An example of an asynchronous context manager class:: controlled conditions. It generally isn't a good idea though, since it can lead to some very strange behaviour if it is handled incorrectly. +.. [#] The :meth:`__hash__`, :meth:`__iter__`, :meth:`__reversed__`, and + :meth:`__contains__` methods have special handling for this; others + will still raise a :exc:`TypeError`, but may do so by relying on + the behavior that ``None`` is not callable. + +.. [#] "Does not support" here means that the class has no such method, or + the method returns ``NotImplemented``. Do not set the method to + ``None`` if you want to force fallback to the right operand's reflected + method--that will instead have the opposite effect of explicitly + *blocking* such fallback. + .. [#] For operands of the same type, it is assumed that if the non-reflected method (such as :meth:`__add__`) fails the operation is not supported, which is why the reflected method is not called. diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 64302b89a8..5e2c1c8b07 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -36,7 +36,7 @@ implement import semantics. When a module is first imported, Python searches for the module and if found, it creates a module object [#fnmo]_, initializing it. If the named module -cannot be found, an :exc:`ImportError` is raised. Python implements various +cannot be found, an :exc:`ModuleNotFoundError` is raised. Python implements various strategies to search for the named module when the import machinery is invoked. These strategies can be modified and extended by using various hooks described in the sections below. @@ -167,7 +167,7 @@ arguments to the :keyword:`import` statement, or from the parameters to the This name will be used in various phases of the import search, and it may be the dotted path to a submodule, e.g. ``foo.bar.baz``. In this case, Python first tries to import ``foo``, then ``foo.bar``, and finally ``foo.bar.baz``. -If any of the intermediate imports fail, an :exc:`ImportError` is raised. +If any of the intermediate imports fail, an :exc:`ModuleNotFoundError` is raised. The module cache @@ -186,7 +186,7 @@ object. During import, the module name is looked up in :data:`sys.modules` and if present, the associated value is the module satisfying the import, and the process completes. However, if the value is ``None``, then an -:exc:`ImportError` is raised. If the module name is missing, Python will +:exc:`ModuleNotFoundError` is raised. If the module name is missing, Python will continue searching for the module. :data:`sys.modules` is writable. Deleting a key may not destroy the @@ -194,7 +194,7 @@ associated module (as other modules may hold references to it), but it will invalidate the cache entry for the named module, causing Python to search anew for the named module upon its next import. The key can also be assigned to ``None``, forcing the next import -of the module to result in an :exc:`ImportError`. +of the module to result in an :exc:`ModuleNotFoundError`. Beware though, as if you keep a reference to the module object, invalidate its cache entry in :data:`sys.modules`, and then re-import the @@ -288,8 +288,8 @@ the named module or not. If the meta path finder knows how to handle the named module, it returns a spec object. If it cannot handle the named module, it returns ``None``. If :data:`sys.meta_path` processing reaches the end of its list without returning -a spec, then an :exc:`ImportError` is raised. Any other exceptions raised -are simply propagated up, aborting the import process. +a spec, then a :exc:`ModuleNotFoundError` is raised. Any other exceptions +raised are simply propagated up, aborting the import process. The :meth:`~importlib.abc.MetaPathFinder.find_spec()` method of meta path finders is called with two or three arguments. The first is the fully @@ -298,9 +298,9 @@ The second argument is the path entries to use for the module search. For top-level modules, the second argument is ``None``, but for submodules or subpackages, the second argument is the value of the parent package's ``__path__`` attribute. If the appropriate ``__path__`` attribute cannot -be accessed, an :exc:`ImportError` is raised. The third argument is an -existing module object that will be the target of loading later. The -import system passes in a target module only during reload. +be accessed, an :exc:`ModuleNotFoundError` is raised. The third argument +is an existing module object that will be the target of loading later. +The import system passes in a target module only during reload. The meta path may be traversed multiple times for a single import request. For example, assuming none of the modules involved has already been cached, @@ -554,19 +554,30 @@ the module. details. This attribute is used instead of ``__name__`` to calculate explicit - relative imports for main modules, as defined in :pep:`366`. + relative imports for main modules, as defined in :pep:`366`. It is + expected to have the same value as ``__spec__.parent``. + + .. versionchanged:: 3.6 + The value of ``__package__`` is expected to be the same as + ``__spec__.parent``. .. attribute:: __spec__ The ``__spec__`` attribute must be set to the module spec that was - used when importing the module. This is used primarily for - introspection and during reloading. Setting ``__spec__`` + used when importing the module. Setting ``__spec__`` appropriately applies equally to :ref:`modules initialized during interpreter startup <programs>`. The one exception is ``__main__``, where ``__spec__`` is :ref:`set to None in some cases <main_spec>`. + When ``__package__`` is not defined, ``__spec__.parent`` is used as + a fallback. + .. versionadded:: 3.4 + .. versionchanged:: 3.6 + ``__spec__.parent`` is used as a fallback when ``__package__`` is + not defined. + .. attribute:: __path__ If the module is a package (either regular or namespace), the module @@ -876,7 +887,7 @@ import statements within that module. To selectively prevent import of some modules from a hook early on the meta path (rather than disabling the standard import system entirely), -it is sufficient to raise :exc:`ImportError` directly from +it is sufficient to raise :exc:`ModuleNoFoundError` directly from :meth:`~importlib.abc.MetaPathFinder.find_spec` instead of returning ``None``. The latter indicates that the meta path search should continue, while raising an exception terminates it immediately. diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 37f25f1def..a7c6a684c0 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -405,7 +405,8 @@ String literals are described by the following lexical definitions: .. productionlist:: stringliteral: [`stringprefix`](`shortstring` | `longstring`) - stringprefix: "r" | "u" | "R" | "U" + stringprefix: "r" | "u" | "R" | "U" | "f" | "F" + : | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF" shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"' longstring: "'''" `longstringitem`* "'''" | '"""' `longstringitem`* '"""' shortstringitem: `shortstringchar` | `stringescapeseq` @@ -464,6 +465,11 @@ is not supported. to simplify the maintenance of dual Python 2.x and 3.x codebases. See :pep:`414` for more information. +A string literal with ``'f'`` or ``'F'`` in its prefix is a +:dfn:`formatted string literal`; see :ref:`f-strings`. The ``'f'`` may be +combined with ``'r'``, but not with ``'b'`` or ``'u'``, therefore raw +formatted strings are possible, but formatted bytes literals are not. + In triple-quoted literals, unescaped newlines and quotes are allowed (and are retained), except that three unescaped quotes in a row terminate the literal. (A "quote" is the character used to open the literal, i.e. either ``'`` or ``"``.) @@ -554,6 +560,10 @@ is more easily recognized as broken.) It is also important to note that the escape sequences only recognized in string literals fall into the category of unrecognized escapes for bytes literals. + .. versionchanged:: 3.6 + Unrecognized escape sequences produce a DeprecationWarning. In + some future version of Python they will be a SyntaxError. + Even in a raw literal, quotes can be escaped with a backslash, but the backslash remains in the result; for example, ``r"\""`` is a valid string literal consisting of two characters: a backslash and a double quote; ``r"\"`` @@ -583,7 +593,106 @@ comments to parts of strings, for example:: Note that this feature is defined at the syntactical level, but implemented at compile time. The '+' operator must be used to concatenate string expressions at run time. Also note that literal concatenation can use different quoting -styles for each component (even mixing raw strings and triple quoted strings). +styles for each component (even mixing raw strings and triple quoted strings), +and formatted string literals may be concatenated with plain string literals. + + +.. index:: + single: formatted string literal + single: interpolated string literal + single: string; formatted literal + single: string; interpolated literal + single: f-string +.. _f-strings: + +Formatted string literals +------------------------- + +.. versionadded:: 3.6 + +A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal +that is prefixed with ``'f'`` or ``'F'``. These strings may contain +replacement fields, which are expressions delimited by curly braces ``{}``. +While other string literals always have a constant value, formatted strings +are really expressions evaluated at run time. + +Escape sequences are decoded like in ordinary string literals (except when +a literal is also marked as a raw string). After decoding, the grammar +for the contents of the string is: + +.. productionlist:: + f_string: (`literal_char` | "{{" | "}}" | `replacement_field`)* + replacement_field: "{" `f_expression` ["!" `conversion`] [":" `format_spec`] "}" + f_expression: (`conditional_expression` | "*" `or_expr`) + : ("," `conditional_expression` | "," "*" `or_expr`)* [","] + : | `yield_expression` + conversion: "s" | "r" | "a" + format_spec: (`literal_char` | NULL | `replacement_field`)* + literal_char: <any code point except "{", "}" or NULL> + +The parts of the string outside curly braces are treated literally, +except that any doubled curly braces ``'{{'`` or ``'}}'`` are replaced +with the corresponding single curly brace. A single opening curly +bracket ``'{'`` marks a replacement field, which starts with a +Python expression. After the expression, there may be a conversion field, +introduced by an exclamation point ``'!'``. A format specifier may also +be appended, introduced by a colon ``':'``. A replacement field ends +with a closing curly bracket ``'}'``. + +Expressions in formatted string literals are treated like regular +Python expressions surrounded by parentheses, with a few exceptions. +An empty expression is not allowed, and a :keyword:`lambda` expression +must be surrounded by explicit parentheses. Replacement expressions +can contain line breaks (e.g. in triple-quoted strings), but they +cannot contain comments. Each expression is evaluated in the context +where the formatted string literal appears, in order from left to right. + +If a conversion is specified, the result of evaluating the expression +is converted before formatting. Conversion ``'!s'`` calls :func:`str` on +the result, ``'!r'`` calls :func:`repr`, and ``'!a'`` calls :func:`ascii`. + +The result is then formatted using the :func:`format` protocol. The +format specifier is passed to the :meth:`__format__` method of the +expression or conversion result. An empty string is passed when the +format specifier is omitted. The formatted result is then included in +the final value of the whole string. + +Top-level format specifiers may include nested replacement fields. +These nested fields may include their own conversion fields and +format specifiers, but may not include more deeply-nested replacement fields. + +Formatted string literals may be concatenated, but replacement fields +cannot be split across literals. + +Some examples of formatted string literals:: + + >>> name = "Fred" + >>> f"He said his name is {name!r}." + "He said his name is 'Fred'." + >>> f"He said his name is {repr(name)}." # repr() is equivalent to !r + "He said his name is 'Fred'." + >>> width = 10 + >>> precision = 4 + >>> value = decimal.Decimal("12.34567") + >>> f"result: {value:{width}.{precision}}" # nested fields + 'result: 12.35' + +A consequence of sharing the same syntax as regular string literals is +that characters in the replacement fields must not conflict with the +quoting used in the outer formatted string literal. Also, escape +sequences normally apply to the outer formatted string literal, +rather than inner string literals:: + + f"abc {a["x"]} def" # error: outer string literal ended prematurely + f"abc {a[\"x\"]} def" # workaround: escape the inner quotes + f"abc {a['x']} def" # workaround: use different quoting + + f"newline: {ord('\n')}" # error: literal line break in inner string + f"newline: {ord('\\n')}" # workaround: double escaping + fr"newline: {ord('\n')}" # workaround: raw outer string + +See also :pep:`498` for the proposal that added formatted string literals, +and :meth:`str.format`, which uses a related format string mechanism. .. _numbers: @@ -612,20 +721,24 @@ Integer literals Integer literals are described by the following lexical definitions: .. productionlist:: - integer: `decimalinteger` | `octinteger` | `hexinteger` | `bininteger` - decimalinteger: `nonzerodigit` `digit`* | "0"+ + integer: `decinteger` | `bininteger` | `octinteger` | `hexinteger` + decinteger: `nonzerodigit` (["_"] `digit`)* | "0"+ (["_"] "0")* + bininteger: "0" ("b" | "B") (["_"] `bindigit`)+ + octinteger: "0" ("o" | "O") (["_"] `octdigit`)+ + hexinteger: "0" ("x" | "X") (["_"] `hexdigit`)+ nonzerodigit: "1"..."9" digit: "0"..."9" - octinteger: "0" ("o" | "O") `octdigit`+ - hexinteger: "0" ("x" | "X") `hexdigit`+ - bininteger: "0" ("b" | "B") `bindigit`+ + bindigit: "0" | "1" octdigit: "0"..."7" hexdigit: `digit` | "a"..."f" | "A"..."F" - bindigit: "0" | "1" There is no limit for the length of integer literals apart from what can be stored in available memory. +Underscores are ignored for determining the numeric value of the literal. They +can be used to group digits for enhanced readability. One underscore can occur +between digits, and after base specifiers like ``0x``. + Note that leading zeros in a non-zero decimal number are not allowed. This is for disambiguation with C-style octal literals, which Python used before version 3.0. @@ -634,6 +747,10 @@ Some examples of integer literals:: 7 2147483647 0o177 0b100110111 3 79228162514264337593543950336 0o377 0xdeadbeef + 100_000_000_000 0b_1110_0101 + +.. versionchanged:: 3.6 + Underscores are now allowed for grouping purposes in literals. .. _floating: @@ -645,23 +762,28 @@ Floating point literals are described by the following lexical definitions: .. productionlist:: floatnumber: `pointfloat` | `exponentfloat` - pointfloat: [`intpart`] `fraction` | `intpart` "." - exponentfloat: (`intpart` | `pointfloat`) `exponent` - intpart: `digit`+ - fraction: "." `digit`+ - exponent: ("e" | "E") ["+" | "-"] `digit`+ + pointfloat: [`digitpart`] `fraction` | `digitpart` "." + exponentfloat: (`digitpart` | `pointfloat`) `exponent` + digitpart: `digit` (["_"] `digit`)* + fraction: "." `digitpart` + exponent: ("e" | "E") ["+" | "-"] `digitpart` Note that the integer and exponent parts are always interpreted using radix 10. For example, ``077e010`` is legal, and denotes the same number as ``77e10``. The -allowed range of floating point literals is implementation-dependent. Some -examples of floating point literals:: +allowed range of floating point literals is implementation-dependent. As in +integer literals, underscores are supported for digit grouping. - 3.14 10. .001 1e100 3.14e-10 0e0 +Some examples of floating point literals:: + + 3.14 10. .001 1e100 3.14e-10 0e0 3.14_15_93 Note that numeric literals do not include a sign; a phrase like ``-1`` is actually an expression composed of the unary operator ``-`` and the literal ``1``. +.. versionchanged:: 3.6 + Underscores are now allowed for grouping purposes in literals. + .. _imaginary: @@ -671,7 +793,7 @@ Imaginary literals Imaginary literals are described by the following lexical definitions: .. productionlist:: - imagnumber: (`floatnumber` | `intpart`) ("j" | "J") + imagnumber: (`floatnumber` | `digitpart`) ("j" | "J") An imaginary literal yields a complex number with a real part of 0.0. Complex numbers are represented as a pair of floating point numbers and have the same @@ -679,7 +801,7 @@ restrictions on their range. To create a complex number with a nonzero real part, add a floating point number to it, e.g., ``(3+4j)``. Some examples of imaginary literals:: - 3.14j 10.j 10j .001j 1e100j 3.14e-10j + 3.14j 10.j 10j .001j 1e100j 3.14e-10j 3.14_15_93j .. _operators: diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index d403c4d546..3dc4418f97 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -16,6 +16,7 @@ simple statements is: : | `assert_stmt` : | `assignment_stmt` : | `augmented_assignment_stmt` + : | `annotated_assignment_stmt` : | `pass_stmt` : | `del_stmt` : | `return_stmt` @@ -84,7 +85,7 @@ attributes or items of mutable objects: assignment_stmt: (`target_list` "=")+ (`starred_expression` | `yield_expression`) target_list: `target` ("," `target`)* [","] target: `identifier` - : | "(" `target_list` ")" + : | "(" [`target_list`] ")" : | "[" [`target_list`] "]" : | `attributeref` : | `subscription` @@ -312,6 +313,50 @@ For targets which are attribute references, the same :ref:`caveat about class and instance attributes <attr-target-note>` applies as for regular assignments. +.. _annassign: + +Annotated assignment statements +------------------------------- + +.. index:: + pair: annotated; assignment + single: statement; assignment, annotated + +Annotation assignment is the combination, in a single statement, +of a variable or attribute annotation and an optional assignment statement: + +.. productionlist:: + annotated_assignment_stmt: `augtarget` ":" `expression` ["=" `expression`] + +The difference from normal :ref:`assignment` is that only single target and +only single right hand side value is allowed. + +For simple names as assignment targets, if in class or module scope, +the annotations are evaluated and stored in a special class or module +attribute :attr:`__annotations__` +that is a dictionary mapping from variable names (mangled if private) to +evaluated annotations. This attribute is writable and is automatically +created at the start of class or module body execution, if annotations +are found statically. + +For expressions as assignment targets, the annotations are evaluated if +in class or module scope, but not stored. + +If a name is annotated in a function scope, then this name is local for +that scope. Annotations are never evaluated and stored in function scopes. + +If the right hand side is present, an annotated +assignment performs the actual assignment before evaluating annotations +(where applicable). If the right hand side is not present for an expression +target, then the interpreter evaluates the target except for the last +:meth:`__setitem__` or :meth:`__setattr__` call. + +.. seealso:: + + :pep:`526` - Variable and attribute annotation syntax + :pep:`484` - Type hints + + .. _assert: The :keyword:`assert` statement @@ -859,11 +904,12 @@ block textually preceding that :keyword:`global` statement. Names listed in a :keyword:`global` statement must not be defined as formal parameters or in a :keyword:`for` loop control target, :keyword:`class` -definition, function definition, or :keyword:`import` statement. +definition, function definition, :keyword:`import` statement, or variable +annotation. .. impl-detail:: - The current implementation does not enforce the two restrictions, but + The current implementation does not enforce some of these restriction, but programs should not abuse this freedom, as future implementations may enforce them or silently change the meaning of the program. diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 63112830ba..273191bbd3 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -34,7 +34,7 @@ import suspicious ISSUE_URI = 'https://bugs.python.org/issue%s' -SOURCE_URI = 'https://hg.python.org/cpython/file/3.5/%s' +SOURCE_URI = 'https://hg.python.org/cpython/file/3.6/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists from docutils.parsers.rst.states import Body diff --git a/Doc/tools/rstlint.py b/Doc/tools/rstlint.py index de78041f77..2c1816eedf 100755 --- a/Doc/tools/rstlint.py +++ b/Doc/tools/rstlint.py @@ -9,8 +9,6 @@ # TODO: - wrong versions in versionadded/changed # - wrong markup after versionchanged directive -from __future__ import with_statement - import os import re import sys diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index dba93bf3a7..5de55a25e8 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -25,6 +25,30 @@ howto/curses,,:blue,"2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white. howto/curses,,:magenta,"2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white. The" howto/curses,,:cyan,"2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white. The" howto/curses,,:white,"2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white. The" +howto/instrumentation,,::,python$target:::function-entry +howto/instrumentation,,:function,python$target:::function-entry +howto/instrumentation,,::,python$target:::function-return +howto/instrumentation,,:function,python$target:::function-return +howto/instrumentation,,:call,156641360502280 function-entry:call_stack.py:start:23 +howto/instrumentation,,:start,156641360502280 function-entry:call_stack.py:start:23 +howto/instrumentation,,:function,156641360518804 function-entry: call_stack.py:function_1:1 +howto/instrumentation,,:function,156641360532797 function-entry: call_stack.py:function_3:9 +howto/instrumentation,,:function,156641360546807 function-return: call_stack.py:function_3:10 +howto/instrumentation,,:function,156641360563367 function-return: call_stack.py:function_1:2 +howto/instrumentation,,:function,156641360578365 function-entry: call_stack.py:function_2:5 +howto/instrumentation,,:function,156641360591757 function-entry: call_stack.py:function_1:1 +howto/instrumentation,,:function,156641360605556 function-entry: call_stack.py:function_3:9 +howto/instrumentation,,:function,156641360617482 function-return: call_stack.py:function_3:10 +howto/instrumentation,,:function,156641360629814 function-return: call_stack.py:function_1:2 +howto/instrumentation,,:function,156641360642285 function-return: call_stack.py:function_2:6 +howto/instrumentation,,:function,156641360656770 function-entry: call_stack.py:function_3:9 +howto/instrumentation,,:function,156641360669707 function-return: call_stack.py:function_3:10 +howto/instrumentation,,:function,156641360687853 function-entry: call_stack.py:function_4:13 +howto/instrumentation,,:function,156641360700719 function-return: call_stack.py:function_4:14 +howto/instrumentation,,:function,156641360719640 function-entry: call_stack.py:function_5:18 +howto/instrumentation,,:function,156641360732567 function-return: call_stack.py:function_5:21 +howto/instrumentation,,:call,156641360747370 function-return:call_stack.py:start:28 +howto/instrumentation,,:start,156641360747370 function-return:call_stack.py:start:28 howto/ipaddress,,:DB8,>>> ipaddress.ip_address('2001:DB8::1') howto/ipaddress,,::,>>> ipaddress.ip_address('2001:DB8::1') howto/ipaddress,,:db8,IPv6Address('2001:db8::1') @@ -106,6 +130,10 @@ library/exceptions,,:err,err.object[err.start:err.end] library/functions,,:step,a[start:stop:step] library/functions,,:stop,"a[start:stop, i]" library/functions,,:stop,a[start:stop:step] +library/hashlib-blake2,,:vatrogasac,>>> cookie = b'user:vatrogasac' +library/hashlib-blake2,,:vatrogasac,"user:vatrogasac,349cf904533767ed2d755279a8df84d0" +library/hashlib-blake2,,:policajac,">>> compare_digest(b'user:policajac', sig)" +library/hashlib-blake2,,:LEAF,"h00 = blake2b(buf[0:LEAF_SIZE], fanout=FANOUT, depth=DEPTH," library/http.client,,:port,host:port library/http.cookies,,`,!#$%&'*+-.^_`|~: library/imaplib,,:MM,"""DD-Mmm-YYYY HH:MM:SS" @@ -205,7 +233,8 @@ library/uuid,,:uuid,urn:uuid:12345678-1234-5678-1234-567812345678 library/venv,,:param,":param nodist: If True, setuptools and pip are not installed into the" library/venv,,:param,":param progress: If setuptools or pip are installed, the progress of the" library/venv,,:param,":param nopip: If True, pip is not installed into the created" -library/venv,,:param,:param context: The information for the environment creation request +library/venv,,:param,:param context: The information for the virtual environment +library/xmlrpc.client,,:nil,ex:nil library/xmlrpc.client,,:pass,http://user:pass@host:port/path library/xmlrpc.client,,:pass,user:pass library/xmlrpc.client,,:port,http://user:pass@host:port/path @@ -239,7 +268,7 @@ using/cmdline,,:line,file:line: category: message using/cmdline,,:message,action:message:category:module:line using/cmdline,,:module,action:message:category:module:line using/unix,,:Packaging,https://en.opensuse.org/Portal:Packaging -whatsnew/2.0,418,:len, +whatsnew/2.0,,:len, whatsnew/2.3,,::, whatsnew/2.3,,:config, whatsnew/2.3,,:Critical, @@ -272,27 +301,27 @@ whatsnew/3.2,,:feed,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe: whatsnew/3.2,,:gz,">>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:" whatsnew/3.2,,:location,zope9-location = ${zope9:location} whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf -whatsnew/changelog,,:gz,": TarFile opened with external fileobj and ""w:gz"" mode didn't" -whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" -library/tarfile,149,:xz,'x:xz' -library/xml.etree.elementtree,290,:sometag,prefix:sometag -library/xml.etree.elementtree,301,:fictional,"<actors xmlns:fictional=""http://characters.example.com""" -library/xml.etree.elementtree,301,:character,<fictional:character>Lancelot</fictional:character> -library/xml.etree.elementtree,301,:character,<fictional:character>Archie Leach</fictional:character> -library/xml.etree.elementtree,301,:character,<fictional:character>Sir Robin</fictional:character> -library/xml.etree.elementtree,301,:character,<fictional:character>Gunther</fictional:character> -library/xml.etree.elementtree,301,:character,<fictional:character>Commander Clement</fictional:character> +library/tarfile,,:xz,'x:xz' +library/xml.etree.elementtree,,:sometag,prefix:sometag +library/xml.etree.elementtree,,:fictional,"<actors xmlns:fictional=""http://characters.example.com""" +library/xml.etree.elementtree,,:character,<fictional:character>Lancelot</fictional:character> +library/xml.etree.elementtree,,:character,<fictional:character>Archie Leach</fictional:character> +library/xml.etree.elementtree,,:character,<fictional:character>Sir Robin</fictional:character> +library/xml.etree.elementtree,,:character,<fictional:character>Gunther</fictional:character> +library/xml.etree.elementtree,,:character,<fictional:character>Commander Clement</fictional:character> library/xml.etree.elementtree,,:actor,"for actor in root.findall('real_person:actor', ns):" library/xml.etree.elementtree,,:name,"name = actor.find('real_person:name', ns)" library/xml.etree.elementtree,,:character,"for char in actor.findall('role:character', ns):" -library/zipapp,31,:main,"$ python -m zipapp myapp -m ""myapp:main""" -library/zipapp,82,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg.mod"" is a" -library/zipapp,155,:callable,"""pkg.module:callable"" and the archive will be run by importing" +library/zipapp,,:main,"$ python -m zipapp myapp -m ""myapp:main""" +library/zipapp,,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg.mod"" is a" +library/zipapp,,:callable,"""pkg.module:callable"" and the archive will be run by importing" library/stdtypes,,::,>>> m[::2].tolist() library/sys,,`,# ``wrapper`` creates a ``wrap(coro)`` coroutine: -tutorial/venv,77,:c7b9645a6f35,"Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25)" whatsnew/3.5,,:root,'WARNING:root:warning\n' whatsnew/3.5,,:warning,'WARNING:root:warning\n' whatsnew/3.5,,::,>>> addr6 = ipaddress.IPv6Address('::1') whatsnew/3.5,,:root,ERROR:root:exception whatsnew/3.5,,:exception,ERROR:root:exception +whatsnew/changelog,,:version,import sys; I = version[:version.index(' ')] +whatsnew/changelog,,:gz,": TarFile opened with external fileobj and ""w:gz"" mode didn't" +whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index 12989b2a53..d43461886e 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -78,6 +78,9 @@ slice notation makes this especially convenient:: >>> words ['defenestrate', 'cat', 'window', 'defenestrate'] +With ``for w in words:``, the example would attempt to create an infinite list, +inserting ``defenestrate`` over and over again. + .. _tut-range: diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index dd9c7cd731..beeaac36b9 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -25,7 +25,8 @@ first way is to do all the string handling yourself; using string slicing and concatenation operations you can create any layout you can imagine. The string type has some methods that perform useful operations for padding strings to a given column width; these will be discussed shortly. The second -way is to use the :meth:`str.format` method. +way is to use :ref:`formatted string literals <f-strings>`, or the +:meth:`str.format` method. The :mod:`string` module contains a :class:`~string.Template` class which offers yet another way to substitute values into strings. diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index e8d8e2b56a..faf57a344c 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -10,13 +10,13 @@ Using the Python Interpreter Invoking the Interpreter ======================== -The Python interpreter is usually installed as :file:`/usr/local/bin/python3.5` +The Python interpreter is usually installed as :file:`/usr/local/bin/python3.6` on those machines where it is available; putting :file:`/usr/local/bin` in your Unix shell's search path makes it possible to start it by typing the command: .. code-block:: text - python3.5 + python3.6 to the shell. [#]_ Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local @@ -24,11 +24,11 @@ Python guru or system administrator. (E.g., :file:`/usr/local/python` is a popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python35`, though you can change this when you're running the +:file:`C:\\Python36`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python35 + set path=%path%;C:\python36 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -98,8 +98,8 @@ before printing the first prompt: .. code-block:: shell-session - $ python3.5 - Python 3.5 (default, Sep 16 2015, 09:25:04) + $ python3.6 + Python 3.6 (default, Sep 16 2015, 09:25:04) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 214032917b..7e8ee3e5ea 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -352,6 +352,9 @@ The built-in function :func:`len` returns the length of a string:: Strings support a large number of methods for basic transformations and searching. + :ref:`f-strings` + String literals that have embedded expressions. + :ref:`formatstrings` Information about string formatting with :meth:`str.format`. diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index 52ffdbe063..1dd06c2338 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -15,7 +15,7 @@ operating system:: >>> import os >>> os.getcwd() # Return the current working directory - 'C:\\Python35' + 'C:\\Python36' >>> os.chdir('/server/accesslogs') # Change current working directory >>> os.system('mkdir today') # Run the command mkdir in the system shell 0 diff --git a/Doc/tutorial/stdlib2.rst b/Doc/tutorial/stdlib2.rst index 3714384eb6..bf4cf87dc3 100644 --- a/Doc/tutorial/stdlib2.rst +++ b/Doc/tutorial/stdlib2.rst @@ -278,7 +278,7 @@ applications include caching objects that are expensive to create:: Traceback (most recent call last): File "<stdin>", line 1, in <module> d['primary'] # entry was automatically removed - File "C:/python35/lib/weakref.py", line 46, in __getitem__ + File "C:/python36/lib/weakref.py", line 46, in __getitem__ o = self.data[key]() KeyError: 'primary' diff --git a/Doc/tutorial/venv.rst b/Doc/tutorial/venv.rst index 3b2ee2e24c..e2dd57d48f 100644 --- a/Doc/tutorial/venv.rst +++ b/Doc/tutorial/venv.rst @@ -20,15 +20,14 @@ the requirements of every application. If application A needs version the requirements are in conflict and installing either version 1.0 or 2.0 will leave one application unable to run. -The solution for this problem is to create a :term:`virtual -environment` (often shortened to "virtualenv"), a self-contained -directory tree that contains a Python installation for a particular -version of Python, plus a number of additional packages. +The solution for this problem is to create a :term:`virtual environment`, a +self-contained directory tree that contains a Python installation for a +particular version of Python, plus a number of additional packages. Different applications can then use different virtual environments. To resolve the earlier example of conflicting requirements, application A can have its own virtual environment with version 1.0 -installed while application B has another virtualenv with version 2.0. +installed while application B has another virtual environment with version 2.0. If application B requires a library be upgraded to version 3.0, this will not affect application A's environment. @@ -36,29 +35,26 @@ not affect application A's environment. Creating Virtual Environments ============================= -The script used to create and manage virtual environments is called -:program:`pyvenv`. :program:`pyvenv` will usually install the most -recent version of Python that you have available; the script is also -installed with a version number, so if you have multiple versions of -Python on your system you can select a specific Python version by -running ``pyvenv-3.4`` or whichever version you want. +The module used to create and manage virtual environments is called +:mod:`venv`. :mod:`venv` will usually install the most recent version of +Python that you have available. If you have multiple versions of Python on your +system, you can select a specific Python version by running ``python3`` or +whichever version you want. -To create a virtualenv, decide upon a directory -where you want to place it and run :program:`pyvenv` with the -directory path:: +To create a virtual environment, decide upon a directory where you want to +place it, and run the :mod:`venv` module as a script with the directory path:: - pyvenv tutorial-env + python3 -m venv tutorial-env This will create the ``tutorial-env`` directory if it doesn't exist, and also create directories inside it containing a copy of the Python interpreter, the standard library, and various supporting files. -Once you've created a virtual environment, you need to -activate it. +Once you've created a virtual environment, you may activate it. On Windows, run:: - tutorial-env/Scripts/activate + tutorial-env\Scripts\activate.bat On Unix or MacOS, run:: @@ -69,33 +65,36 @@ On Unix or MacOS, run:: ``activate.csh`` and ``activate.fish`` scripts you should use instead.) -Activating the virtualenv will change your shell's prompt to show what -virtualenv you're using, and modify the environment so that running -``python`` will get you that particular version and installation of -Python. For example:: +Activating the virtual environment will change your shell's prompt to show what +virtual environment you're using, and modify the environment so that running +``python`` will get you that particular version and installation of Python. +For example: - -> source ~/envs/tutorial-env/bin/activate - (tutorial-env) -> python - Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25) +.. code-block:: bash + + $ source ~/envs/tutorial-env/bin/activate + (tutorial-env) $ python + Python 3.5.1 (default, May 6 2016, 10:59:36) ... >>> import sys >>> sys.path - ['', '/usr/local/lib/python34.zip', ..., - '~/envs/tutorial-env/lib/python3.4/site-packages'] + ['', '/usr/local/lib/python35.zip', ..., + '~/envs/tutorial-env/lib/python3.5/site-packages'] >>> Managing Packages with pip ========================== -Once you've activated a virtual environment, you can install, upgrade, -and remove packages using a program called :program:`pip`. By default -``pip`` will install packages from the Python Package Index, -<https://pypi.python.org/pypi>. You can browse the Python Package Index -by going to it in your web browser, or you can use ``pip``'s -limited search feature:: +You can install, upgrade, and remove packages using a program called +:program:`pip`. By default ``pip`` will install packages from the Python +Package Index, <https://pypi.python.org/pypi>. You can browse the Python +Package Index by going to it in your web browser, or you can use ``pip``'s +limited search feature: + +.. code-block:: bash - (tutorial-env) -> pip search astronomy + (tutorial-env) $ pip search astronomy skyfield - Elegant astronomy for Python gary - Galactic astronomy and gravitational dynamics. novas - The United States Naval Observatory NOVAS astronomy library @@ -107,9 +106,11 @@ limited search feature:: "freeze", etc. (Consult the :ref:`installing-index` guide for complete documentation for ``pip``.) -You can install the latest version of a package by specifying a package's name:: +You can install the latest version of a package by specifying a package's name: + +.. code-block:: bash - -> pip install novas + (tutorial-env) $ pip install novas Collecting novas Downloading novas-3.1.1.3.tar.gz (136kB) Installing collected packages: novas @@ -117,9 +118,11 @@ You can install the latest version of a package by specifying a package's name:: Successfully installed novas-3.1.1.3 You can also install a specific version of a package by giving the -package name followed by ``==`` and the version number:: +package name followed by ``==`` and the version number: - -> pip install requests==2.6.0 +.. code-block:: bash + + (tutorial-env) $ pip install requests==2.6.0 Collecting requests==2.6.0 Using cached requests-2.6.0-py2.py3-none-any.whl Installing collected packages: requests @@ -128,9 +131,11 @@ package name followed by ``==`` and the version number:: If you re-run this command, ``pip`` will notice that the requested version is already installed and do nothing. You can supply a different version number to get that version, or you can run ``pip -install --upgrade`` to upgrade the package to the latest version:: +install --upgrade`` to upgrade the package to the latest version: + +.. code-block:: bash - -> pip install --upgrade requests + (tutorial-env) $ pip install --upgrade requests Collecting requests Installing collected packages: requests Found existing installation: requests 2.6.0 @@ -141,9 +146,11 @@ install --upgrade`` to upgrade the package to the latest version:: ``pip uninstall`` followed by one or more package names will remove the packages from the virtual environment. -``pip show`` will display information about a particular package:: +``pip show`` will display information about a particular package: - (tutorial-env) -> pip show requests +.. code-block:: bash + + (tutorial-env) $ pip show requests --- Metadata-Version: 2.0 Name: requests @@ -157,9 +164,11 @@ packages from the virtual environment. Requires: ``pip list`` will display all of the packages installed in the virtual -environment:: +environment: + +.. code-block:: bash - (tutorial-env) -> pip list + (tutorial-env) $ pip list novas (3.1.1.3) numpy (1.9.2) pip (7.0.3) @@ -168,19 +177,23 @@ environment:: ``pip freeze`` will produce a similar list of the installed packages, but the output uses the format that ``pip install`` expects. -A common convention is to put this list in a ``requirements.txt`` file:: +A common convention is to put this list in a ``requirements.txt`` file: - (tutorial-env) -> pip freeze > requirements.txt - (tutorial-env) -> cat requirements.txt +.. code-block:: bash + + (tutorial-env) $ pip freeze > requirements.txt + (tutorial-env) $ cat requirements.txt novas==3.1.1.3 numpy==1.9.2 requests==2.7.0 The ``requirements.txt`` can then be committed to version control and shipped as part of an application. Users can then install all the -necessary packages with ``install -r``:: +necessary packages with ``install -r``: + +.. code-block:: bash - -> pip install -r requirements.txt + (tutorial-env) $ pip install -r requirements.txt Collecting novas==3.1.1.3 (from -r requirements.txt (line 1)) ... Collecting numpy==1.9.2 (from -r requirements.txt (line 2)) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 57faf89e1a..75cb8ea24b 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -395,6 +395,8 @@ Miscellaneous options stored in a traceback of a trace. Use ``-X tracemalloc=NFRAME`` to start tracing with a traceback limit of *NFRAME* frames. See the :func:`tracemalloc.start` for more information. + * ``-X showalloccount`` to enable the output of the total count of allocated + objects for each type (only works when built with ``COUNT_ALLOCS`` defined); It also allows passing arbitrary values and retrieving them through the :data:`sys._xoptions` dictionary. @@ -408,6 +410,9 @@ Miscellaneous options .. versionadded:: 3.4 The ``-X showrefcount`` and ``-X tracemalloc`` options. + .. versionadded:: 3.6 + The ``-X showalloccount`` option. + Options you shouldn't use ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -554,6 +559,10 @@ conflict. .. versionchanged:: 3.4 The ``encodingname`` part is now optional. + .. versionchanged:: 3.6 + On Windows, the encoding specified by this variable is ignored for interactive + console buffers unless :envvar:`PYTHONLEGACYWINDOWSIOENCODING` is also specified. + Files and pipes redirected through the standard streams are not affected. .. envvar:: PYTHONNOUSERSITE @@ -619,6 +628,81 @@ conflict. .. versionadded:: 3.4 +.. envvar:: PYTHONMALLOC + + Set the Python memory allocators and/or install debug hooks. + + Set the family of memory allocators used by Python: + + * ``malloc``: use the :c:func:`malloc` function of the C library + for all domains (:c:data:`PYMEM_DOMAIN_RAW`, :c:data:`PYMEM_DOMAIN_MEM`, + :c:data:`PYMEM_DOMAIN_OBJ`). + * ``pymalloc``: use the :ref:`pymalloc allocator <pymalloc>` for + :c:data:`PYMEM_DOMAIN_MEM` and :c:data:`PYMEM_DOMAIN_OBJ` domains and use + the :c:func:`malloc` function for the :c:data:`PYMEM_DOMAIN_RAW` domain. + + Install debug hooks: + + * ``debug``: install debug hooks on top of the default memory allocator + * ``malloc_debug``: same as ``malloc`` but also install debug hooks + * ``pymalloc_debug``: same as ``pymalloc`` but also install debug hooks + + When Python is compiled in release mode, the default is ``pymalloc``. When + compiled in debug mode, the default is ``pymalloc_debug`` and the debug hooks + are used automatically. + + If Python is configured without ``pymalloc`` support, ``pymalloc`` and + ``pymalloc_debug`` are not available, the default is ``malloc`` in release + mode and ``malloc_debug`` in debug mode. + + See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python + memory allocators. + + .. versionadded:: 3.6 + + +.. envvar:: PYTHONMALLOCSTATS + + If set to a non-empty string, Python will print statistics of the + :ref:`pymalloc memory allocator <pymalloc>` every time a new pymalloc object + arena is created, and on shutdown. + + This variable is ignored if the :envvar:`PYTHONMALLOC` environment variable + is used to force the :c:func:`malloc` allocator of the C library, or if + Python is configured without ``pymalloc`` support. + + .. versionchanged:: 3.6 + This variable can now also be used on Python compiled in release mode. + It now has no effect if set to an empty string. + + +.. envvar:: PYTHONLEGACYWINDOWSFSENCODING + + If set to a non-empty string, the default filesystem encoding and errors mode + will revert to their pre-3.6 values of 'mbcs' and 'replace', respectively. + Otherwise, the new defaults 'utf-8' and 'surrogatepass' are used. + + This may also be enabled at runtime with + :func:`sys._enablelegacywindowsfsencoding()`. + + Availability: Windows + + .. versionadded:: 3.6 + See :pep:`529` for more details. + +.. envvar:: PYTHONLEGACYWINDOWSIOENCODING + + If set to a non-empty string, does not use the new console reader and + writer. This means that Unicode characters will be encoded according to + the active console code page, rather than using utf-8. + + This variable is ignored if the standard streams are redirected (to files + or pipes) rather than referring to console buffers. + + Availability: Windows + + .. versionadded:: 3.6 + Debug-mode variables ~~~~~~~~~~~~~~~~~~~~ @@ -634,9 +718,3 @@ if Python was configured with the ``--with-pydebug`` build option. If set, Python will dump objects and reference counts still alive after shutting down the interpreter. - - -.. envvar:: PYTHONMALLOCSTATS - - If set, Python will print memory allocation statistics every time a new - object arena is created, and on shutdown. diff --git a/Doc/using/index.rst b/Doc/using/index.rst index 502afa9033..a5df713944 100644 --- a/Doc/using/index.rst +++ b/Doc/using/index.rst @@ -17,4 +17,3 @@ interpreter and things that make working with Python easier. unix.rst windows.rst mac.rst - scripts.rst diff --git a/Doc/using/mac.rst b/Doc/using/mac.rst index 05c91bba59..8f1ac3f3fd 100644 --- a/Doc/using/mac.rst +++ b/Doc/using/mac.rst @@ -25,7 +25,7 @@ there. What you get after installing is a number of things: -* A :file:`MacPython 3.5` folder in your :file:`Applications` folder. In here +* A :file:`MacPython 3.6` folder in your :file:`Applications` folder. In here you find IDLE, the development environment that is a standard part of official Python distributions; PythonLauncher, which handles double-clicking Python scripts from the Finder; and the "Build Applet" tool, which allows you to @@ -93,7 +93,7 @@ aware of: programs that talk to the Aqua window manager (in other words, anything that has a GUI) need to be run in a special way. Use :program:`pythonw` instead of :program:`python` to start such scripts. -With Python 3.5, you can use either :program:`python` or :program:`pythonw`. +With Python 3.6, you can use either :program:`python` or :program:`pythonw`. Configuration @@ -159,7 +159,7 @@ https://riverbankcomputing.com/software/pyqt/intro. Distributing Python Applications on the Mac =========================================== -The "Build Applet" tool that is placed in the MacPython 3.5 folder is fine for +The "Build Applet" tool that is placed in the MacPython 3.6 folder is fine for packaging small Python scripts on your own machine to run as a standard Mac application. This tool, however, is not robust enough to distribute Python applications to other users. diff --git a/Doc/using/scripts.rst b/Doc/using/scripts.rst deleted file mode 100644 index 2c87416f1c..0000000000 --- a/Doc/using/scripts.rst +++ /dev/null @@ -1,12 +0,0 @@ -.. _tools-and-scripts: - -Additional Tools and Scripts -============================ - -.. _scripts-pyvenv: - -pyvenv - Creating virtual environments --------------------------------------- - -.. include:: venv-create.inc - diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index 7ad3008826..53f431b5cf 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -1,31 +1,39 @@ Creation of :ref:`virtual environments <venv-def>` is done by executing the -``pyvenv`` script:: +command ``venv``:: - pyvenv /path/to/new/virtual/environment + python3 -m venv /path/to/new/virtual/environment Running this command creates the target directory (creating any parent directories that don't exist already) and places a ``pyvenv.cfg`` file in it -with a ``home`` key pointing to the Python installation the command was run -from. It also creates a ``bin`` (or ``Scripts`` on Windows) subdirectory +with a ``home`` key pointing to the Python installation from which the command +was run. It also creates a ``bin`` (or ``Scripts`` on Windows) subdirectory containing a copy of the ``python`` binary (or binaries, in the case of Windows). It also creates an (initially empty) ``lib/pythonX.Y/site-packages`` subdirectory (on Windows, this is ``Lib\site-packages``). +.. deprecated:: 3.6 + ``pyvenv`` was the recommended tool for creating virtual environments for + Python 3.3 and 3.4, and is `deprecated in Python 3.6 + <https://docs.python.org/dev/whatsnew/3.6.html#deprecated-features>`_. + +.. versionchanged:: 3.5 + The use of ``venv`` is now recommended for creating virtual environments. + .. seealso:: `Python Packaging User Guide: Creating and using virtual environments - <https://packaging.python.org/en/latest/installing/#creating-virtual-environments>`__ + <https://packaging.python.org/installing/#creating-virtual-environments>`__ .. highlight:: none -On Windows, you may have to invoke the ``pyvenv`` script as follows, if you -don't have the relevant PATH and PATHEXT settings:: +On Windows, invoke the ``venv`` command as follows:: - c:\Temp>c:\Python35\python c:\Python35\Tools\Scripts\pyvenv.py myenv + c:\>c:\Python35\python -m venv c:\path\to\myenv -or equivalently:: +Alternatively, if you configured the ``PATH`` and ``PATHEXT`` variables for +your :ref:`Python installation <using-on-windows>`:: - c:\Temp>c:\Python35\python -m venv myenv + c:\>python -m venv myenv c:\path\to\myenv The command, if run with ``-h``, will show the available options:: @@ -36,25 +44,26 @@ The command, if run with ``-h``, will show the available options:: Creates virtual Python environments in one or more target directories. positional arguments: - ENV_DIR A directory to create the environment in. + ENV_DIR A directory to create the environment in. optional arguments: - -h, --help show this help message and exit - --system-site-packages Give the virtual environment access to the system - site-packages dir. - --symlinks Try to use symlinks rather than copies, when symlinks - are not the default for the platform. - --copies Try to use copies rather than symlinks, even when - symlinks are the default for the platform. - --clear Delete the contents of the environment directory if it - already exists, before environment creation. - --upgrade Upgrade the environment directory to use this version - of Python, assuming Python has been upgraded in-place. - --without-pip Skips installing or upgrading pip in the virtual - environment (pip is bootstrapped by default) - -Depending on how the ``venv`` functionality has been invoked, the usage message -may vary slightly, e.g. referencing ``pyvenv`` rather than ``venv``. + -h, --help show this help message and exit + --system-site-packages + Give the virtual environment access to the system + site-packages dir. + --symlinks Try to use symlinks rather than copies, when symlinks + are not the default for the platform. + --copies Try to use copies rather than symlinks, even when + symlinks are the default for the platform. + --clear Delete the contents of the environment directory if it + already exists, before environment creation. + --upgrade Upgrade the environment directory to use this version + of Python, assuming Python has been upgraded in-place. + --without-pip Skips installing or upgrading pip in the virtual + environment (pip is bootstrapped by default) + + Once an environment has been created, you may wish to activate it, e.g. by + sourcing an activate script in its bin directory. .. versionchanged:: 3.4 Installs pip by default, added the ``--without-pip`` and ``--copies`` @@ -73,12 +82,13 @@ run with the ``--system-site-packages`` option, ``false`` otherwise. Unless the ``--without-pip`` option is given, :mod:`ensurepip` will be invoked to bootstrap ``pip`` into the virtual environment. -Multiple paths can be given to ``pyvenv``, in which case an identical -virtualenv will be created, according to the given options, at each -provided path. +Multiple paths can be given to ``venv``, in which case an identical virtual +environment will be created, according to the given options, at each provided +path. -Once a venv has been created, it can be "activated" using a script in the -venv's binary directory. The invocation of the script is platform-specific: +Once a virtual environment has been created, it can be "activated" using a +script in the virtual environment's binary directory. The invocation of the +script is platform-specific: +-------------+-----------------+-----------------------------------------+ | Platform | Shell | Command to activate virtual environment | @@ -95,16 +105,17 @@ venv's binary directory. The invocation of the script is platform-specific: +-------------+-----------------+-----------------------------------------+ You don't specifically *need* to activate an environment; activation just -prepends the venv's binary directory to your path, so that "python" invokes the -venv's Python interpreter and you can run installed scripts without having to -use their full path. However, all scripts installed in a venv should be -runnable without activating it, and run with the venv's Python automatically. - -You can deactivate a venv by typing "deactivate" in your shell. The exact -mechanism is platform-specific: for example, the Bash activation script defines -a "deactivate" function, whereas on Windows there are separate scripts called -``deactivate.bat`` and ``Deactivate.ps1`` which are installed when the venv is -created. +prepends the virtual environment's binary directory to your path, so that +"python" invokes the virtual environment's Python interpreter and you can run +installed scripts without having to use their full path. However, all scripts +installed in a virtual environment should be runnable without activating it, +and run with the virtual environment's Python automatically. + +You can deactivate a virtual environment by typing "deactivate" in your shell. +The exact mechanism is platform-specific: for example, the Bash activation +script defines a "deactivate" function, whereas on Windows there are separate +scripts called ``deactivate.bat`` and ``Deactivate.ps1`` which are installed +when the virtual environment is created. .. versionadded:: 3.4 ``fish`` and ``csh`` activation scripts. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index a4a6a30c36..81efbb0c98 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -29,15 +29,15 @@ Supported Versions As specified in :pep:`11`, a Python release only supports a Windows platform while Microsoft considers the platform under extended support. This means that -Python 3.5 supports Windows Vista and newer. If you require Windows XP support -then please install Python 3.4. +Python |version| supports Windows Vista and newer. If you require Windows XP +support then please install Python 3.4. Installation Steps ------------------ -Four Python 3.5 installers are available for download - two each for the 32-bit -and 64-bit versions of the interpreter. The *web installer* is a small initial -download, and it will automatically download the required components as +Four Python |version| installers are available for download - two each for the +32-bit and 64-bit versions of the interpreter. The *web installer* is a small +initial download, and it will automatically download the required components as necessary. The *offline installer* includes the components necessary for a default installation and only requires an internet connection for optional features. See :ref:`install-layout-option` for other ways to avoid downloading @@ -74,6 +74,31 @@ installation". In this case: * If selected, the install directory will be added to the system :envvar:`PATH` * Shortcuts are available for all users +.. _max-path: + +Removing the MAX_PATH Limitation +-------------------------------- + +Windows historically has limited path lengths to 260 characters. This meant that +paths longer than this would not resolve and errors would result. + +In the latest versions of Windows, this limitation can be expanded to +approximately 32,000 characters. Your administrator will need to activate the +"Enable Win32 long paths" group policy, or set the registry value +``HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem@LongPathsEnabled`` +to ``1``. + +This allows the :func:`open` function, the :mod:`os` module and most other +path functionality to accept and return paths longer than 260 characters when +using strings. (Use of bytes as paths is deprecated on Windows, and this feature +is not available when using bytes.) + +After changing the above option, no further configuration is required. + +.. versionchanged:: 3.6 + + Support for long paths was enabled in Python. + .. _install-quiet-option: Installing Without UI @@ -168,13 +193,13 @@ of available options is shown below. For example, to silently install a default, system-wide Python installation, you could use the following command (from an elevated command prompt):: - python-3.5.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0 + python-3.6.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0 To allow users to easily install a personal copy of Python without the test suite, you could provide a shortcut with the following command. This will display a simplified initial page and disallow customization:: - python-3.5.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0 + python-3.6.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0 SimpleInstall=1 SimpleInstallDescription="Just for me, no test suite." (Note that omitting the launcher also omits file associations, and is only @@ -209,13 +234,13 @@ where a large number of installations are going to be performed it is very useful to have a locally cached copy. Execute the following command from Command Prompt to download all possible -required files. Remember to substitute ``python-3.5.0.exe`` for the actual +required files. Remember to substitute ``python-3.6.0.exe`` for the actual name of your installer, and to create layouts in their own directories to avoid collisions between files with the same name. :: - python-3.5.0.exe /layout [optional target directory] + python-3.6.0.exe /layout [optional target directory] You may also specify the ``/quiet`` option to hide the progress display. @@ -320,7 +345,7 @@ User level and the System level, or temporarily in a command prompt. To temporarily set environment variables, open Command Prompt and use the :command:`set` command:: - C:\>set PATH=C:\Program Files\Python 3.5;%PATH% + C:\>set PATH=C:\Program Files\Python 3.6;%PATH% C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib C:\>python @@ -376,10 +401,10 @@ Finding the Python executable Besides using the automatically created start menu entry for the Python interpreter, you might want to start Python in the command prompt. The -installer for Python 3.5 and later has an option to set that up for you. +installer has an option to set that up for you. -On the first page of the installer, an option labelled "Add Python 3.5 to -PATH" can be selected to have the installer add the install location into the +On the first page of the installer, an option labelled "Add Python to PATH" +may be selected to have the installer add the install location into the :envvar:`PATH`. The location of the :file:`Scripts\\` folder is also added. This allows you to type :command:`python` to run the interpreter, and :command:`pip` for the package installer. Thus, you can also execute your @@ -393,7 +418,7 @@ of your Python installation, delimited by a semicolon from other entries. An example variable could look like this (assuming the first two entries already existed):: - C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.5 + C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.6 .. _launcher: @@ -418,6 +443,8 @@ Getting started From the command-line ^^^^^^^^^^^^^^^^^^^^^ +.. versionchanged:: 3.6 + System-wide installations of Python 3.3 and later will put the launcher on your :envvar:`PATH`. The launcher is compatible with all available versions of Python, so it does not matter which version is installed. To check that the @@ -427,25 +454,26 @@ launcher is available, execute the following command in Command Prompt: py -You should find that the latest version of Python 2.x you have installed is +You should find that the latest version of Python you have installed is started - it can be exited as normal, and any additional command-line arguments specified will be sent directly to Python. -If you have multiple versions of Python 2.x installed (e.g., 2.6 and 2.7) you -will have noticed that Python 2.7 was started - to launch Python 2.6, try the -command: +If you have multiple versions of Python installed (e.g., 2.7 and |version|) you +will have noticed that Python |version| was started - to launch Python 2.7, try +the command: :: - py -2.6 + py -2.7 -If you have a Python 3.x installed, try the command: +If you want the latest version of Python 2.x you have installed, try the +command: :: - py -3 + py -2 -You should find the latest version of Python 3.x starts. +You should find the latest version of Python 2.x starts. If you see the following error, you do not have the launcher installed: @@ -500,6 +528,11 @@ version qualifier. Assuming you have Python 2.6 installed, try changing the first line to ``#! python2.6`` and you should find the 2.6 version information printed. +Note that unlike interactive use, a bare "python" will use the latest +version of Python 2.x that you have installed. This is for backward +compatibility and for compatibility with Unix, where the command ``python`` +typically refers to Python 2. + From file associations ^^^^^^^^^^^^^^^^^^^^^^ @@ -676,7 +709,7 @@ target Python. -.. finding_modules: +.. _finding_modules: Finding modules =============== @@ -687,7 +720,24 @@ installation directory. So, if you had installed Python to :file:`C:\\Python\\Lib\\` and third-party modules should be stored in :file:`C:\\Python\\Lib\\site-packages\\`. -This is how :data:`sys.path` is populated on Windows: +To completely override :data:`sys.path`, create a ``._pth`` file with the same +name as the DLL (``python36._pth``) or the executable (``python._pth``) and +specify one line for each path to add to :data:`sys.path`. The file based on the +DLL name overrides the one based on the executable, which allows paths to be +restricted for any program loading the runtime if desired. + +When the file exists, all registry and environment variables are ignored, +isolated mode is enabled, and :mod:`site` is not imported unless one line in the +file specifies ``import site``. Blank paths and lines starting with ``#`` are +ignored. Each path may be absolute or relative to the location of the file. +Import statements other than to ``site`` are not permitted, and arbitrary code +cannot be specified. + +Note that ``.pth`` files (without leading underscore) will be processed normally +by the :mod:`site` module. + +When no ``._pth`` file is found, this is how :data:`sys.path` is populated on +Windows: * An empty entry is added at the start, which corresponds to the current directory. @@ -706,10 +756,11 @@ This is how :data:`sys.path` is populated on Windows: * If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as "Python Home". Otherwise, the path of the main Python executable is used to - locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a - Python home is found, the relevant sub-directories added to :data:`sys.path` - (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core - Python path is constructed from the PythonPath stored in the registry. + locate a "landmark file" (either ``Lib\os.py`` or ``pythonXY.zip``) to deduce + the "Python Home". If a Python home is found, the relevant sub-directories + added to :data:`sys.path` (``Lib``, ``plat-win``, etc) are based on that + folder. Otherwise, the core Python path is constructed from the PythonPath + stored in the registry. * If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in the environment, and no registry entries can be found, a default path with @@ -722,10 +773,6 @@ directory one level above the executable, the following variations apply: path is used instead of the path to the main executable when deducing the home location. -* If ``applocal`` is set to true, the ``home`` property or the main executable - is always used as the home path, and all environment variables or registry - values affecting the path are ignored. The landmark file is not checked. - The end result of all this is: * When running :file:`python.exe`, or any other .exe in the main Python @@ -744,13 +791,12 @@ The end result of all this is: For those who want to bundle Python into their application or distribution, the following advice will prevent conflicts with other installations: -* Include a ``pyvenv.cfg`` file alongside your executable containing - ``applocal = true``. This will ensure that your own directory will be used to - resolve paths even if you have included the standard library in a ZIP file. - It will also ignore user site-packages and other paths listed in the - registry. +* Include a ``._pth`` file alongside your executable containing the + directories to include. This will ignore paths listed in the registry and + environment variables, and also ignore :mod:`site` unless ``import site`` is + listed. -* If you are loading :file:`python3.dll` or :file:`python35.dll` in your own +* If you are loading :file:`python3.dll` or :file:`python36.dll` in your own executable, explicitly call :c:func:`Py_SetPath` or (at least) :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`. @@ -760,7 +806,8 @@ following advice will prevent conflicts with other installations: * If you cannot use the previous suggestions (for example, you are a distribution that allows people to run :file:`python.exe` directly), ensure that the landmark file (:file:`Lib\\os.py`) exists in your install directory. - (Note that it will not be detected inside a ZIP file.) + (Note that it will not be detected inside a ZIP file, but a correctly named + ZIP file will be detected instead.) These will ensure that the files in a system-wide installation will not take precedence over the copy of the standard library bundled with your application. @@ -768,6 +815,14 @@ Otherwise, your users may experience problems using your application. Note that the first suggestion is the best, as the other may still be susceptible to non-standard paths in the registry and user site-packages. +.. versionchanged:: + 3.6 + + * Adds ``._pth`` file support and removes ``applocal`` option from + ``pyvenv.cfg``. + * Adds ``pythonXX.zip`` as a potential landmark when directly adjacent + to the executable. + Additional modules ================== @@ -867,7 +922,7 @@ directly accessed by end-users. When extracted, the embedded distribution is (almost) fully isolated from the user's system, including environment variables, system registry settings, and installed packages. The standard library is included as pre-compiled and -optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python35.dll``, +optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python36.dll``, ``python.exe`` and ``pythonw.exe`` are all provided. Tcl/tk (including all dependants, such as Idle), pip and the Python documentation are not included. diff --git a/Doc/whatsnew/2.1.rst b/Doc/whatsnew/2.1.rst index 06366b8ba0..6aae726c31 100644 --- a/Doc/whatsnew/2.1.rst +++ b/Doc/whatsnew/2.1.rst @@ -367,7 +367,7 @@ dictionary:: This version works for simple things such as integers, but it has a side effect; the ``_cache`` dictionary holds a reference to the return values, so they'll -never be deallocated until the Python process exits and cleans up This isn't +never be deallocated until the Python process exits and cleans up. This isn't very noticeable for integers, but if :func:`f` returns an object, or a data structure that takes up a lot of memory, this can be a problem. diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst index 9acc7347dd..b900e7733f 100644 --- a/Doc/whatsnew/3.2.rst +++ b/Doc/whatsnew/3.2.rst @@ -1253,9 +1253,9 @@ definition:: @contextmanager def track_entry_and_exit(name): - logging.info('Entering: {}'.format(name)) + logging.info('Entering: %s', name) yield - logging.info('Exiting: {}'.format(name)) + logging.info('Exiting: %s', name) Formerly, this would have only been usable as a context manager:: diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst index 0c9f2bb24f..93b297cf65 100644 --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -108,7 +108,7 @@ packages. Their concept and implementation are inspired by the popular with the interpreter core. This PEP adds the :mod:`venv` module for programmatic access, and the -:ref:`pyvenv <scripts-pyvenv>` script for command-line access and +``pyvenv`` script for command-line access and administration. The Python interpreter checks for a ``pyvenv.cfg``, file whose existence signals the base of a virtual environment's directory tree. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst index 1e5c9d1fbd..72398f9250 100644 --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -197,7 +197,7 @@ will also be installed. On other platforms, the system wide unversioned ``pip`` command typically refers to the separately installed Python 2 version. -The :ref:`pyvenv <scripts-pyvenv>` command line utility and the :mod:`venv` +The ``pyvenv`` command line utility and the :mod:`venv` module make use of the :mod:`ensurepip` module to make ``pip`` readily available in virtual environments. When using the command line utility, ``pip`` is installed by default, while when using the :mod:`venv` module @@ -824,7 +824,7 @@ currently in the new module, which is being added as part of email's new :term:`provisional API`. These classes provide a number of new methods that make extracting content from and inserting content into email messages much easier. For details, see the :mod:`~email.contentmanager` documentation and -the :ref:`email-contentmanager-api-examples`. These API additions complete the +the :ref:`email-examples`. These API additions complete the bulk of the work that was planned as part of the email6 project. The currently provisional API is scheduled to become final in Python 3.5 (possibly with a few minor additions in the area of error handling). (Contributed by R. David @@ -1989,11 +1989,11 @@ Other Improvements Stinner using his :pep:`445`-based ``pyfailmalloc`` tool (:issue:`18408`, :issue:`18520`). -* The :ref:`pyvenv <scripts-pyvenv>` command now accepts a ``--copies`` option +* The ``pyvenv`` command now accepts a ``--copies`` option to use copies rather than symlinks even on systems where symlinks are the default. (Contributed by Vinay Sajip in :issue:`18807`.) -* The :ref:`pyvenv <scripts-pyvenv>` command also accepts a ``--without-pip`` +* The ``pyvenv`` command also accepts a ``--without-pip`` option to suppress the otherwise-automatic bootstrapping of pip into the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` as part of the :pep:`453` implementation.) @@ -2459,7 +2459,7 @@ Changes in the Python API stream in :mod:`~io.TextIOWrapper` to use its *newline* argument (:issue:`15204`). -* If you use :ref:`pyvenv <scripts-pyvenv>` in a script and desire that pip +* If you use ``pyvenv`` in a script and desire that pip *not* be installed, you must add ``--without-pip`` to your command invocation. diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst index 2ef4e915b6..246d9c57cb 100644 --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -46,7 +46,8 @@ This article explains the new features in Python 3.5, compared to 3.4. Python 3.5 was released on September 13, 2015. See the -:ref:`changelog <changelog>` for a full list of changes. +`changelog <https://docs.python.org/3.5/whatsnew/changelog.html>`_ for a full +list of changes. .. seealso:: @@ -2175,8 +2176,7 @@ New ``calloc`` functions were added: * :c:func:`PyMem_RawCalloc`, * :c:func:`PyMem_Calloc`, -* :c:func:`PyObject_Calloc`, -* :c:func:`_PyObject_GC_Calloc`. +* :c:func:`PyObject_Calloc`. (Contributed by Victor Stinner in :issue:`21233`.) diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst new file mode 100644 index 0000000000..b7c857445c --- /dev/null +++ b/Doc/whatsnew/3.6.rst @@ -0,0 +1,1476 @@ +**************************** + What's New In Python 3.6 +**************************** + +:Release: |release| +:Date: |today| + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.6, compared to 3.5. + +For full details, see the :ref:`changelog <changelog>`. + +.. note:: + + Prerelease users should be aware that this document is currently in draft + form. It will be updated substantially as Python 3.6 moves towards release, + so it's worth checking back even after reading earlier versions. + + +Summary -- Release highlights +============================= + +.. This section singles out the most important changes in Python 3.6. + Brevity is key. + +New syntax features: + +* A ``global`` or ``nonlocal`` statement must now textually appear + before the first use of the affected name in the same scope. + Previously this was a SyntaxWarning. + +* PEP 498: :ref:`Formatted string literals <whatsnew-fstrings>` + +* PEP 515: Underscores in Numeric Literals + +* PEP 526: :ref:`Syntax for Variable Annotations <variable-annotations>` + +* PEP 525: Asynchronous Generators + +* PEP 530: Asynchronous Comprehensions + +Standard library improvements: + +Security improvements: + +* On Linux, :func:`os.urandom` now blocks until the system urandom entropy pool + is initialized to increase the security. See the :pep:`524` for the + rationale. + +* :mod:`hashlib` and :mod:`ssl` now support OpenSSL 1.1.0. + +* The default settings and feature set of the :mod:`ssl` have been improved. + +* The :mod:`hashlib` module has got support for BLAKE2, SHA-3 and SHAKE hash + algorithms and :func:`~hashlib.scrypt` key derivation function. + +Windows improvements: + +* PEP 529: :ref:`Change Windows filesystem encoding to UTF-8 <pep-529>` + +* PEP 528: :ref:`Change Windows console encoding to UTF-8 <pep-528>` + +* The ``py.exe`` launcher, when used interactively, no longer prefers + Python 2 over Python 3 when the user doesn't specify a version (via + command line arguments or a config file). Handling of shebang lines + remains unchanged - "python" refers to Python 2 in that case. + +* ``python.exe`` and ``pythonw.exe`` have been marked as long-path aware, + which means that when the 260 character path limit may no longer apply. + See :ref:`removing the MAX_PATH limitation <max-path>` for details. + +* A ``._pth`` file can be added to force isolated mode and fully specify + all search paths to avoid registry and environment lookup. See + :ref:`the documentation <finding_modules>` for more information. + +* A ``python36.zip`` file now works as a landmark to infer + :envvar:`PYTHONHOME`. See :ref:`the documentation <finding_modules>` for + more information. + +.. PEP-sized items next. + +.. _pep-4XX: + +.. PEP 4XX: Virtual Environments +.. ============================= + + +.. (Implemented by Foo Bar.) + +.. .. seealso:: + + :pep:`4XX` - Python Virtual Environments + PEP written by Carl Meyer + +New built-in features: + +* PEP 520: :ref:`Preserving Class Attribute Definition Order<whatsnew-deforder>` + +* PEP 468: :ref:`Preserving Keyword Argument Order<whatsnew-kwargs>` + +A complete list of PEP's implemented in Python 3.6: + +* :pep:`468`, :ref:`Preserving Keyword Argument Order<whatsnew-kwargs>` +* :pep:`487`, :ref:`Simpler customization of class creation<whatsnew-pep487>` +* :pep:`495`, Local Time Disambiguation +* :pep:`498`, :ref:`Formatted string literals <whatsnew-fstrings>` +* :pep:`506`, Adding A Secrets Module To The Standard Library +* :pep:`509`, :ref:`Add a private version to dict<whatsnew-pep509>` +* :pep:`515`, :ref:`Underscores in Numeric Literals<pep-515>` +* :pep:`519`, :ref:`Adding a file system path protocol<pep-519>` +* :pep:`520`, :ref:`Preserving Class Attribute Definition Order<whatsnew-deforder>` +* :pep:`523`, :ref:`Adding a frame evaluation API to CPython<pep-523>` +* :pep:`524`, Make os.urandom() blocking on Linux (during system startup) +* :pep:`525`, Asynchronous Generators (provisional) +* :pep:`526`, :ref:`Syntax for Variable Annotations (provisional)<variable-annotations>` +* :pep:`528`, :ref:`Change Windows console encoding to UTF-8 (provisional)<pep-528>` +* :pep:`529`, :ref:`Change Windows filesystem encoding to UTF-8 (provisional)<pep-529>` +* :pep:`530`, Asynchronous Comprehensions + + +New Features +============ + +.. _pep-515: + +PEP 515: Underscores in Numeric Literals +---------------------------------------- + +Prior to PEP 515, there was no support for writing long numeric +literals with some form of separator to improve readability. For +instance, how big is ``1000000000000000``? With :pep:`515`, though, +you can use underscores to separate digits as desired to make numeric +literals easier to read: ``1_000_000_000_000_000``. Underscores can be +used with other numeric literals beyond integers, e.g. +``0x_FF_FF_FF_FF``. + +Single underscores are allowed between digits and after any base +specifier. More than a single underscore in a row, leading, or +trailing underscores are not allowed. + +.. seealso:: + + :pep:`515` -- Underscores in Numeric Literals + PEP written by Georg Brandl and Serhiy Storchaka. + + +.. _pep-523: + +PEP 523: Adding a frame evaluation API to CPython +------------------------------------------------- + +While Python provides extensive support to customize how code +executes, one place it has not done so is in the evaluation of frame +objects. If you wanted some way to intercept frame evaluation in +Python there really wasn't any way without directly manipulating +function pointers for defined functions. + +:pep:`523` changes this by providing an API to make frame +evaluation pluggable at the C level. This will allow for tools such +as debuggers and JITs to intercept frame evaluation before the +execution of Python code begins. This enables the use of alternative +evaluation implementations for Python code, tracking frame +evaluation, etc. + +This API is not part of the limited C API and is marked as private to +signal that usage of this API is expected to be limited and only +applicable to very select, low-level use-cases. Semantics of the +API will change with Python as necessary. + +.. seealso:: + + :pep:`523` -- Adding a frame evaluation API to CPython + PEP written by Brett Cannon and Dino Viehland. + + +.. _pep-519: + +PEP 519: Adding a file system path protocol +------------------------------------------- + +File system paths have historically been represented as :class:`str` +or :class:`bytes` objects. This has led to people who write code which +operate on file system paths to assume that such objects are only one +of those two types (an :class:`int` representing a file descriptor +does not count as that is not a file path). Unfortunately that +assumption prevents alternative object representations of file system +paths like :mod:`pathlib` from working with pre-existing code, +including Python's standard library. + +To fix this situation, a new interface represented by +:class:`os.PathLike` has been defined. By implementing the +:meth:`~os.PathLike.__fspath__` method, an object signals that it +represents a path. An object can then provide a low-level +representation of a file system path as a :class:`str` or +:class:`bytes` object. This means an object is considered +:term:`path-like <path-like object>` if it implements +:class:`os.PathLike` or is a :class:`str` or :class:`bytes` object +which represents a file system path. Code can use :func:`os.fspath`, +:func:`os.fsdecode`, or :func:`os.fsencode` to explicitly get a +:class:`str` and/or :class:`bytes` representation of a path-like +object. + +The built-in :func:`open` function has been updated to accept +:class:`os.PathLike` objects as have all relevant functions in the +:mod:`os` and :mod:`os.path` modules. :c:func:`PyUnicode_FSConverter` +and :c:func:`PyUnicode_FSConverter` have been changed to accept +path-like objects. The :class:`os.DirEntry` class +and relevant classes in :mod:`pathlib` have also been updated to +implement :class:`os.PathLike`. + +The hope in is that updating the fundamental functions for operating +on file system paths will lead to third-party code to implicitly +support all :term:`path-like objects <path-like object>` without any +code changes or at least very minimal ones (e.g. calling +:func:`os.fspath` at the beginning of code before operating on a +path-like object). + +Here are some examples of how the new interface allows for +:class:`pathlib.Path` to be used more easily and transparently with +pre-existing code:: + + >>> import pathlib + >>> with open(pathlib.Path("README")) as f: + ... contents = f.read() + ... + >>> import os.path + >>> os.path.splitext(pathlib.Path("some_file.txt")) + ('some_file', '.txt') + >>> os.path.join("/a/b", pathlib.Path("c")) + '/a/b/c' + >>> import os + >>> os.fspath(pathlib.Path("some_file.txt")) + 'some_file.txt' + +(Implemented by Brett Cannon, Ethan Furman, Dusty Phillips, and Jelle Zijlstra.) + +.. seealso:: + + :pep:`519` -- Adding a file system path protocol + PEP written by Brett Cannon and Koos Zevenhoven. + + +.. _whatsnew-fstrings: + +PEP 498: Formatted string literals +---------------------------------- + +Formatted string literals are a new kind of string literal, prefixed +with ``'f'``. They are similar to the format strings accepted by +:meth:`str.format`. They contain replacement fields surrounded by +curly braces. The replacement fields are expressions, which are +evaluated at run time, and then formatted using the :func:`format` protocol:: + + >>> name = "Fred" + >>> f"He said his name is {name}." + 'He said his name is Fred.' + +See :pep:`498` and the main documentation at :ref:`f-strings`. + + +.. _variable-annotations: + +PEP 526: Syntax for variable annotations +---------------------------------------- + +:pep:`484` introduced standard for type annotations of function parameters, +a.k.a. type hints. This PEP adds syntax to Python for annotating the +types of variables including class variables and instance variables:: + + primes: List[int] = [] + + captain: str # Note: no initial value! + + class Starship: + stats: Dict[str, int] = {} + +Just as for function annotations, the Python interpreter does not attach any +particular meaning to variable annotations and only stores them in a special +attribute ``__annotations__`` of a class or module. +In contrast to variable declarations in statically typed languages, +the goal of annotation syntax is to provide an easy way to specify structured +type metadata for third party tools and libraries via the abstract syntax tree +and the ``__annotations__`` attribute. + +.. seealso:: + + :pep:`526` -- Syntax for variable annotations. + PEP written by Ryan Gonzalez, Philip House, Ivan Levkivskyi, Lisa Roach, + and Guido van Rossum. Implemented by Ivan Levkivskyi. + + Tools that use or will use the new syntax: + `mypy <http://github.com/python/mypy>`_, + `pytype <http://github.com/google/pytype>`_, PyCharm, etc. + + +.. _pep-529: + +PEP 529: Change Windows filesystem encoding to UTF-8 +---------------------------------------------------- + +Representing filesystem paths is best performed with str (Unicode) rather than +bytes. However, there are some situations where using bytes is sufficient and +correct. + +Prior to Python 3.6, data loss could result when using bytes paths on Windows. +With this change, using bytes to represent paths is now supported on Windows, +provided those bytes are encoded with the encoding returned by +:func:`sys.getfilesystemencoding()`, which now defaults to ``'utf-8'``. + +Applications that do not use str to represent paths should use +:func:`os.fsencode()` and :func:`os.fsdecode()` to ensure their bytes are +correctly encoded. To revert to the previous behaviour, set +:envvar:`PYTHONLEGACYWINDOWSFSENCODING` or call +:func:`sys._enablelegacywindowsfsencoding`. + +See :pep:`529` for more information and discussion of code modifications that +may be required. + +.. note:: + + This change is considered experimental for 3.6.0 beta releases. The default + encoding may change before the final release. + +.. _whatsnew-pep487: + +PEP 487: Simpler customization of class creation +------------------------------------------------ + +Upon subclassing a class, the ``__init_subclass__`` classmethod (if defined) is +called on the base class. This makes it straightforward to write classes that +customize initialization of future subclasses without introducing the +complexity of a full custom metaclass. + +The descriptor protocol has also been expanded to include a new optional method, +``__set_name__``. Whenever a new class is defined, the new method will be called +on all descriptors included in the definition, providing them with a reference +to the class being defined and the name given to the descriptor within the +class namespace. + +Also see :pep:`487` and the updated class customization documentation at +:ref:`class-customization` and :ref:`descriptors`. + +(Contributed by Martin Teichmann in :issue:`27366`) + +.. _pep-528: + +PEP 528: Change Windows console encoding to UTF-8 +------------------------------------------------- + +The default console on Windows will now accept all Unicode characters and +provide correctly read str objects to Python code. ``sys.stdin``, +``sys.stdout`` and ``sys.stderr`` now default to utf-8 encoding. + +This change only applies when using an interactive console, and not when +redirecting files or pipes. To revert to the previous behaviour for interactive +console use, set :envvar:`PYTHONLEGACYWINDOWSIOENCODING`. + +.. seealso:: + + :pep:`528` -- Change Windows console encoding to UTF-8 + PEP written and implemented by Steve Dower. + +PYTHONMALLOC environment variable +--------------------------------- + +The new :envvar:`PYTHONMALLOC` environment variable allows setting the Python +memory allocators and/or install debug hooks. + +It is now possible to install debug hooks on Python memory allocators on Python +compiled in release mode using ``PYTHONMALLOC=debug``. Effects of debug hooks: + +* Newly allocated memory is filled with the byte ``0xCB`` +* Freed memory is filled with the byte ``0xDB`` +* Detect violations of Python memory allocator API. For example, + :c:func:`PyObject_Free` called on a memory block allocated by + :c:func:`PyMem_Malloc`. +* Detect write before the start of the buffer (buffer underflow) +* Detect write after the end of the buffer (buffer overflow) +* Check that the :term:`GIL <global interpreter lock>` is held when allocator + functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) and + :c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) domains are called. + +Checking if the GIL is held is also a new feature of Python 3.6. + +See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python +memory allocators. + +It is now also possible to force the usage of the :c:func:`malloc` allocator of +the C library for all Python memory allocations using ``PYTHONMALLOC=malloc``. +It helps to use external memory debuggers like Valgrind on a Python compiled in +release mode. + +On error, the debug hooks on Python memory allocators now use the +:mod:`tracemalloc` module to get the traceback where a memory block was +allocated. + +Example of fatal error on buffer overflow using +``python3.6 -X tracemalloc=5`` (store 5 frames in traces):: + + Debug memory block at address p=0x7fbcd41666f8: API 'o' + 4 bytes originally requested + The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. + The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb): + at tail+0: 0x02 *** OUCH + at tail+1: 0xfb + at tail+2: 0xfb + at tail+3: 0xfb + at tail+4: 0xfb + at tail+5: 0xfb + at tail+6: 0xfb + at tail+7: 0xfb + The block was made by call #1233329 to debug malloc/realloc. + Data at p: 1a 2b 30 00 + + Memory block allocated at (most recent call first): + File "test/test_bytes.py", line 323 + File "unittest/case.py", line 600 + File "unittest/case.py", line 648 + File "unittest/suite.py", line 122 + File "unittest/suite.py", line 84 + + Fatal Python error: bad trailing pad byte + + Current thread 0x00007fbcdbd32700 (most recent call first): + File "test/test_bytes.py", line 323 in test_hex + File "unittest/case.py", line 600 in run + File "unittest/case.py", line 648 in __call__ + File "unittest/suite.py", line 122 in run + File "unittest/suite.py", line 84 in __call__ + File "unittest/suite.py", line 122 in run + File "unittest/suite.py", line 84 in __call__ + ... + +(Contributed by Victor Stinner in :issue:`26516` and :issue:`26564`.) + + +DTrace and SystemTap probing support +------------------------------------ + +Python can now be built ``--with-dtrace`` which enables static markers +for the following events in the interpreter: + +* function call/return + +* garbage collection started/finished + +* line of code executed. + +This can be used to instrument running interpreters in production, +without the need to recompile specific debug builds or providing +application-specific profiling/debugging code. + +More details in :ref:`instrumentation`. + +The current implementation is tested on Linux and macOS. Additional +markers may be added in the future. + +(Contributed by Łukasz Langa in :issue:`21590`, based on patches by +Jesús Cea Avión, David Malcolm, and Nikhil Benesch.) + + +.. _whatsnew-deforder: + +PEP 520: Preserving Class Attribute Definition Order +---------------------------------------------------- + +Attributes in a class definition body have a natural ordering: the same +order in which the names appear in the source. This order is now +preserved in the new class's ``__dict__`` attribute. + +Also, the effective default class *execution* namespace (returned from +``type.__prepare__()``) is now an insertion-order-preserving mapping. + +.. seealso:: + + :pep:`520` -- Preserving Class Attribute Definition Order + PEP written and implemented by Eric Snow. + + +.. _whatsnew-kwargs: + +PEP 468: Preserving Keyword Argument Order +------------------------------------------ + +``**kwargs`` in a function signature is now guaranteed to be an +insertion-order-preserving mapping. + +.. seealso:: + + :pep:`468` -- Preserving Keyword Argument Order + PEP written and implemented by Eric Snow. + +.. _whatsnew-pep509: + +PEP 509: Add a private version to dict +-------------------------------------- + +Add a new private version to the builtin ``dict`` type, incremented at +each dictionary creation and at each dictionary change, to implement +fast guards on namespaces. + +(Contributed by Victor Stinner in :issue:`26058`.) + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* :func:`dict` now uses a "compact" representation `pioneered by PyPy + <https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html>`_. + The memory usage of the new :func:`dict` is between 20% and 25% smaller + compared to Python 3.5. + :pep:`468` (Preserving the order of ``**kwargs`` in a function.) is + implemented by this. The order-preserving aspect of this new + implementation is considered an implementation detail and should + not be relied upon (this may change in the future, but it is desired + to have this new dict implementation in the language for a few + releases before changing the language spec to mandate + order-preserving semantics for all current and future Python + implementations; this also helps preserve backwards-compatibility + with older versions of the language where random iteration order is + still in effect, e.g. Python 3.5). + (Contributed by INADA Naoki in :issue:`27350`. Idea + `originally suggested by Raymond Hettinger + <https://mail.python.org/pipermail/python-dev/2012-December/123028.html>`_.) + +* Long sequences of repeated traceback lines are now abbreviated as + ``"[Previous line repeated {count} more times]"`` (see + :ref:`py36-traceback` for an example). + (Contributed by Emanuel Barry in :issue:`26823`.) + +* Import now raises the new exception :exc:`ModuleNotFoundError` + (subclass of :exc:`ImportError`) when it cannot find a module. Code + that current checks for ImportError (in try-except) will still work. + + +New Modules +=========== + +* None yet. + + +Improved Modules +================ + +On Linux, :func:`os.urandom` now blocks until the system urandom entropy pool +is initialized to increase the security. See the :pep:`524` for the rationale. + + +asyncio +------- + +Since the :mod:`asyncio` module is :term:`provisional <provisional api>`, +all changes introduced in Python 3.6 have also been backported to Python +3.5.x. + +Notable changes in the :mod:`asyncio` module since Python 3.5.0: + +* The :func:`~asyncio.ensure_future` function and all functions that + use it, such as :meth:`loop.run_until_complete() <asyncio.BaseEventLoop.run_until_complete>`, + now accept all kinds of :term:`awaitable objects <awaitable>`. + (Contributed by Yury Selivanov.) + +* New :func:`~asyncio.run_coroutine_threadsafe` function to submit + coroutines to event loops from other threads. + (Contributed by Vincent Michel.) + +* New :meth:`Transport.is_closing() <asyncio.BaseTransport.is_closing>` + method to check if the transport is closing or closed. + (Contributed by Yury Selivanov.) + +* The :meth:`loop.create_server() <asyncio.BaseEventLoop.create_server>` + method can now accept a list of hosts. + (Contributed by Yann Sionneau.) + +* New :meth:`loop.create_future() <asyncio.BaseEventLoop.create_future>` + method to create Future objects. This allows alternative event + loop implementations, such as + `uvloop <https://github.com/MagicStack/uvloop>`_, to provide a faster + :class:`asyncio.Future` implementation. + (Contributed by Yury Selivanov.) + +* New :meth:`loop.get_exception_handler() <asyncio.BaseEventLoop.get_exception_handler>` + method to get the current exception handler. + (Contributed by Yury Selivanov.) + +* New :meth:`StreamReader.readuntil() <asyncio.StreamReader.readuntil>` + method to read data from the stream until a separator bytes + sequence appears. + (Contributed by Mark Korenberg.) + +* The :meth:`loop.getaddrinfo() <asyncio.BaseEventLoop.getaddrinfo>` + method is optimized to avoid calling the system ``getaddrinfo`` + function if the address is already resolved. + (Contributed by A. Jesse Jiryu Davis.) + + +contextlib +---------- + +The :class:`contextlib.AbstractContextManager` class has been added to +provide an abstract base class for context managers. It provides a +sensible default implementation for `__enter__()` which returns +``self`` and leaves `__exit__()` an abstract method. A matching +class has been added to the :mod:`typing` module as +:class:`typing.ContextManager`. +(Contributed by Brett Cannon in :issue:`25609`.) + + +venv +---- + +:mod:`venv` accepts a new parameter ``--prompt``. This parameter provides an +alternative prefix for the virtual environment. (Proposed by Łukasz.Balcerzak +and ported to 3.6 by Stéphane Wirtel in :issue:`22829`.) + + +datetime +-------- + +The :meth:`datetime.strftime() <datetime.datetime.strftime>` and +:meth:`date.strftime() <datetime.date.strftime>` methods now support ISO 8601 date +directives ``%G``, ``%u`` and ``%V``. +(Contributed by Ashley Anderson in :issue:`12006`.) + + +distutils.command.sdist +----------------------- + +The ``default_format`` attribute has been removed from +:class:`distutils.command.sdist.sdist` and the ``formats`` +attribute defaults to ``['gztar']``. Although not anticipated, +Any code relying on the presence of ``default_format`` may +need to be adapted. See :issue:`27819` for more details. + + +email +----- + +The new email API, enabled via the *policy* keyword to various constructors, is +no longer provisional. The :mod:`email` documentation has been reorganized and +rewritten to focus on the new API, while retaining the old documentation for +the legacy API. (Contributed by R. David Murray in :issue:`24277`.) + +The :mod:`email.mime` classes now all accept an optional *policy* keyword. +(Contributed by Berker Peksag in :issue:`27331`.) + +The :class:`~email.generator.DecodedGenerator` now supports the *policy* +keyword. + +There is a new :mod:`~email.policy` attribute, +:attr:`~email.policy.Policy.message_factory`, that controls what class is used +by default when the parser creates new message objects. For the +:attr:`email.policy.compat32` policy this is :class:`~email.message.Message`, +for the new policies it is :class:`~email.message.EmailMessage`. +(Contributed by R. David Murray in :issue:`20476`.) + + +encodings +--------- + +On Windows, added the ``'oem'`` encoding to use ``CP_OEMCP`` and the ``'ansi'`` +alias for the existing ``'mbcs'`` encoding, which uses the ``CP_ACP`` code page. + + +faulthandler +------------ + +On Windows, the :mod:`faulthandler` module now installs a handler for Windows +exceptions: see :func:`faulthandler.enable`. (Contributed by Victor Stinner in +:issue:`23848`.) + + +hashlib +------- + +:mod:`hashlib` supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. +It has been tested with 0.9.8zc, 0.9.8zh and 1.0.1t as well as LibreSSL 2.3 +and 2.4. +(Contributed by Christian Heimes in :issue:`26470`.) + +BLAKE2 hash functions were added to the module. :func:`~hashlib.blake2b` +and :func:`~hashlib.blake2s` are always available and support the full +feature set of BLAKE2. +(Contributed by Christian Heimes in :issue:`26798` based on code by +Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.) + +The SHA-3 hash functions :func:`~hashlib.sha3_224`, :func:`~hashlib.sha3_256`, +:func:`~hashlib.sha3_384`, :func:`~hashlib.sha3_512`, and SHAKE hash functions +:func:`~hashlib.shake_128` and :func:`~hashlib.shake_256` were added. +(Contributed by Christian Heimes in :issue:`16113`. Keccak Code Package +by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and +Ronny Van Keer.) + +The password-based key derivation function :func:`~hashlib.scrypt` is now +available with OpenSSL 1.1.0 and newer. +(Contributed by Christian Heimes in :issue:`27928`.) + +http.client +----------- + +:meth:`HTTPConnection.request() <http.client.HTTPConnection.request>` and +:meth:`~http.client.HTTPConnection.endheaders` both now support +chunked encoding request bodies. +(Contributed by Demian Brecht and Rolf Krahl in :issue:`12319`.) + + +idlelib and IDLE +---------------- + +The idlelib package is being modernized and refactored to make IDLE look and work better and to make the code easier to understand, test, and improve. Part of making IDLE look better, especially on Linux and Mac, is using ttk widgets, mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of either. + +'Modernizing' includes renaming and consolidation of idlelib modules. The renaming of files with partial uppercase names is similar to the renaming of, for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a result, imports of idlelib files that worked in 3.5 will usually not work in 3.6. At least a module name change will be needed (see idlelib/README.txt), sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in :issue:`24225`. Most idlelib patches since have been and will be part of the process.) + +In compensation, the eventual result with be that some idlelib classes will be easier to use, with better APIs and docstrings explaining them. Additional useful information will be added to idlelib when available. + + +importlib +--------- + +:class:`importlib.util.LazyLoader` now calls +:meth:`~importlib.abc.Loader.create_module` on the wrapped loader, removing the +restriction that :class:`importlib.machinery.BuiltinImporter` and +:class:`importlib.machinery.ExtensionFileLoader` couldn't be used with +:class:`importlib.util.LazyLoader`. + +:func:`importlib.util.cache_from_source`, +:func:`importlib.util.source_from_cache`, and +:func:`importlib.util.spec_from_file_location` now accept a +:term:`path-like object`. + + +json +---- + +:func:`json.load` and :func:`json.loads` now support binary input. Encoded +JSON should be represented using either UTF-8, UTF-16, or UTF-32. +(Contributed by Serhiy Storchaka in :issue:`17909`.) + + +os +-- + +A new :meth:`~os.scandir.close` method allows explicitly closing a +:func:`~os.scandir` iterator. The :func:`~os.scandir` iterator now +supports the :term:`context manager` protocol. If a :func:`scandir` +iterator is neither exhausted nor explicitly closed a :exc:`ResourceWarning` +will be emitted in its destructor. +(Contributed by Serhiy Storchaka in :issue:`25994`.) + +The Linux ``getrandom()`` syscall (get random bytes) is now exposed as the new +:func:`os.getrandom` function. +(Contributed by Victor Stinner, part of the :pep:`524`) + +See the summary for :ref:`PEP 519 <pep-519>` for details on how the +:mod:`os` and :mod:`os.path` modules now support +:term:`path-like objects <path-like object>`. + + +pickle +------ + +Objects that need calling ``__new__`` with keyword arguments can now be pickled +using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. +Protocol version 4 already supports this case. (Contributed by Serhiy +Storchaka in :issue:`24164`.) + + +re +-- + +Added support of modifier spans in regular expressions. Examples: +``'(?i:p)ython'`` matches ``'python'`` and ``'Python'``, but not ``'PYTHON'``; +``'(?i)g(?-i:v)r'`` matches ``'GvR'`` and ``'gvr'``, but not ``'GVR'``. +(Contributed by Serhiy Storchaka in :issue:`433028`.) + +Match object groups can be accessed by ``__getitem__``, which is +equivalent to ``group()``. So ``mo['name']`` is now equivalent to +``mo.group('name')``. (Contributed by Eric Smith in :issue:`24454`.) + + +readline +-------- + +Added :func:`~readline.set_auto_history` to enable or disable +automatic addition of input to the history list. (Contributed by +Tyler Crompton in :issue:`26870`.) + + +rlcompleter +----------- + +Private and special attribute names now are omitted unless the prefix starts +with underscores. A space or a colon is added after some completed keywords. +(Contributed by Serhiy Storchaka in :issue:`25011` and :issue:`25209`.) + +Names of most attributes listed by :func:`dir` are now completed. +Previously, names of properties and slots which were not yet created on +an instance were excluded. (Contributed by Martin Panter in :issue:`25590`.) + + +site +---- + +When specifying paths to add to :attr:`sys.path` in a `.pth` file, +you may now specify file paths on top of directories (e.g. zip files). +(Contributed by Wolfgang Langner in :issue:`26587`). + + +sqlite3 +------- + +:attr:`sqlite3.Cursor.lastrowid` now supports the ``REPLACE`` statement. +(Contributed by Alex LordThorsen in :issue:`16864`.) + + +socket +------ + +The :func:`~socket.socket.ioctl` function now supports the :data:`~socket.SIO_LOOPBACK_FAST_PATH` +control code. +(Contributed by Daniel Stokes in :issue:`26536`.) + +The :meth:`~socket.socket.getsockopt` constants ``SO_DOMAIN``, +``SO_PROTOCOL``, ``SO_PEERSEC``, and ``SO_PASSSEC`` are now supported. +(Contributed by Christian Heimes in :issue:`26907`.) + +The socket module now supports the address family +:data:`~socket.AF_ALG` to interface with Linux Kernel crypto API. ``ALG_*``, +``SOL_ALG`` and :meth:`~socket.socket.sendmsg_afalg` were added. +(Contributed by Christian Heimes in :issue:`27744` with support from +Victor Stinner.) + +socketserver +------------ + +Servers based on the :mod:`socketserver` module, including those +defined in :mod:`http.server`, :mod:`xmlrpc.server` and +:mod:`wsgiref.simple_server`, now support the :term:`context manager` +protocol. +(Contributed by Aviv Palivoda in :issue:`26404`.) + +The :attr:`~socketserver.StreamRequestHandler.wfile` attribute of +:class:`~socketserver.StreamRequestHandler` classes now implements +the :class:`io.BufferedIOBase` writable interface. In particular, +calling :meth:`~io.BufferedIOBase.write` is now guaranteed to send the +data in full. (Contributed by Martin Panter in :issue:`26721`.) + +ssl +--- + +:mod:`ssl` supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. +It has been tested with 0.9.8zc, 0.9.8zh and 1.0.1t as well as LibreSSL 2.3 +and 2.4. +(Contributed by Christian Heimes in :issue:`26470`.) + +3DES has been removed from the default cipher suites and ChaCha20 Poly1305 +cipher suites are now in the right position. +(Contributed by Christian Heimes in :issue:`27850` and :issue:`27766`.) + +:class:`~ssl.SSLContext` has better default configuration for options +and ciphers. +(Contributed by Christian Heimes in :issue:`28043`.) + +SSL session can be copied from one client-side connection to another +with :class:`~ssl.SSLSession`. TLS session resumption can speed up +the initial handshake, reduce latency and improve performance +(Contributed by Christian Heimes in :issue:`19500` based on a draft by +Alex Warhawk.) + +All constants and flags have been converted to :class:`~enum.IntEnum` and +:class:`~enum.IntFlags`. +(Contributed by Christian Heimes in :issue:`28025`.) + +Server and client-side specific TLS protocols for :class:`~ssl.SSLContext` +were added. +(Contributed by Christian Heimes in :issue:`28085`.) + +General resource ids (``GEN_RID``) in subject alternative name extensions +no longer case a SystemError. +(Contributed by Christian Heimes in :issue:`27691`.) + +subprocess +---------- + +:class:`subprocess.Popen` destructor now emits a :exc:`ResourceWarning` warning +if the child process is still running. Use the context manager protocol (``with +proc: ...``) or call explicitly the :meth:`~subprocess.Popen.wait` method to +read the exit status of the child process (Contributed by Victor Stinner in +:issue:`26741`). + +The :class:`subprocess.Popen` constructor and all functions that pass arguments +through to it now accept *encoding* and *errors* arguments. Specifying either +of these will enable text mode for the *stdin*, *stdout* and *stderr* streams. + +telnetlib +--------- + +:class:`~telnetlib.Telnet` is now a context manager (contributed by +Stéphane Wirtel in :issue:`25485`). + + +tkinter +------- + +Added methods :meth:`~tkinter.Variable.trace_add`, +:meth:`~tkinter.Variable.trace_remove` and :meth:`~tkinter.Variable.trace_info` +in the :class:`tkinter.Variable` class. They replace old methods +:meth:`~tkinter.Variable.trace_variable`, :meth:`~tkinter.Variable.trace`, +:meth:`~tkinter.Variable.trace_vdelete` and +:meth:`~tkinter.Variable.trace_vinfo` that use obsolete Tcl commands and might +not work in future versions of Tcl. +(Contributed by Serhiy Storchaka in :issue:`22115`). + + +.. _py36-traceback: + +traceback +--------- + +Both the traceback module and the interpreter's builtin exception display now +abbreviate long sequences of repeated lines in tracebacks as shown in the +following example:: + + >>> def f(): f() + ... + >>> f() + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + File "<stdin>", line 1, in f + File "<stdin>", line 1, in f + File "<stdin>", line 1, in f + [Previous line repeated 995 more times] + RecursionError: maximum recursion depth exceeded + +(Contributed by Emanuel Barry in :issue:`26823`.) + + +typing +------ + +The :class:`typing.ContextManager` class has been added for +representing :class:`contextlib.AbstractContextManager`. +(Contributed by Brett Cannon in :issue:`25609`.) + + +unicodedata +----------- + +The internal database has been upgraded to use Unicode 9.0.0. (Contributed by +Benjamin Peterson.) + + +unittest.mock +------------- + +The :class:`~unittest.mock.Mock` class has the following improvements: + +* Two new methods, :meth:`Mock.assert_called() + <unittest.mock.Mock.assert_called>` and :meth:`Mock.assert_called_once() + <unittest.mock.Mock.assert_called_once>` to check if the mock object + was called. + (Contributed by Amit Saha in :issue:`26323`.) + + +urllib.request +-------------- + +If a HTTP request has a file or iterable body (other than a +bytes object) but no Content-Length header, rather than +throwing an error, :class:`~urllib.request.AbstractHTTPHandler` now +falls back to use chunked transfer encoding. +(Contributed by Demian Brecht and Rolf Krahl in :issue:`12319`.) + + +urllib.robotparser +------------------ + +:class:`~urllib.robotparser.RobotFileParser` now supports the ``Crawl-delay`` and +``Request-rate`` extensions. +(Contributed by Nikolay Bogoychev in :issue:`16099`.) + + +warnings +-------- + +A new optional *source* parameter has been added to the +:func:`warnings.warn_explicit` function: the destroyed object which emitted a +:exc:`ResourceWarning`. A *source* attribute has also been added to +:class:`warnings.WarningMessage` (contributed by Victor Stinner in +:issue:`26568` and :issue:`26567`). + +When a :exc:`ResourceWarning` warning is logged, the :mod:`tracemalloc` is now +used to try to retrieve the traceback where the detroyed object was allocated. + +Example with the script ``example.py``:: + + import warnings + + def func(): + return open(__file__) + + f = func() + f = None + +Output of the command ``python3.6 -Wd -X tracemalloc=5 example.py``:: + + example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'> + f = None + Object allocated at (most recent call first): + File "example.py", lineno 4 + return open(__file__) + File "example.py", lineno 6 + f = func() + +The "Object allocated at" traceback is new and only displayed if +:mod:`tracemalloc` is tracing Python memory allocations and if the +:mod:`warnings` was already imported. + + +winreg +------ + +Added the 64-bit integer type :data:`REG_QWORD <winreg.REG_QWORD>`. +(Contributed by Clement Rouault in :issue:`23026`.) + + +winsound +-------- + +Allowed keyword arguments to be passed to :func:`Beep <winsound.Beep>`, +:func:`MessageBeep <winsound.MessageBeep>`, and :func:`PlaySound +<winsound.PlaySound>` (:issue:`27982`). + + +xmlrpc.client +------------- + +The module now supports unmarshalling additional data types used by +Apache XML-RPC implementation for numerics and ``None``. +(Contributed by Serhiy Storchaka in :issue:`26885`.) + + +zipfile +------- + +A new :meth:`ZipInfo.from_file() <zipfile.ZipInfo.from_file>` class method +allows making a :class:`~zipfile.ZipInfo` instance from a filesystem file. +A new :meth:`ZipInfo.is_dir() <zipfile.ZipInfo.is_dir>` method can be used +to check if the :class:`~zipfile.ZipInfo` instance represents a directory. +(Contributed by Thomas Kluyver in :issue:`26039`.) + +The :meth:`ZipFile.open() <zipfile.ZipFile.open>` method can now be used to +write data into a ZIP file, as well as for extracting data. +(Contributed by Thomas Kluyver in :issue:`26039`.) + + +zlib +---- + +The :func:`~zlib.compress` function now accepts keyword arguments. +(Contributed by Aviv Palivoda in :issue:`26243`.) + + +fileinput +--------- + +:func:`~fileinput.hook_encoded` now supports the *errors* argument. +(Contributed by Joseph Hackman in :issue:`25788`.) + + +Optimizations +============= + +* The ASCII decoder is now up to 60 times as fast for error handlers + ``surrogateescape``, ``ignore`` and ``replace`` (Contributed + by Victor Stinner in :issue:`24870`). + +* The ASCII and the Latin1 encoders are now up to 3 times as fast for the + error handler ``surrogateescape`` (Contributed by Victor Stinner in :issue:`25227`). + +* The UTF-8 encoder is now up to 75 times as fast for error handlers + ``ignore``, ``replace``, ``surrogateescape``, ``surrogatepass`` (Contributed + by Victor Stinner in :issue:`25267`). + +* The UTF-8 decoder is now up to 15 times as fast for error handlers + ``ignore``, ``replace`` and ``surrogateescape`` (Contributed + by Victor Stinner in :issue:`25301`). + +* ``bytes % args`` is now up to 2 times faster. (Contributed by Victor Stinner + in :issue:`25349`). + +* ``bytearray % args`` is now between 2.5 and 5 times faster. (Contributed by + Victor Stinner in :issue:`25399`). + +* Optimize :meth:`bytes.fromhex` and :meth:`bytearray.fromhex`: they are now + between 2x and 3.5x faster. (Contributed by Victor Stinner in :issue:`25401`). + +* Optimize ``bytes.replace(b'', b'.')`` and ``bytearray.replace(b'', b'.')``: + up to 80% faster. (Contributed by Josh Snider in :issue:`26574`). + +* Allocator functions of the :c:func:`PyMem_Malloc` domain + (:c:data:`PYMEM_DOMAIN_MEM`) now use the :ref:`pymalloc memory allocator + <pymalloc>` instead of :c:func:`malloc` function of the C library. The + pymalloc allocator is optimized for objects smaller or equal to 512 bytes + with a short lifetime, and use :c:func:`malloc` for larger memory blocks. + (Contributed by Victor Stinner in :issue:`26249`). + +* :func:`pickle.load` and :func:`pickle.loads` are now up to 10% faster when + deserializing many small objects (Contributed by Victor Stinner in + :issue:`27056`). + +- Passing :term:`keyword arguments <keyword argument>` to a function has an + overhead in comparison with passing :term:`positional arguments + <positional argument>`. Now in extension functions implemented with using + Argument Clinic this overhead is significantly decreased. + (Contributed by Serhiy Storchaka in :issue:`27574`). + +* Optimized :func:`~glob.glob` and :func:`~glob.iglob` functions in the + :mod:`glob` module; they are now about 3--6 times faster. + (Contributed by Serhiy Storchaka in :issue:`25596`). + +* Optimized globbing in :mod:`pathlib` by using :func:`os.scandir`; + it is now about 1.5--4 times faster. + (Contributed by Serhiy Storchaka in :issue:`26032`). + +Build and C API Changes +======================= + +* Python now requires some C99 support in the toolchain to build. For more + information, see :pep:`7`. + +* Cross-compiling CPython with the Android NDK and the Android API level set to + 21 (Android 5.0 Lollilop) or greater, runs successfully. While Android is not + yet a supported platform, the Python test suite runs on the Android emulator + with only about 16 tests failures. See the Android meta-issue :issue:`26865`. + +* The ``--with-optimizations`` configure flag has been added. Turning it on + will activate LTO and PGO build support (when available). + (Original patch by Alecsandru Patrascu of Intel in :issue:`26539`.) + +* New :c:func:`Py_FinalizeEx` API which indicates if flushing buffered data + failed (:issue:`5319`). + +* :c:func:`PyArg_ParseTupleAndKeywords` now supports :ref:`positional-only + parameters <positional-only_parameter>`. Positional-only parameters are + defined by empty names. + (Contributed by Serhiy Storchaka in :issue:`26282`). + +* ``PyTraceback_Print`` method now abbreviates long sequences of repeated lines + as ``"[Previous line repeated {count} more times]"``. + (Contributed by Emanuel Barry in :issue:`26823`.) + + +Deprecated +========== + +Deprecated Build Options +------------------------ + +The ``--with-system-ffi`` configure flag is now on by default on non-OSX UNIX +platforms. It may be disabled by using ``--without-system-ffi``, but using the +flag is deprecated and will not be accepted in Python 3.7. OSX is unaffected +by this change. Note that many OS distributors already use the +``--with-system-ffi`` flag when building their system Python. + + +New Keywords +------------ + +``async`` and ``await`` are not recommended to be used as variable, class, +function or module names. Introduced by :pep:`492` in Python 3.5, they will +become proper keywords in Python 3.7. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +* :meth:`importlib.machinery.SourceFileLoader.load_module` and + :meth:`importlib.machinery.SourcelessFileLoader.load_module` are now + deprecated. They were the only remaining implementations of + :meth:`importlib.abc.Loader.load_module` in :mod:`importlib` that had not + been deprecated in previous versions of Python in favour of + :meth:`importlib.abc.Loader.exec_module`. + +* The :mod:`tkinter.tix` module is now deprecated. :mod:`tkinter` users should + use :mod:`tkinter.ttk` instead. + + +Deprecated functions and types of the C API +------------------------------------------- + +* None yet. + + +Deprecated features +------------------- + +* The ``pyvenv`` script has been deprecated in favour of ``python3 -m venv``. + This prevents confusion as to what Python interpreter ``pyvenv`` is + connected to and thus what Python interpreter will be used by the virtual + environment. (Contributed by Brett Cannon in :issue:`25154`.) + +* When performing a relative import, falling back on ``__name__`` and + ``__path__`` from the calling module when ``__spec__`` or + ``__package__`` are not defined now raises an :exc:`ImportWarning`. + (Contributed by Rose Ames in :issue:`25791`.) + +* Unlike to other :mod:`dbm` implementations, the :mod:`dbm.dumb` module + creates database in ``'r'`` and ``'w'`` modes if it doesn't exist and + allows modifying database in ``'r'`` mode. This behavior is now deprecated + and will be removed in 3.8. + (Contributed by Serhiy Storchaka in :issue:`21708`.) + +* Undocumented support of general :term:`bytes-like objects <bytes-like object>` + as paths in :mod:`os` functions, :func:`compile` and similar functions is + now deprecated. + (Contributed by Serhiy Storchaka in :issue:`25791` and :issue:`26754`.) + +* The undocumented ``extra_path`` argument to a distutils Distribution + is now considered + deprecated, will raise a warning during install if set. Support for this + parameter will be dropped in a future Python release and likely earlier + through third party tools. See :issue:`27919` for details. + +* A backslash-character pair that is not a valid escape sequence now generates + a DeprecationWarning. Although this will eventually become a SyntaxError, + that will not be for several Python releases. (Contributed by Emanuel Barry + in :issue:`27364`.) + +* Inline flags ``(?letters)`` now should be used only at the start of the + regular expression. Inline flags in the middle of the regular expression + affects global flags in Python :mod:`re` module. This is an exception to + other regular expression engines that either apply flags to only part of + the regular expression or treat them as an error. To avoid distinguishing + inline flags in the middle of the regular expression now emit a deprecation + warning. It will be an error in future Python releases. + (Contributed by Serhiy Storchaka in :issue:`22493`.) + +* SSL-related arguments like ``certfile``, ``keyfile`` and ``check_hostname`` + in :mod:`ftplib`, :mod:`http.client`, :mod:`imaplib`, :mod:`poplib`, + and :mod:`smtplib` have been deprecated in favor of ``context``. + (Contributed by Christian Heimes in :issue:`28022`.) + +* A couple of protocols and functions of the :mod:`ssl` module are now + deprecated. Some features will no longer be available in future versions + of OpenSSL. Other features are deprecated in favor of a different API. + (Contributed by Christian Heimes in :issue:`28022` and :issue:`26470`.) + + +Deprecated Python behavior +-------------------------- + +* Raising the :exc:`StopIteration` exception inside a generator will now generate a + :exc:`DeprecationWarning`, and will trigger a :exc:`RuntimeError` in Python 3.7. + See :ref:`whatsnew-pep-479` for details. + + +Removed +======= + +API and Feature Removals +------------------------ + +* ``inspect.getmoduleinfo()`` was removed (was deprecated since CPython 3.3). + :func:`inspect.getmodulename` should be used for obtaining the module + name for a given path. + +* ``traceback.Ignore`` class and ``traceback.usage``, ``traceback.modname``, + ``traceback.fullmodname``, ``traceback.find_lines_from_code``, + ``traceback.find_lines``, ``traceback.find_strings``, + ``traceback.find_executable_lines`` methods were removed from the + :mod:`traceback` module. They were undocumented methods deprecated since + Python 3.2 and equivalent functionality is available from private methods. + +* The ``tk_menuBar()`` and ``tk_bindForTraversal()`` dummy methods in + :mod:`tkinter` widget classes were removed (corresponding Tk commands + were obsolete since Tk 4.0). + +* The :meth:`~zipfile.ZipFile.open` method of the :class:`zipfile.ZipFile` + class no longer supports the ``'U'`` mode (was deprecated since Python 3.4). + Use :class:`io.TextIOWrapper` for reading compressed text files in + :term:`universal newlines` mode. + +* The undocumented ``IN``, ``CDROM``, ``DLFCN``, ``TYPES``, ``CDIO``, and + ``STROPTS`` modules have been removed. They had been available in the + platform specific ``Lib/plat-*/`` directories, but were chronically out of + date, inconsistently available across platforms, and unmaintained. The + script that created these modules is still available in the source + distribution at :source:`Tools/scripts/h2py.py`. + + +Porting to Python 3.6 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + +Changes in 'python' Command Behavior +------------------------------------ + +* The output of a special Python build with defined ``COUNT_ALLOCS``, + ``SHOW_ALLOC_COUNT`` or ``SHOW_TRACK_COUNT`` macros is now off by + default. It can be re-enabled using the ``-X showalloccount`` option. + It now outputs to ``stderr`` instead of ``stdout``. + (Contributed by Serhiy Storchaka in :issue:`23034`.) + + +Changes in the Python API +------------------------- + +* :mod:`sqlite3` no longer implicitly commit an open transaction before DDL + statements. + +* On Linux, :func:`os.urandom` now blocks until the system urandom entropy pool + is initialized to increase the security. + +* When :meth:`importlib.abc.Loader.exec_module` is defined, + :meth:`importlib.abc.Loader.create_module` must also be defined. + +* :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** + argument is not set. Previously only ``NULL`` was returned. + +* The format of the ``co_lnotab`` attribute of code objects changed to support + negative line number delta. By default, Python does not emit bytecode with + negative line number delta. Functions using ``frame.f_lineno``, + ``PyFrame_GetLineNumber()`` or ``PyCode_Addr2Line()`` are not affected. + Functions decoding directly ``co_lnotab`` should be updated to use a signed + 8-bit integer type for the line number delta, but it's only required to + support applications using negative line number delta. See + ``Objects/lnotab_notes.txt`` for the ``co_lnotab`` format and how to decode + it, and see the :pep:`511` for the rationale. + +* The functions in the :mod:`compileall` module now return booleans instead + of ``1`` or ``0`` to represent success or failure, respectively. Thanks to + booleans being a subclass of integers, this should only be an issue if you + were doing identity checks for ``1`` or ``0``. See :issue:`25768`. + +* Reading the :attr:`~urllib.parse.SplitResult.port` attribute of + :func:`urllib.parse.urlsplit` and :func:`~urllib.parse.urlparse` results + now raises :exc:`ValueError` for out-of-range values, rather than + returning :const:`None`. See :issue:`20059`. + +* The :mod:`imp` module now raises a :exc:`DeprecationWarning` instead of + :exc:`PendingDeprecationWarning`. + +* The following modules have had missing APIs added to their :attr:`__all__` + attributes to match the documented APIs: + :mod:`calendar`, :mod:`cgi`, :mod:`csv`, + :mod:`~xml.etree.ElementTree`, :mod:`enum`, + :mod:`fileinput`, :mod:`ftplib`, :mod:`logging`, :mod:`mailbox`, + :mod:`mimetypes`, :mod:`optparse`, :mod:`plistlib`, :mod:`smtpd`, + :mod:`subprocess`, :mod:`tarfile`, :mod:`threading` and + :mod:`wave`. This means they will export new symbols when ``import *`` + is used. See :issue:`23883`. + +* When performing a relative import, if ``__package__`` does not compare equal + to ``__spec__.parent`` then :exc:`ImportWarning` is raised. + (Contributed by Brett Cannon in :issue:`25791`.) + +* When a relative import is performed and no parent package is known, then + :exc:`ImportError` will be raised. Previously, :exc:`SystemError` could be + raised. (Contributed by Brett Cannon in :issue:`18018`.) + +* Servers based on the :mod:`socketserver` module, including those + defined in :mod:`http.server`, :mod:`xmlrpc.server` and + :mod:`wsgiref.simple_server`, now only catch exceptions derived + from :exc:`Exception`. Therefore if a request handler raises + an exception like :exc:`SystemExit` or :exc:`KeyboardInterrupt`, + :meth:`~socketserver.BaseServer.handle_error` is no longer called, and + the exception will stop a single-threaded server. (Contributed by + Martin Panter in :issue:`23430`.) + +* :func:`spwd.getspnam` now raises a :exc:`PermissionError` instead of + :exc:`KeyError` if the user doesn't have privileges. + +* The :meth:`socket.socket.close` method now raises an exception if + an error (e.g. EBADF) was reported by the underlying system call. + See :issue:`26685`. + +* The *decode_data* argument for :class:`smtpd.SMTPChannel` and + :class:`smtpd.SMTPServer` constructors is now ``False`` by default. + This means that the argument passed to + :meth:`~smtpd.SMTPServer.process_message` is now a bytes object by + default, and ``process_message()`` will be passed keyword arguments. + Code that has already been updated in accordance with the deprecation + warning generated by 3.5 will not be affected. + +* All optional parameters of the :func:`~json.dump`, :func:`~json.dumps`, + :func:`~json.load` and :func:`~json.loads` functions and + :class:`~json.JSONEncoder` and :class:`~json.JSONDecoder` class + constructors in the :mod:`json` module are now :ref:`keyword-only + <keyword-only_parameter>`. + (Contributed by Serhiy Storchaka in :issue:`18726`.) + +* As part of :pep:`487`, the handling of keyword arguments passed to + :class:`type` (other than the metaclass hint, ``metaclass``) is now + consistently delegated to :meth:`object.__init_subclass__`. This means that + :meth:`type.__new__` and :meth:`type.__init__` both now accept arbitrary + keyword arguments, but :meth:`object.__init_subclass__` (which is called from + :meth:`type.__new__`) will reject them by default. Custom metaclasses + accepting additional keyword arguments will need to adjust their calls to + :meth:`type.__new__` (whether direct or via :class:`super`) accordingly. + +* In :class:`distutils.command.sdist.sdist`, the ``default_format`` + attribute has been removed and is no longer honored. Instead, the + gzipped tarfile format is the default on all platforms and no + platform-specific selection is made. + In environments where distributions are + built on Windows and zip distributions are required, configure + the project with a ``setup.cfg`` file containing the following:: + + [sdist] + formats=zip + + This behavior has also been backported to earlier Python versions + by Setuptools 26.0.0. + +* In the :mod:`urllib.request` module and the + :meth:`http.client.HTTPConnection.request` method, if no Content-Length + header field has been specified and the request body is a file object, + it is now sent with HTTP 1.1 chunked encoding. If a file object has to + be sent to a HTTP 1.0 server, the Content-Length value now has to be + specified by the caller. See :issue:`12319`. + +Changes in the C API +-------------------- + +* :c:func:`PyMem_Malloc` allocator family now uses the :ref:`pymalloc allocator + <pymalloc>` rather than system :c:func:`malloc`. Applications calling + :c:func:`PyMem_Malloc` without holding the GIL can now crash. Set the + :envvar:`PYTHONMALLOC` environment variable to ``debug`` to validate the + usage of memory allocators in your application. See :issue:`26249`. + +* :c:func:`Py_Exit` (and the main interpreter) now override the exit status + with 120 if flushing buffered data failed. See :issue:`5319`. diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index edb55022bd..7c9252401e 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 3.6.rst 3.5.rst 3.4.rst 3.3.rst |
