diff options
Diffstat (limited to 'Doc')
189 files changed, 9821 insertions, 2039 deletions
diff --git a/Doc/Makefile b/Doc/Makefile index ea3023157d..a42e98bd9c 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -15,11 +15,12 @@ ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees -D latex_paper_size=$(PAPER) \ .PHONY: help build html htmlhelp latex text changes linkcheck \ suspicious coverage doctest pydoc-topics htmlview clean dist check serve \ - autobuild-dev autobuild-stable + autobuild-dev autobuild-stable venv help: @echo "Please use \`make <target>' where <target> is one of" @echo " clean to remove build files" + @echo " venv to create a venv with necessary tools" @echo " html to make standalone HTML files" @echo " htmlview to open the index page built by the html target in your browser" @echo " htmlhelp to make HTML files and a HTML help project" @@ -95,14 +96,18 @@ doctest: pydoc-topics: BUILDER = pydoc-topics pydoc-topics: build - @echo "Building finished; now copy build/pydoc-topics/topics.py" \ - "to ../Lib/pydoc_data/topics.py" + @echo "Building finished; now run this:" \ + "cp build/pydoc-topics/topics.py ../Lib/pydoc_data/topics.py" htmlview: html $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" clean: - -rm -rf build/* + -rm -rf build/* venv/* + +venv: + $(PYTHON) -m venv venv + ./venv/bin/python3 -m pip install -U Sphinx dist: rm -rf dist @@ -172,4 +177,3 @@ autobuild-stable: exit 1;; \ esac @make autobuild-dev - diff --git a/Doc/README.txt b/Doc/README.txt index 58bda62697..580d10241d 100644 --- a/Doc/README.txt +++ b/Doc/README.txt @@ -3,7 +3,7 @@ Python Documentation README This directory contains the reStructuredText (reST) sources to the Python documentation. You don't need to build them yourself, prebuilt versions are -available at <https://docs.python.org/3.4/download.html>. +available at <https://docs.python.org/dev/download.html>. Documentation on authoring Python documentation, including information about both style and markup, is available in the "Documenting Python" chapter of the diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index 3c0f4b98cd..ed62deaec1 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -152,7 +152,7 @@ Unless otherwise stated, buffers are not NUL-terminated. any conversion. Raises :exc:`TypeError` if the object is not a Unicode object. The C variable may also be declared as :c:type:`PyObject\*`. -``w*`` (:class:`bytearray` or read-write byte-oriented buffer) [Py_buffer] +``w*`` (read-write :term:`bytes-like object`) [Py_buffer] This format accepts any object which implements the read-write buffer interface. It fills a :c:type:`Py_buffer` structure provided by the caller. The buffer may contain embedded null bytes. The caller have to call diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index d099ace9ef..46c19d3ca3 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -96,8 +96,8 @@ a buffer, see :c:func:`PyObject_GetBuffer`. block of the exporter. For example, with negative :c:member:`~Py_buffer.strides` the value may point to the end of the memory block. - For contiguous arrays, the value points to the beginning of the memory - block. + For :term:`contiguous` arrays, the value points to the beginning of + the memory block. .. c:member:: void \*obj @@ -281,11 +281,14 @@ of the flags below it. +-----------------------------+-------+---------+------------+ +.. index:: contiguous, C-contiguous, Fortran contiguous + contiguity requests ~~~~~~~~~~~~~~~~~~~ -C or Fortran contiguity can be explicitly requested, with and without stride -information. Without stride information, the buffer must be C-contiguous. +C or Fortran :term:`contiguity <contiguous>` can be explicitly requested, +with and without stride information. Without stride information, the buffer +must be C-contiguous. .. tabularcolumns:: |p{0.35\linewidth}|l|l|l|l| @@ -466,13 +469,13 @@ Buffer-related functions .. c:function:: int PyBuffer_IsContiguous(Py_buffer *view, char order) Return 1 if the memory defined by the *view* is C-style (*order* is - ``'C'``) or Fortran-style (*order* is ``'F'``) contiguous or either one + ``'C'``) or Fortran-style (*order* is ``'F'``) :term:`contiguous` or either one (*order* is ``'A'``). Return 0 otherwise. .. c:function:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char order) - Fill the *strides* array with byte-strides of a contiguous (C-style if + Fill the *strides* array with byte-strides of a :term:`contiguous` (C-style if *order* is ``'C'`` or Fortran-style if *order* is ``'F'``) array of the given shape with the given number of bytes per element. diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst index 83252afbb7..dfe3d436e5 100644 --- a/Doc/c-api/codec.rst +++ b/Doc/c-api/codec.rst @@ -116,3 +116,8 @@ Registry API for Unicode encoding error handlers Replace the unicode encode error with backslash escapes (``\x``, ``\u`` and ``\U``). +.. c:function:: PyObject* PyCodec_NameReplaceErrors(PyObject *exc) + + Replace the unicode encode error with ``\N{...}`` escapes. + + .. versionadded:: 3.5 diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst index 2d5638670e..47dab81891 100644 --- a/Doc/c-api/concrete.rst +++ b/Doc/c-api/concrete.rst @@ -112,5 +112,6 @@ Other Objects weakref.rst capsule.rst gen.rst + coro.rst datetime.rst diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst new file mode 100644 index 0000000000..2fe50b5d8c --- /dev/null +++ b/Doc/c-api/coro.rst @@ -0,0 +1,34 @@ +.. highlightlang:: c + +.. _coro-objects: + +Coroutine Objects +----------------- + +.. versionadded:: 3.5 + +Coroutine objects are what functions declared with an ``async`` keyword +return. + + +.. c:type:: PyCoroObject + + The C structure used for coroutine objects. + + +.. c:var:: PyTypeObject PyCoro_Type + + The type object corresponding to coroutine objects. + + +.. c:function:: int PyCoro_CheckExact(PyObject *ob) + + Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be *NULL*. + + +.. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname) + + Create and return a new coroutine object based on the *frame* object, + with ``__name__`` and ``__qualname__`` set to *name* and *qualname*. + A reference to *frame* is stolen by this function. The *frame* argument + must not be *NULL*. diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index c2df767998..3fd69ba80f 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -9,13 +9,19 @@ Exception Handling The functions described in this chapter will let you handle and raise Python exceptions. It is important to understand some of the basics of Python -exception handling. It works somewhat like the Unix :c:data:`errno` variable: +exception handling. It works somewhat like the POSIX :c:data:`errno` variable: there is a global indicator (per thread) of the last error that occurred. Most -functions don't clear this on success, but will set it to indicate the cause of -the error on failure. Most functions also return an error indicator, usually -*NULL* if they are supposed to return a pointer, or ``-1`` if they return an -integer (exception: the :c:func:`PyArg_\*` functions return ``1`` for success and -``0`` for failure). +C API functions don't clear this on success, but will set it to indicate the +cause of the error on failure. Most C API functions also return an error +indicator, usually *NULL* if they are supposed to return a pointer, or ``-1`` +if they return an integer (exception: the :c:func:`PyArg_\*` functions +return ``1`` for success and ``0`` for failure). + +Concretely, the error indicator consists of three object pointers: the +exception's type, the exception's value, and the traceback object. Any +of those pointers can be NULL if non-set (although some combinations are +forbidden, for example you can't have a non-NULL traceback if the exception +type is NULL). When a function must fail because some function it called failed, it generally doesn't set the error indicator; the function it called already set it. It is @@ -27,12 +33,21 @@ the caller that an error has been set. If the error is not handled or carefully propagated, additional calls into the Python/C API may not behave as intended and may fail in mysterious ways. -The error indicator consists of three Python objects corresponding to the result -of ``sys.exc_info()``. API functions exist to interact with the error indicator -in various ways. There is a separate error indicator for each thread. +.. note:: + The error indicator is **not** the result of :func:`sys.exc_info()`. + The former corresponds to an exception that is not yet caught (and is + therefore still propagating), while the latter returns an exception after + it is caught (and has therefore stopped propagating). -.. XXX Order of these should be more thoughtful. - Either alphabetical or some kind of structure. + +Printing and clearing +===================== + + +.. c:function:: void PyErr_Clear() + + Clear the error indicator. If the error indicator is not set, there is no + effect. .. c:function:: void PyErr_PrintEx(int set_sys_last_vars) @@ -51,127 +66,24 @@ in various ways. There is a separate error indicator for each thread. Alias for ``PyErr_PrintEx(1)``. -.. c:function:: PyObject* PyErr_Occurred() - - Test whether the error indicator is set. If set, return the exception *type* - (the first argument to the last call to one of the :c:func:`PyErr_Set\*` - functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not - own a reference to the return value, so you do not need to :c:func:`Py_DECREF` - it. - - .. note:: - - Do not compare the return value to a specific exception; use - :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could - easily fail since the exception may be an instance instead of a class, in the - case of a class exception, or it may be a subclass of the expected exception.) - - -.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) - - Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This - should only be called when an exception is actually set; a memory access - violation will occur if no exception has been raised. - - -.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) - - Return true if the *given* exception matches the exception in *exc*. If - *exc* is a class object, this also returns true when *given* is an instance - of a subclass. If *exc* is a tuple, all exceptions in the tuple (and - recursively in subtuples) are searched for a match. - - -.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) - - Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below - can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is - not an instance of the same class. This function can be used to instantiate - the class in that case. If the values are already normalized, nothing happens. - The delayed normalization is implemented to improve performance. - - .. note:: - - This function *does not* implicitly set the ``__traceback__`` - attribute on the exception value. If setting the traceback - appropriately is desired, the following additional snippet is needed:: - - if (tb != NULL) { - PyException_SetTraceback(val, tb); - } - - -.. c:function:: void PyErr_Clear() - - Clear the error indicator. If the error indicator is not set, there is no - effect. - - -.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) - - Retrieve the error indicator into three variables whose addresses are passed. - If the error indicator is not set, set all three variables to *NULL*. If it is - set, it will be cleared and you own a reference to each object retrieved. The - value and traceback object may be *NULL* even when the type object is not. - - .. note:: - - This function is normally only used by code that needs to handle exceptions or - by code that needs to save and restore the error indicator temporarily. - - -.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) - - Set the error indicator from the three objects. If the error indicator is - already set, it is cleared first. If the objects are *NULL*, the error - indicator is cleared. Do not pass a *NULL* type and non-*NULL* value or - traceback. The exception type should be a class. Do not pass an invalid - exception type or value. (Violating these rules will cause subtle problems - later.) This call takes away a reference to each object: you must own a - reference to each object before the call and after the call you no longer own - these references. (If you don't understand this, don't use this function. I - warned you.) - - .. note:: - - This function is normally only used by code that needs to save and restore the - error indicator temporarily; use :c:func:`PyErr_Fetch` to save the current - exception state. - - -.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) - - Retrieve the exception info, as known from ``sys.exc_info()``. This refers - to an exception that was already caught, not to an exception that was - freshly raised. Returns new references for the three objects, any of which - may be *NULL*. Does not modify the exception info state. - - .. note:: - - This function is not normally used by code that wants to handle exceptions. - Rather, it can be used when code needs to save and restore the exception - state temporarily. Use :c:func:`PyErr_SetExcInfo` to restore or clear the - exception state. - - .. versionadded:: 3.3 - +.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) -.. c:function:: void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) + This utility function prints a warning message to ``sys.stderr`` when an + exception has been set but it is impossible for the interpreter to actually + raise the exception. It is used, for example, when an exception occurs in an + :meth:`__del__` method. - Set the exception info, as known from ``sys.exc_info()``. This refers - to an exception that was already caught, not to an exception that was - freshly raised. This function steals the references of the arguments. - To clear the exception state, pass *NULL* for all three arguments. - For general rules about the three arguments, see :c:func:`PyErr_Restore`. + The function is called with a single argument *obj* that identifies the context + in which the unraisable exception occurred. The repr of *obj* will be printed in + the warning message. - .. note:: - This function is not normally used by code that wants to handle exceptions. - Rather, it can be used when code needs to save and restore the exception - state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception - state. +Raising exceptions +================== - .. versionadded:: 3.3 +These functions help you set the current thread's error indicator. +For convenience, some of these functions will always return a +NULL pointer for use in a ``return`` statement. .. c:function:: void PyErr_SetString(PyObject *type, const char *message) @@ -197,6 +109,14 @@ in various ways. There is a separate error indicator for each thread. string. +.. c:function:: PyObject* PyErr_FormatV(PyObject *exception, const char *format, va_list vargs) + + Same as :c:func:`PyErr_Format`, but taking a :c:type:`va_list` argument rather + than a variable number of arguments. + + .. versionadded:: 3.5 + + .. c:function:: void PyErr_SetNone(PyObject *type) This is a shorthand for ``PyErr_SetObject(type, Py_None)``. @@ -346,6 +266,22 @@ in various ways. There is a separate error indicator for each thread. use. +Issuing warnings +================ + +Use these functions to issue warnings from C code. They mirror similar +functions exported by the Python :mod:`warnings` module. They normally +print a warning message to *sys.stderr*; however, it is +also possible that the user has specified that warnings are to be turned into +errors, and in that case they will raise an exception. It is also possible that +the functions raise an exception because of a problem with the warning machinery. +The return value is ``0`` if no exception is raised, or ``-1`` if an exception +is raised. (It is not possible to determine whether a warning message is +actually printed, nor what the reason is for the exception; this is +intentional.) If an exception is raised, the caller should do its normal +exception handling (for example, :c:func:`Py_DECREF` owned references and return +an error value). + .. c:function:: int PyErr_WarnEx(PyObject *category, const char *message, Py_ssize_t stack_level) Issue a warning message. The *category* argument is a warning category (see @@ -355,18 +291,6 @@ in various ways. There is a separate error indicator for each thread. is the function calling :c:func:`PyErr_WarnEx`, 2 is the function above that, and so forth. - This function normally prints a warning message to *sys.stderr*; however, it is - also possible that the user has specified that warnings are to be turned into - errors, and in that case this will raise an exception. It is also possible that - the function raises an exception because of a problem with the warning machinery - (the implementation imports the :mod:`warnings` module to do the heavy lifting). - The return value is ``0`` if no exception is raised, or ``-1`` if an exception - is raised. (It is not possible to determine whether a warning message is - actually printed, nor what the reason is for the exception; this is - intentional.) If an exception is raised, the caller should do its normal - exception handling (for example, :c:func:`Py_DECREF` owned references and return - an error value). - Warning categories must be subclasses of :c:data:`Warning`; the default warning category is :c:data:`RuntimeWarning`. The standard Python warning categories are available as global variables whose names are ``PyExc_`` followed by the Python @@ -410,6 +334,139 @@ in various ways. There is a separate error indicator for each thread. .. versionadded:: 3.2 +Querying the error indicator +============================ + +.. c:function:: PyObject* PyErr_Occurred() + + Test whether the error indicator is set. If set, return the exception *type* + (the first argument to the last call to one of the :c:func:`PyErr_Set\*` + functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not + own a reference to the return value, so you do not need to :c:func:`Py_DECREF` + it. + + .. note:: + + Do not compare the return value to a specific exception; use + :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could + easily fail since the exception may be an instance instead of a class, in the + case of a class exception, or it may be a subclass of the expected exception.) + + +.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) + + Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This + should only be called when an exception is actually set; a memory access + violation will occur if no exception has been raised. + + +.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) + + Return true if the *given* exception matches the exception type in *exc*. If + *exc* is a class object, this also returns true when *given* is an instance + of a subclass. If *exc* is a tuple, all exception types in the tuple (and + recursively in subtuples) are searched for a match. + + +.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the error indicator into three variables whose addresses are passed. + If the error indicator is not set, set all three variables to *NULL*. If it is + set, it will be cleared and you own a reference to each object retrieved. The + value and traceback object may be *NULL* even when the type object is not. + + .. note:: + + This function is normally only used by code that needs to catch exceptions or + by code that needs to save and restore the error indicator temporarily, e.g.:: + + { + PyObject **type, **value, **traceback; + PyErr_Fetch(&type, &value, &traceback); + + /* ... code that might produce other errors ... */ + + PyErr_Restore(type, value, traceback); + } + + +.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) + + Set the error indicator from the three objects. If the error indicator is + already set, it is cleared first. If the objects are *NULL*, the error + indicator is cleared. Do not pass a *NULL* type and non-*NULL* value or + traceback. The exception type should be a class. Do not pass an invalid + exception type or value. (Violating these rules will cause subtle problems + later.) This call takes away a reference to each object: you must own a + reference to each object before the call and after the call you no longer own + these references. (If you don't understand this, don't use this function. I + warned you.) + + .. note:: + + This function is normally only used by code that needs to save and restore the + error indicator temporarily. Use :c:func:`PyErr_Fetch` to save the current + error indicator. + + +.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) + + Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below + can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is + not an instance of the same class. This function can be used to instantiate + the class in that case. If the values are already normalized, nothing happens. + The delayed normalization is implemented to improve performance. + + .. note:: + + This function *does not* implicitly set the ``__traceback__`` + attribute on the exception value. If setting the traceback + appropriately is desired, the following additional snippet is needed:: + + if (tb != NULL) { + PyException_SetTraceback(val, tb); + } + + +.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the exception info, as known from ``sys.exc_info()``. This refers + to an exception that was *already caught*, not to an exception that was + freshly raised. Returns new references for the three objects, any of which + may be *NULL*. Does not modify the exception info state. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_SetExcInfo` to restore or clear the + exception state. + + .. versionadded:: 3.3 + + +.. c:function:: void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) + + Set the exception info, as known from ``sys.exc_info()``. This refers + to an exception that was *already caught*, not to an exception that was + freshly raised. This function steals the references of the arguments. + To clear the exception state, pass *NULL* for all three arguments. + For general rules about the three arguments, see :c:func:`PyErr_Restore`. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception + state. + + .. versionadded:: 3.3 + + +Signal Handling +=============== + + .. c:function:: int PyErr_CheckSignals() .. index:: @@ -443,13 +500,21 @@ in various ways. There is a separate error indicator for each thread. .. c:function:: int PySignal_SetWakeupFd(int fd) - This utility function specifies a file descriptor to which a ``'\0'`` byte will - be written whenever a signal is received. It returns the previous such file - descriptor. The value ``-1`` disables the feature; this is the initial state. + This utility function specifies a file descriptor to which the signal number + is written as a single byte whenever a signal is received. *fd* must be + non-blocking. It returns the previous such file descriptor. + + The value ``-1`` disables the feature; this is the initial state. This is equivalent to :func:`signal.set_wakeup_fd` in Python, but without any error checking. *fd* should be a valid file descriptor. The function should only be called from the main thread. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + + +Exception Classes +================= .. c:function:: PyObject* PyErr_NewException(const char *name, PyObject *base, PyObject *dict) @@ -475,18 +540,6 @@ in various ways. There is a separate error indicator for each thread. .. versionadded:: 3.2 -.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) - - This utility function prints a warning message to ``sys.stderr`` when an - exception has been set but it is impossible for the interpreter to actually - raise the exception. It is used, for example, when an exception occurs in an - :meth:`__del__` method. - - The function is called with a single argument *obj* that identifies the context - in which the unraisable exception occurred. The repr of *obj* will be printed in - the warning message. - - Exception Objects ================= @@ -630,12 +683,12 @@ recursion depth automatically). sets a :exc:`MemoryError` and returns a nonzero value. The function then checks if the recursion limit is reached. If this is the - case, a :exc:`RuntimeError` is set and a nonzero value is returned. + case, a :exc:`RecursionError` is set and a nonzero value is returned. Otherwise, zero is returned. *where* should be a string such as ``" in instance check"`` to be - concatenated to the :exc:`RuntimeError` message caused by the recursion depth - limit. + concatenated to the :exc:`RecursionError` message caused by the recursion + depth limit. .. c:function:: void Py_LeaveRecursiveCall() @@ -747,6 +800,8 @@ the variables: +-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_ProcessLookupError` | :exc:`ProcessLookupError` | | +-----------------------------------------+---------------------------------+----------+ +| :c:data:`PyExc_RecursionError` | :exc:`RecursionError` | | ++-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_ReferenceError` | :exc:`ReferenceError` | \(2) | +-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_RuntimeError` | :exc:`RuntimeError` | | @@ -776,6 +831,9 @@ the variables: :c:data:`PyExc_PermissionError`, :c:data:`PyExc_ProcessLookupError` and :c:data:`PyExc_TimeoutError` were introduced following :pep:`3151`. +.. versionadded:: 3.5 + :c:data:`PyExc_RecursionError`. + These are compatibility aliases to :c:data:`PyExc_OSError`: @@ -824,6 +882,7 @@ These are compatibility aliases to :c:data:`PyExc_OSError`: single: PyExc_OverflowError single: PyExc_PermissionError single: PyExc_ProcessLookupError + single: PyExc_RecursionError single: PyExc_ReferenceError single: PyExc_RuntimeError single: PyExc_SyntaxError diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst index 33cd27a5aa..3ab073b191 100644 --- a/Doc/c-api/gen.rst +++ b/Doc/c-api/gen.rst @@ -7,7 +7,7 @@ Generator Objects Generator objects are what Python uses to implement generator iterators. They are normally created by iterating over a function that yields values, rather -than explicitly calling :c:func:`PyGen_New`. +than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`. .. c:type:: PyGenObject @@ -20,19 +20,25 @@ than explicitly calling :c:func:`PyGen_New`. The type object corresponding to generator objects -.. c:function:: int PyGen_Check(ob) +.. c:function:: int PyGen_Check(PyObject *ob) Return true if *ob* is a generator object; *ob* must not be *NULL*. -.. c:function:: int PyGen_CheckExact(ob) +.. c:function:: int PyGen_CheckExact(PyObject *ob) - Return true if *ob*'s type is *PyGen_Type* is a generator object; *ob* must not - be *NULL*. + Return true if *ob*'s type is *PyGen_Type*; *ob* must not be *NULL*. .. c:function:: PyObject* PyGen_New(PyFrameObject *frame) - Create and return a new generator object based on the *frame* object. A - reference to *frame* is stolen by this function. The parameter must not be + Create and return a new generator object based on the *frame* object. + A reference to *frame* is stolen by this function. The argument must not be *NULL*. + +.. c:function:: PyObject* PyGen_NewWithQualName(PyFrameObject *frame, PyObject *name, PyObject *qualname) + + Create and return a new generator object based on the *frame* object, + with ``__name__`` and ``__qualname__`` set to *name* and *qualname*. + A reference to *frame* is stolen by this function. The *frame* argument + must not be *NULL*. diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst index 60865f456f..bf1b49577e 100644 --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -183,9 +183,9 @@ Importing Modules .. c:function:: long PyImport_GetMagicNumber() - Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` and - :file:`.pyo` files). The magic number should be present in the first four bytes - of the bytecode file, in little-endian byte order. Returns -1 on error. + Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` file). + The magic number should be present in the first four bytes of the bytecode + file, in little-endian byte order. Returns -1 on error. .. versionchanged:: 3.3 Return value of -1 upon failure. diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 4bb50649ef..81823bf383 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -134,6 +134,9 @@ Process-wide parameters change for the duration of the program's execution. No code in the Python interpreter will change the contents of this storage. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. c:function:: wchar* Py_GetProgramName() @@ -245,6 +248,9 @@ Process-wide parameters :data:`sys.exec_prefix` to be empty. It is up to the caller to modify these if required after calling :c:func:`Py_Initialize`. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + The path argument is copied internally, so the caller may free it after the call completes. @@ -344,6 +350,9 @@ Process-wide parameters :data:`sys.path`, which is the same as prepending the current working directory (``"."``). + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. note:: It is recommended that applications embedding the Python interpreter for purposes other than executing a single script pass 0 as *updatepath*, @@ -368,6 +377,9 @@ Process-wide parameters to 1 unless the :program:`python` interpreter was started with the :option:`-I`. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. versionchanged:: 3.4 The *updatepath* value depends on :option:`-I`. @@ -382,6 +394,9 @@ Process-wide parameters execution. No code in the Python interpreter will change the contents of this storage. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. c:function:: w_char* Py_GetPythonHome() @@ -858,6 +873,8 @@ been created. instead. +.. _sub-interpreter-support: + Sub-interpreter support ======================= diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 7908622d2b..7339006b5e 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -92,8 +92,8 @@ functions are thread-safe, the :term:`GIL <global interpreter lock>` does not need to be held. The default raw memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when -requesting zero bytes. +:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call +``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. .. versionadded:: 3.4 @@ -106,6 +106,17 @@ requesting zero bytes. been initialized in any way. +.. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. Requesting zero elements + or elements of size zero bytes returns a distinct non-*NULL* pointer if + possible, as if ``PyMem_RawCalloc(1, 1)`` had been called instead. + + .. versionadded:: 3.5 + + .. c:function:: void* PyMem_RawRealloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will @@ -136,8 +147,8 @@ behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap. The default memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when -requesting zero bytes. +:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call +``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. .. warning:: @@ -152,6 +163,17 @@ requesting zero bytes. been called instead. The memory will not have been initialized in any way. +.. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. Requesting zero elements + or elements of size zero bytes returns a distinct non-*NULL* pointer if + possible, as if ``PyMem_Calloc(1, 1)`` had been called instead. + + .. versionadded:: 3.5 + + .. c:function:: void* PyMem_Realloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will be @@ -210,7 +232,7 @@ Customize Memory Allocators .. versionadded:: 3.4 -.. c:type:: PyMemAllocator +.. c:type:: PyMemAllocatorEx Structure used to describe a memory block allocator. The structure has four fields: @@ -222,11 +244,19 @@ Customize Memory Allocators +----------------------------------------------------------+---------------------------------------+ | ``void* malloc(void *ctx, size_t size)`` | allocate a memory block | +----------------------------------------------------------+---------------------------------------+ + | ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized | + | | with zeros | + +----------------------------------------------------------+---------------------------------------+ | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block | +----------------------------------------------------------+---------------------------------------+ | ``void free(void *ctx, void *ptr)`` | free a memory block | +----------------------------------------------------------+---------------------------------------+ + .. versionchanged:: 3.5 + The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + + .. c:type:: PyMemAllocatorDomain Enum used to identify an allocator domain. Domains: @@ -239,12 +269,12 @@ Customize Memory Allocators :c:func:`PyObject_Realloc` and :c:func:`PyObject_Free` -.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) Get the memory block allocator of the specified domain. -.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) Set the memory block allocator of the specified domain. diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst index 5e50977cbe..9f6bfd751a 100644 --- a/Doc/c-api/memoryview.rst +++ b/Doc/c-api/memoryview.rst @@ -35,7 +35,7 @@ any other object. .. c:function:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) - Create a memoryview object to a contiguous chunk of memory (in either + Create a memoryview object to a :term:`contiguous` chunk of memory (in either 'C' or 'F'ortran *order*) from an object that defines the buffer interface. If memory is contiguous, the memoryview object points to the original memory. Otherwise, a copy is made and the memoryview points to a diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 985a347d63..ef778ccaed 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -7,8 +7,6 @@ Module Objects .. index:: object: module -There are only a few functions special to module objects. - .. c:var:: PyTypeObject PyModule_Type @@ -84,6 +82,18 @@ There are only a few functions special to module objects. Similar to :c:func:`PyModule_GetNameObject` but return the name encoded to ``'utf-8'``. +.. c:function:: void* PyModule_GetState(PyObject *module) + + Return the "state" of the module, that is, a pointer to the block of memory + allocated at module creation time, or *NULL*. See + :c:member:`PyModuleDef.m_size`. + + +.. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module) + + Return a pointer to the :c:type:`PyModuleDef` struct from which the module was + created, or *NULL* if the module wasn't created from a definition. + .. c:function:: PyObject* PyModule_GetFilenameObject(PyObject *module) @@ -109,55 +119,109 @@ There are only a few functions special to module objects. unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead. -.. c:function:: void* PyModule_GetState(PyObject *module) +.. _initializing-modules: - Return the "state" of the module, that is, a pointer to the block of memory - allocated at module creation time, or *NULL*. See - :c:member:`PyModuleDef.m_size`. +Initializing C modules +^^^^^^^^^^^^^^^^^^^^^^ +Modules objects are usually created from extension modules (shared libraries +which export an initialization function), or compiled-in modules +(where the initialization function is added using :c:func:`PyImport_AppendInittab`). +See :ref:`building` or :ref:`extending-with-embedding` for details. -.. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module) +The initialization function can either pass pass a module definition instance +to :c:func:`PyModule_Create`, and return the resulting module object, +or request "multi-phase initialization" by returning the definition struct itself. - Return a pointer to the :c:type:`PyModuleDef` struct from which the module was - created, or *NULL* if the module wasn't created with - :c:func:`PyModule_Create`. +.. c:type:: PyModuleDef -.. c:function:: PyObject* PyState_FindModule(PyModuleDef *def) + The module definition struct, which holds all information needed to create + a module object. There is usually only one statically initialized variable + of this type for each module. - Returns the module object that was created from *def* for the current interpreter. - This method requires that the module object has been attached to the interpreter state with - :c:func:`PyState_AddModule` beforehand. In case the corresponding module object is not - found or has not been attached to the interpreter state yet, it returns NULL. + .. c:member:: PyModuleDef_Base m_base -.. c:function:: int PyState_AddModule(PyObject *module, PyModuleDef *def) + Always initialize this member to :const:`PyModuleDef_HEAD_INIT`. - Attaches the module object passed to the function to the interpreter state. This allows - the module object to be accessible via - :c:func:`PyState_FindModule`. + .. c:member:: char* m_name - .. versionadded:: 3.3 + Name for the new module. -.. c:function:: int PyState_RemoveModule(PyModuleDef *def) + .. c:member:: char* m_doc - Removes the module object created from *def* from the interpreter state. + Docstring for the module; usually a docstring variable created with + :c:func:`PyDoc_STRVAR` is used. - .. versionadded:: 3.3 + .. c:member:: Py_ssize_t m_size -Initializing C modules -^^^^^^^^^^^^^^^^^^^^^^ + Module state may be kept in a per-module memory area that can be + retrieved with :c:func:`PyModule_GetState`, rather than in static globals. + This makes modules safe for use in multiple sub-interpreters. + + This memory area is allocated based on *m_size* on module creation, + and freed when the module object is deallocated, after the + :c:member:`m_free` function has been called, if present. + + Setting ``m_size`` to ``-1`` means that the module does not support + sub-interpreters, because it has global state. -These functions are usually used in the module initialization function. + Setting it to a non-negative value means that the module can be + re-initialized and specifies the additional amount of memory it requires + for its state. Non-negative ``m_size`` is required for multi-phase + initialization. -.. c:function:: PyObject* PyModule_Create(PyModuleDef *module) + See :PEP:`3121` for more details. + + .. c:member:: PyMethodDef* m_methods - Create a new module object, given the definition in *module*. This behaves + A pointer to a table of module-level functions, described by + :c:type:`PyMethodDef` values. Can be *NULL* if no functions are present. + + .. c:member:: PyModuleDef_Slot* m_slots + + An array of slot definitions for multi-phase initialization, terminated by + a ``{0, NULL}`` entry. + When using single-phase initialization, *m_slots* must be *NULL*. + + .. versionchanged:: 3.5 + + Prior to version 3.5, this member was always set to *NULL*, + and was defined as: + + .. c:member:: inquiry m_reload + + .. c:member:: traverseproc m_traverse + + A traversal function to call during GC traversal of the module object, or + *NULL* if not needed. + + .. c:member:: inquiry m_clear + + A clear function to call during GC clearing of the module object, or + *NULL* if not needed. + + .. c:member:: freefunc m_free + + A function to call during deallocation of the module object, or *NULL* if + not needed. + +Single-phase initialization +........................... + +The module initialization function may create and return the module object +directly. This is referred to as "single-phase initialization", and uses one +of the following two module creation functions: + +.. c:function:: PyObject* PyModule_Create(PyModuleDef *def) + + Create a new module object, given the definition in *def*. This behaves like :c:func:`PyModule_Create2` with *module_api_version* set to :const:`PYTHON_API_VERSION`. -.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version) +.. c:function:: PyObject* PyModule_Create2(PyModuleDef *def, int module_api_version) - Create a new module object, given the definition in *module*, assuming the + Create a new module object, given the definition in *def*, assuming the API version *module_api_version*. If that version does not match the version of the running interpreter, a :exc:`RuntimeWarning` is emitted. @@ -166,69 +230,179 @@ These functions are usually used in the module initialization function. Most uses of this function should be using :c:func:`PyModule_Create` instead; only use this if you are sure you need it. +Before it is returned from in the initialization function, the resulting module +object is typically populated using functions like :c:func:`PyModule_AddObject`. -.. c:type:: PyModuleDef +.. _multi-phase-initialization: - This struct holds all information that is needed to create a module object. - There is usually only one static variable of that type for each module, which - is statically initialized and then passed to :c:func:`PyModule_Create` in the - module initialization function. +Multi-phase initialization +.......................... - .. c:member:: PyModuleDef_Base m_base +An alternate way to specify extensions is to request "multi-phase initialization". +Extension modules created this way behave more like Python modules: the +initialization is split between the *creation phase*, when the module object +is created, and the *execution phase*, when it is populated. +The distinction is similar to the :py:meth:`__new__` and :py:meth:`__init__` methods +of classes. - Always initialize this member to :const:`PyModuleDef_HEAD_INIT`. +Unlike modules created using single-phase initialization, these modules are not +singletons: if the *sys.modules* entry is removed and the module is re-imported, +a new module object is created, and the old module is subject to normal garbage +collection -- as with Python modules. +By default, multiple modules created from the same definition should be +independent: changes to one should not affect the others. +This means that all state should be specific to the module object (using e.g. +using :c:func:`PyModule_GetState`), or its contents (such as the module's +:attr:`__dict__` or individual classes created with :c:func:`PyType_FromSpec`). - .. c:member:: char* m_name +All modules created using multi-phase initialization are expected to support +:ref:`sub-interpreters <sub-interpreter-support>`. Making sure multiple modules +are independent is typically enough to achieve this. - Name for the new module. +To request multi-phase initialization, the initialization function +(PyInit_modulename) returns a :c:type:`PyModuleDef` instance with non-empty +:c:member:`~PyModuleDef.m_slots`. Before it is returned, the ``PyModuleDef`` +instance must be initialized with the following function: - .. c:member:: char* m_doc +.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *def) - Docstring for the module; usually a docstring variable created with - :c:func:`PyDoc_STRVAR` is used. + Ensures a module definition is a properly initialized Python object that + correctly reports its type and reference count. - .. c:member:: Py_ssize_t m_size + Returns *def* cast to ``PyObject*``, or *NULL* if an error occurred. - Some modules allow re-initialization (calling their ``PyInit_*`` function - more than once). These modules should keep their state in a per-module - memory area that can be retrieved with :c:func:`PyModule_GetState`. + .. versionadded:: 3.5 - This memory should be used, rather than static globals, to hold per-module - state, since it is then safe for use in multiple sub-interpreters. It is - freed when the module object is deallocated, after the :c:member:`m_free` - function has been called, if present. +The *m_slots* member of the module definition must point to an array of +``PyModuleDef_Slot`` structures: - Setting ``m_size`` to ``-1`` means that the module can not be - re-initialized because it has global state. Setting it to a non-negative - value means that the module can be re-initialized and specifies the - additional amount of memory it requires for its state. +.. c:type:: PyModuleDef_Slot - See :PEP:`3121` for more details. + .. c:member:: int slot - .. c:member:: PyMethodDef* m_methods + A slot ID, chosen from the available values explained below. - A pointer to a table of module-level functions, described by - :c:type:`PyMethodDef` values. Can be *NULL* if no functions are present. + .. c:member:: void* value - .. c:member:: inquiry m_reload + Value of the slot, whose meaning depends on the slot ID. - Currently unused, should be *NULL*. + .. versionadded:: 3.5 - .. c:member:: traverseproc m_traverse +The *m_slots* array must be terminated by a slot with id 0. - A traversal function to call during GC traversal of the module object, or - *NULL* if not needed. +The available slot types are: - .. c:member:: inquiry m_clear +.. c:var:: Py_mod_create - A clear function to call during GC clearing of the module object, or - *NULL* if not needed. + Specifies a function that is called to create the module object itself. + The *value* pointer of this slot must point to a function of the signature: - .. c:member:: freefunc m_free + .. c:function:: PyObject* create_module(PyObject *spec, PyModuleDef *def) - A function to call during deallocation of the module object, or *NULL* if - not needed. + The function receives a :py:class:`~importlib.machinery.ModuleSpec` + instance, as defined in :PEP:`451`, and the module definition. + It should return a new module object, or set an error + and return *NULL*. + + This function should be kept minimal. In particular, it should not + call arbitrary Python code, as trying to import the same module again may + result in an infinite loop. + + Multiple ``Py_mod_create`` slots may not be specified in one module + definition. + + If ``Py_mod_create`` is not specified, the import machinery will create + a normal module object using :c:func:`PyModule_New`. The name is taken from + *spec*, not the definition, to allow extension modules to dynamically adjust + to their place in the module hierarchy and be imported under different + names through symlinks, all while sharing a single module definition. + + There is no requirement for the returned object to be an instance of + :c:type:`PyModule_Type`. Any type can be used, as long as it supports + setting and getting import-related attributes. + However, only ``PyModule_Type`` instances may be returned if the + ``PyModuleDef`` has non-*NULL* ``m_methods``, ``m_traverse``, ``m_clear``, + ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``. + +.. c:var:: Py_mod_exec + + Specifies a function that is called to *execute* the module. + This is equivalent to executing the code of a Python module: typically, + this function adds classes and constants to the module. + The signature of the function is: + + .. c:function:: int exec_module(PyObject* module) + + If multiple ``Py_mod_exec`` slots are specified, they are processed in the + order they appear in the *m_slots* array. + +See :PEP:`489` for more details on multi-phase initialization. + +Low-level module creation functions +................................... + +The following functions are called under the hood when using multi-phase +initialization. They can be used directly, for example when creating module +objects dynamically. Note that both ``PyModule_FromDefAndSpec`` and +``PyModule_ExecDef`` must be called to fully initialize a module. + +.. c:function:: PyObject * PyModule_FromDefAndSpec(PyModuleDef *def, PyObject *spec) + + Create a new module object, given the definition in *module* and the + ModuleSpec *spec*. This behaves like :c:func:`PyModule_FromDefAndSpec2` + with *module_api_version* set to :const:`PYTHON_API_VERSION`. + + .. versionadded:: 3.5 + +.. c:function:: PyObject * PyModule_FromDefAndSpec2(PyModuleDef *def, PyObject *spec, int module_api_version) + + Create a new module object, given the definition in *module* and the + ModuleSpec *spec*, assuming the API version *module_api_version*. + If that version does not match the version of the running interpreter, + a :exc:`RuntimeWarning` is emitted. + + .. note:: + Most uses of this function should be using :c:func:`PyModule_FromDefAndSpec` + instead; only use this if you are sure you need it. + + .. versionadded:: 3.5 + +.. c:function:: int PyModule_ExecDef(PyObject *module, PyModuleDef *def) + + Process any execution slots (:c:data:`Py_mod_exec`) given in *def*. + + .. versionadded:: 3.5 + +.. c:function:: int PyModule_SetDocString(PyObject *module, const char *docstring) + + Set the docstring for *module* to *docstring*. + This function is called automatically when creating a module from + ``PyModuleDef``, using either ``PyModule_Create`` or + ``PyModule_FromDefAndSpec``. + + .. versionadded:: 3.5 + +.. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions) + + Add the functions from the *NULL* terminated *functions* array to *module*. + Refer to the :c:type:`PyMethodDef` documentation for details on individual + entries (due to the lack of a shared module namespace, module level + "functions" implemented in C typically receive the module as their first + parameter, making them similar to instance methods on Python classes). + This function is called automatically when creating a module from + ``PyModuleDef``, using either ``PyModule_Create`` or + ``PyModule_FromDefAndSpec``. + + .. versionadded:: 3.5 + +Support functions +................. + +The module initialization function (if using single phase initialization) or +a function called from a module execution slot (if using multi-phase +initialization), can use the following functions to help initialize the module +state: .. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value) @@ -236,7 +410,6 @@ These functions are usually used in the module initialization function. be used from the module's initialization function. This steals a reference to *value*. Return ``-1`` on error, ``0`` on success. - .. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value) Add an integer constant to *module* as *name*. This convenience function can be @@ -248,7 +421,7 @@ These functions are usually used in the module initialization function. Add a string constant to *module* as *name*. This convenience function can be used from the module's initialization function. The string *value* must be - null-terminated. Return ``-1`` on error, ``0`` on success. + *NULL*-terminated. Return ``-1`` on error, ``0`` on success. .. c:function:: int PyModule_AddIntMacro(PyObject *module, macro) @@ -262,3 +435,36 @@ These functions are usually used in the module initialization function. .. c:function:: int PyModule_AddStringMacro(PyObject *module, macro) Add a string constant to *module*. + + +Module lookup +^^^^^^^^^^^^^ + +Single-phase initialization creates singleton modules that can be looked up +in the context of the current interpreter. This allows the module object to be +retrieved later with only a reference to the module definition. + +These functions will not work on modules created using multi-phase initialization, +since multiple such modules can be created from a single definition. + +.. c:function:: PyObject* PyState_FindModule(PyModuleDef *def) + + Returns the module object that was created from *def* for the current interpreter. + This method requires that the module object has been attached to the interpreter state with + :c:func:`PyState_AddModule` beforehand. In case the corresponding module object is not + found or has not been attached to the interpreter state yet, it returns *NULL*. + +.. c:function:: int PyState_AddModule(PyObject *module, PyModuleDef *def) + + Attaches the module object passed to the function to the interpreter state. This allows + the module object to be accessible via :c:func:`PyState_FindModule`. + + Only effective on modules created using single-phase initialization. + + .. versionadded:: 3.3 + +.. c:function:: int PyState_RemoveModule(PyModuleDef *def) + + Removes the module object created from *def* from the interpreter state. + + .. versionadded:: 3.3 diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst index 21951c38c0..9bcb649c9d 100644 --- a/Doc/c-api/number.rst +++ b/Doc/c-api/number.rst @@ -30,6 +30,14 @@ Number Protocol the equivalent of the Python expression ``o1 * o2``. +.. c:function:: PyObject* PyNumber_MatrixMultiply(PyObject *o1, PyObject *o2) + + Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on + failure. This is the equivalent of the Python expression ``o1 @ o2``. + + .. versionadded:: 3.5 + + .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) Return the floor of *o1* divided by *o2*, or *NULL* on failure. This is @@ -146,6 +154,15 @@ Number Protocol the Python statement ``o1 *= o2``. +.. c:function:: PyObject* PyNumber_InPlaceMatrixMultiply(PyObject *o1, PyObject *o2) + + Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on + failure. The operation is done *in-place* when *o1* supports it. This is + the equivalent of the Python statement ``o1 @= o2``. + + .. versionadded:: 3.5 + + .. c:function:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2) Returns the mathematical floor of dividing *o1* by *o2*, or *NULL* on failure. diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index 7cead07081..3d83b279c2 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -47,6 +47,60 @@ Operating System Utilities not call those functions directly! :c:type:`PyOS_sighandler_t` is a typedef alias for :c:type:`void (\*)(int)`. +.. c:function:: wchar_t* Py_DecodeLocale(const char* arg, size_t *size) + + Decode a byte string from the locale encoding with the :ref:`surrogateescape + error handler <surrogateescape>`: undecodable bytes are decoded as + characters in range U+DC80..U+DCFF. If a byte sequence can be decoded as a + surrogate character, escape the bytes using the surrogateescape error + handler instead of decoding them. + + Return a pointer to a newly allocated wide character string, use + :c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write + the number of wide characters excluding the null character into ``*size`` + + Return ``NULL`` on decoding error or memory allocation error. If *size* is + not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to + ``(size_t)-2`` on decoding error. + + Decoding errors should never happen, unless there is a bug in the C + library. + + Use the :c:func:`Py_EncodeLocale` function to encode the character string + back to a byte string. + + .. seealso:: + + The :c:func:`PyUnicode_DecodeFSDefaultAndSize` and + :c:func:`PyUnicode_DecodeLocaleAndSize` functions. + + .. versionadded:: 3.5 + + +.. c:function:: char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos) + + Encode a wide character string to the locale encoding with the + :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters + in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF. + + Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free` + to free the memory. Return ``NULL`` on encoding error or memory allocation + error + + If error_pos is not ``NULL``, ``*error_pos`` is set to the index of the + invalid character on encoding error, or set to ``(size_t)-1`` otherwise. + + Use the :c:func:`Py_DecodeLocale` function to decode the bytes string back + to a wide character string. + + .. seealso:: + + The :c:func:`PyUnicode_EncodeFSDefault` and + :c:func:`PyUnicode_EncodeLocale` functions. + + .. versionadded:: 3.5 + + .. _systemfunctions: System Functions diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 5de8be0680..b5113aaef1 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -220,9 +220,14 @@ type objects) *must* have the :attr:`ob_size` field. the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*. -.. c:member:: void* PyTypeObject.tp_reserved +.. c:member:: PyAsyncMethods* tp_as_async - Reserved slot, formerly known as tp_compare. + Pointer to an additional structure that contains fields relevant only to + objects which implement :term:`awaitable` and :term:`asynchronous iterator` + protocols at the C-level. See :ref:`async-structs` for details. + + .. versionadded:: 3.5 + Formerly known as ``tp_compare`` and ``tp_reserved``. .. c:member:: reprfunc PyTypeObject.tp_repr @@ -1118,6 +1123,9 @@ Number Object Structures binaryfunc nb_inplace_true_divide; unaryfunc nb_index; + + binaryfunc nb_matrix_multiply; + binaryfunc nb_inplace_matrix_multiply; } PyNumberMethods; .. note:: @@ -1329,3 +1337,58 @@ Buffer Object Structures :c:func:`PyBuffer_Release` is the interface for the consumer that wraps this function. + + +.. _async-structs: + + +Async Object Structures +======================= + +.. sectionauthor:: Yury Selivanov <yselivanov@sprymix.com> + +.. versionadded:: 3.5 + +.. c:type:: PyAsyncMethods + + This structure holds pointers to the functions required to implement + :term:`awaitable` and :term:`asynchronous iterator` objects. + + Here is the structure definition:: + + typedef struct { + unaryfunc am_await; + unaryfunc am_aiter; + unaryfunc am_anext; + } PyAsyncMethods; + +.. c:member:: unaryfunc PyAsyncMethods.am_await + + The signature of this function is:: + + PyObject *am_await(PyObject *self) + + The returned object must be an iterator, i.e. :c:func:`PyIter_Check` must + return ``1`` for it. + + This slot may be set to *NULL* if an object is not an :term:`awaitable`. + +.. c:member:: unaryfunc PyAsyncMethods.am_aiter + + The signature of this function is:: + + PyObject *am_aiter(PyObject *self) + + Must return an :term:`awaitable` object. See :meth:`__anext__` for details. + + This slot may be set to *NULL* if an object does not implement + asynchronous iteration protocol. + +.. c:member:: unaryfunc PyAsyncMethods.am_anext + + The signature of this function is:: + + PyObject *am_anext(PyObject *self) + + Must return an :term:`awaitable` object. See :meth:`__anext__` for details. + This slot may be set to *NULL*. diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index f7e99d6d99..2eeadb52c9 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -765,11 +765,13 @@ system. *errors* is ``NULL``. *str* must end with a null character but cannot contain embedded null characters. + Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + .. seealso:: - Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from - :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at - Python startup). + The :c:func:`Py_DecodeLocale` function. .. versionadded:: 3.3 @@ -790,11 +792,13 @@ system. *errors* is ``NULL``. Return a :class:`bytes` object. *str* cannot contain embedded null characters. + Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + .. seealso:: - Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to - :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at - Python startup). + The :c:func:`Py_EncodeLocale` function. .. versionadded:: 3.3 @@ -839,12 +843,14 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to decode a string + from the current locale encoding, use + :c:func:`PyUnicode_DecodeLocaleAndSize`. + .. seealso:: - :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the - locale encoding and cannot be modified later. If you need to decode a - string from the current locale encoding, use - :c:func:`PyUnicode_DecodeLocaleAndSize`. + The :c:func:`Py_DecodeLocale` function. .. versionchanged:: 3.2 Use ``"strict"`` error handler on Windows. @@ -874,12 +880,13 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to encode a string + to the current locale encoding, use :c:func:`PyUnicode_EncodeLocale`. + .. seealso:: - :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the - locale encoding and cannot be modified later. If you need to encode a - string to the current locale encoding, use - :c:func:`PyUnicode_EncodeLocale`. + The :c:func:`Py_EncodeLocale` function. .. versionadded:: 3.2 diff --git a/Doc/conf.py b/Doc/conf.py index f803de238e..28dd80f859 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -36,6 +36,9 @@ highlight_language = 'python3' # Require Sphinx 1.2 for build. needs_sphinx = '1.2' +# Ignore any .rst files in the venv/ directory. +exclude_patterns = ['venv/*'] + # Options for HTML output # ----------------------- diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index 6025617a1b..e388195757 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -349,6 +349,11 @@ PyErr_Format:PyObject*:exception:+1: PyErr_Format:const char*:format:: PyErr_Format::...:: +PyErr_FormatV:PyObject*::null: +PyErr_FormatV:PyObject*:exception:+1: +PyErr_FormatV:const char*:format:: +PyErr_FormatV:va_list:vargs:: + PyErr_WarnEx:int::: PyErr_WarnEx:PyObject*:category:0: PyErr_WarnEx:const char*:message:: @@ -486,6 +491,12 @@ PyFunction_SetDefaults:PyObject*:defaults:+1: PyGen_New:PyObject*::+1: PyGen_New:PyFrameObject*:frame:0: +PyGen_NewWithQualName:PyObject*::+1: +PyGen_NewWithQualName:PyFrameObject*:frame:0: + +PyCoro_New:PyObject*::+1: +PyCoro_New:PyFrameObject*:frame:0: + Py_InitModule:PyObject*::0: Py_InitModule:const char*:name:: Py_InitModule:PyMethodDef[]:methods:: diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst index 6eb87cb2f7..35aa114984 100644 --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -868,23 +868,31 @@ tarballs or zipfiles. Create an archive file (eg. ``zip`` or ``tar``). *base_name* is the name of the file to create, minus any format-specific extension; *format* is the - archive format: one of ``zip``, ``tar``, ``ztar``, or ``gztar``. *root_dir* is - a directory that will be the root directory of the archive; ie. we typically - ``chdir`` into *root_dir* before creating the archive. *base_dir* is the - directory where we start archiving from; ie. *base_dir* will be the common - prefix of all files and directories in the archive. *root_dir* and *base_dir* - both default to the current directory. Returns the name of the archive file. + archive format: one of ``zip``, ``tar``, ``gztar``, ``bztar``, ``xztar``, or + ``ztar``. *root_dir* is a directory that will be the root directory of the + archive; ie. we typically ``chdir`` into *root_dir* before creating the + archive. *base_dir* is the directory where we start archiving from; ie. + *base_dir* will be the common prefix of all files and directories in the + archive. *root_dir* and *base_dir* both default to the current directory. + Returns the name of the archive file. + + .. versionchanged: 3.5 + Added support for the ``xztar`` format. .. function:: make_tarball(base_name, base_dir[, compress='gzip', verbose=0, dry_run=0]) 'Create an (optional compressed) archive as a tar file from all files in and - under *base_dir*. *compress* must be ``'gzip'`` (the default), ``'compress'``, - ``'bzip2'``, or ``None``. Both :program:`tar` and the compression utility named - by *compress* must be on the default program search path, so this is probably - Unix-specific. The output tar file will be named :file:`base_dir.tar`, - possibly plus the appropriate compression extension (:file:`.gz`, :file:`.bz2` - or :file:`.Z`). Return the output filename. + under *base_dir*. *compress* must be ``'gzip'`` (the default), + ``'bzip2'``, ``'xz'``, ``'compress'``, or ``None``. For the ``'compress'`` + method the compression utility named by :program:`compress` must be on the + default program search path, so this is probably Unix-specific. The output + tar file will be named :file:`base_dir.tar`, possibly plus the appropriate + compression extension (``.gz``, ``.bz2``, ``.xz`` or ``.Z``). Return the + output filename. + + .. versionchanged: 3.5 + Added support for the ``xz`` compression. .. function:: make_zipfile(base_name, base_dir[, verbose=0, dry_run=0]) @@ -1193,12 +1201,12 @@ other utility module. .. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None]) - Byte-compile a collection of Python source files to either :file:`.pyc` or - :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`). + Byte-compile a collection of Python source files to :file:`.pyc` files in a + :file:`__pycache__` subdirectory (see :pep:`3147` and :pep:`488`). *py_files* is a list of files to compile; any files that don't end in :file:`.py` are silently skipped. *optimize* must be one of the following: - * ``0`` - don't optimize (generate :file:`.pyc`) + * ``0`` - don't optimize * ``1`` - normal optimization (like ``python -O``) * ``2`` - extra optimization (like ``python -OO``) @@ -1222,10 +1230,13 @@ other utility module. doing, leave it set to ``None``. .. versionchanged:: 3.2.3 - Create ``.pyc`` or ``.pyo`` files with an :func:`import magic tag + Create ``.pyc`` files with an :func:`import magic tag <imp.get_tag>` in their name, in a :file:`__pycache__` subdirectory instead of files without tag in the current directory. + .. versionchanged: 3.5 + Create ``.pyc`` files according to :pep:`488`. + .. function:: rfc822_escape(header) diff --git a/Doc/distutils/builtdist.rst b/Doc/distutils/builtdist.rst index c5827b63cf..523d1e0fff 100644 --- a/Doc/distutils/builtdist.rst +++ b/Doc/distutils/builtdist.rst @@ -72,13 +72,19 @@ The available formats for built distributions are: +-------------+------------------------------+---------+ | Format | Description | Notes | +=============+==============================+=========+ -| ``gztar`` | gzipped tar file | (1),(3) | +| ``gztar`` | gzipped tar file | \(1) | | | (:file:`.tar.gz`) | | +-------------+------------------------------+---------+ +| ``bztar`` | bzipped tar file | | +| | (:file:`.tar.bz2`) | | ++-------------+------------------------------+---------+ +| ``xztar`` | xzipped tar file | | +| | (:file:`.tar.xz`) | | ++-------------+------------------------------+---------+ | ``ztar`` | compressed tar file | \(3) | | | (:file:`.tar.Z`) | | +-------------+------------------------------+---------+ -| ``tar`` | tar file (:file:`.tar`) | \(3) | +| ``tar`` | tar file (:file:`.tar`) | | +-------------+------------------------------+---------+ | ``zip`` | zip file (:file:`.zip`) | (2),(4) | +-------------+------------------------------+---------+ @@ -94,6 +100,9 @@ The available formats for built distributions are: | ``msi`` | Microsoft Installer. | | +-------------+------------------------------+---------+ +.. versionchanged:: 3.5 + Added support for the ``xztar`` format. + Notes: @@ -104,8 +113,7 @@ Notes: default on Windows (3) - requires external utilities: :program:`tar` and possibly one of :program:`gzip`, - :program:`bzip2`, or :program:`compress` + requires external :program:`compress` utility. (4) requires either external :program:`zip` utility or :mod:`zipfile` module (part @@ -119,21 +127,22 @@ You don't have to use the :command:`bdist` command with the :option:`--formats` option; you can also use the command that directly implements the format you're interested in. Some of these :command:`bdist` "sub-commands" actually generate several similar formats; for instance, the :command:`bdist_dumb` command -generates all the "dumb" archive formats (``tar``, ``ztar``, ``gztar``, and -``zip``), and :command:`bdist_rpm` generates both binary and source RPMs. The -:command:`bdist` sub-commands, and the formats generated by each, are: - -+--------------------------+-----------------------+ -| Command | Formats | -+==========================+=======================+ -| :command:`bdist_dumb` | tar, ztar, gztar, zip | -+--------------------------+-----------------------+ -| :command:`bdist_rpm` | rpm, srpm | -+--------------------------+-----------------------+ -| :command:`bdist_wininst` | wininst | -+--------------------------+-----------------------+ -| :command:`bdist_msi` | msi | -+--------------------------+-----------------------+ +generates all the "dumb" archive formats (``tar``, ``gztar``, ``bztar``, +``xztar``, ``ztar``, and ``zip``), and :command:`bdist_rpm` generates both +binary and source RPMs. The :command:`bdist` sub-commands, and the formats +generated by each, are: + ++--------------------------+-------------------------------------+ +| Command | Formats | ++==========================+=====================================+ +| :command:`bdist_dumb` | tar, gztar, bztar, xztar, ztar, zip | ++--------------------------+-------------------------------------+ +| :command:`bdist_rpm` | rpm, srpm | ++--------------------------+-------------------------------------+ +| :command:`bdist_wininst` | wininst | ++--------------------------+-------------------------------------+ +| :command:`bdist_msi` | msi | ++--------------------------+-------------------------------------+ The following sections give details on the individual :command:`bdist_\*` commands. diff --git a/Doc/distutils/introduction.rst b/Doc/distutils/introduction.rst index 0ece646c34..8f46bd74c5 100644 --- a/Doc/distutils/introduction.rst +++ b/Doc/distutils/introduction.rst @@ -156,8 +156,8 @@ module pure Python module a module written in Python and contained in a single :file:`.py` file (and - possibly associated :file:`.pyc` and/or :file:`.pyo` files). Sometimes referred - to as a "pure module." + possibly associated :file:`.pyc` files). Sometimes referred to as a + "pure module." extension module a module written in the low-level language of the Python implementation: C/C++ @@ -210,5 +210,3 @@ distribution root the top-level directory of your source tree (or source distribution); the directory where :file:`setup.py` exists. Generally :file:`setup.py` will be run from this directory. - - diff --git a/Doc/distutils/sourcedist.rst b/Doc/distutils/sourcedist.rst index b9f0cc863b..fb70514b79 100644 --- a/Doc/distutils/sourcedist.rst +++ b/Doc/distutils/sourcedist.rst @@ -32,12 +32,18 @@ to create a gzipped tarball and a zip file. The available formats are: | ``bztar`` | bzip2'ed tar file | | | | (:file:`.tar.bz2`) | | +-----------+-------------------------+---------+ +| ``xztar`` | xz'ed tar file | | +| | (:file:`.tar.xz`) | | ++-----------+-------------------------+---------+ | ``ztar`` | compressed tar file | \(4) | | | (:file:`.tar.Z`) | | +-----------+-------------------------+---------+ | ``tar`` | tar file (:file:`.tar`) | | +-----------+-------------------------+---------+ +.. versionchanged:: 3.5 + Added support for the ``xztar`` format. + Notes: (1) @@ -54,7 +60,7 @@ Notes: requires the :program:`compress` program. Notice that this format is now pending for deprecation and will be removed in the future versions of Python. -When using any ``tar`` format (``gztar``, ``bztar``, ``ztar`` or +When using any ``tar`` format (``gztar``, ``bztar``, ``xztar``, ``ztar`` or ``tar``), under Unix you can specify the ``owner`` and ``group`` names that will be set for each member of the archive. diff --git a/Doc/extending/building.rst b/Doc/extending/building.rst index 656af88131..163179d683 100644 --- a/Doc/extending/building.rst +++ b/Doc/extending/building.rst @@ -1,27 +1,58 @@ .. highlightlang:: c - .. _building: -******************************************** -Building C and C++ Extensions with distutils -******************************************** +***************************** +Building C and C++ Extensions +***************************** -.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +A C extension for CPython is a shared library (e.g. a ``.so`` file on Linux, +``.pyd`` on Windows), which exports an *initialization function*. + +To be importable, the shared library must be available on :envvar:`PYTHONPATH`, +and must be named after the module name, with an appropriate extension. +When using distutils, the correct filename is generated automatically. + +The initialization function has the signature: + +.. c:function:: PyObject* PyInit_modulename(void) + +It returns either a fully-initialized module, or a :c:type:`PyModuleDef` +instance. See :ref:`initializing-modules` for details. + +.. highlightlang:: python +For modules with ASCII-only names, the function must be named +``PyInit_<modulename>``, with ``<modulename>`` replaced by the name of the +module. When using :ref:`multi-phase-initialization`, non-ASCII module names +are allowed. In this case, the initialization function name is +``PyInitU_<modulename>``, with ``<modulename>`` encoded using Python's +*punycode* encoding with hyphens replaced by underscores. In Python:: -Starting in Python 1.4, Python provides, on Unix, a special make file for -building make files for building dynamically-linked extensions and custom -interpreters. Starting with Python 2.0, this mechanism (known as related to -Makefile.pre.in, and Setup files) is no longer supported. Building custom -interpreters was rarely used, and extension modules can be built using -distutils. + def initfunc_name(name): + try: + suffix = b'_' + name.encode('ascii') + except UnicodeEncodeError: + suffix = b'U_' + name.encode('punycode').replace(b'-', b'_') + return b'PyInit' + suffix + +It is possible to export multiple modules from a single shared library by +defining multiple initialization functions. However, importing them requires +using symbolic links or a custom importer, because by default only the +function corresponding to the filename is found. +See :PEP:`489#multiple-modules-in-one-library` for details. + + +.. highlightlang:: c + +Building C and C++ Extensions with distutils +============================================ + +.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> -Building an extension module using distutils requires that distutils is -installed on the build machine, which is included in Python 2.x and available -separately for Python 1.5. Since distutils also supports creation of binary -packages, users don't necessarily need a compiler and distutils to install the -extension. +Extension modules can be built using distutils, which is included in Python. +Since distutils also supports creation of binary packages, users don't +necessarily need a compiler and distutils to install the extension. A distutils package contains a driver script, :file:`setup.py`. This is a plain Python file, which, in the most simple case, could look like this:: diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst index 6cb686ab09..acd60aef8c 100644 --- a/Doc/extending/embedding.rst +++ b/Doc/extending/embedding.rst @@ -58,12 +58,18 @@ perform some operation on a file. :: int main(int argc, char *argv[]) { - Py_SetProgramName(argv[0]); /* optional but recommended */ - Py_Initialize(); - PyRun_SimpleString("from time import time,ctime\n" - "print('Today is', ctime(time()))\n"); - Py_Finalize(); - return 0; + wchar_t *program = Py_DecodeLocale(argv[0], NULL); + if (program == NULL) { + fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); + exit(1); + } + Py_SetProgramName(program); /* optional but recommended */ + Py_Initialize(); + PyRun_SimpleString("from time import time,ctime\n" + "print('Today is', ctime(time()))\n"); + Py_Finalize(); + PyMem_RawFree(program); + return 0; } The :c:func:`Py_SetProgramName` function should be called before @@ -160,7 +166,7 @@ for data conversion between Python and C, and for error reporting. The interesting part with respect to embedding Python starts with :: Py_Initialize(); - pName = PyUnicode_FromString(argv[1]); + pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst index ba6bfa70a2..8cc41840d7 100644 --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -375,11 +375,17 @@ optionally followed by an import of the module:: int main(int argc, char *argv[]) { + wchar_t *program = Py_DecodeLocale(argv[0], NULL); + if (program == NULL) { + fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); + exit(1); + } + /* Add a built-in module, before Py_Initialize */ PyImport_AppendInittab("spam", PyInit_spam); /* Pass argv[0] to the Python interpreter */ - Py_SetProgramName(argv[0]); + Py_SetProgramName(program); /* Initialize the Python interpreter. Required. */ Py_Initialize(); @@ -391,6 +397,10 @@ optionally followed by an import of the module:: ... + PyMem_RawFree(program); + return 0; + } + .. note:: Removing entries from ``sys.modules`` or importing compiled modules into @@ -403,6 +413,13 @@ A more substantial example module is included in the Python source distribution as :file:`Modules/xxmodule.c`. This file may be used as a template or simply read as an example. +.. note:: + + Unlike our ``spam`` example, ``xxmodule`` uses *multi-phase initialization* + (new in Python 3.5), where a PyModuleDef structure is returned from + ``PyInit_spam``, and creation of the module is left to the import machinery. + For details on multi-phase initialization, see :PEP:`489`. + .. _compilation: diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index 0884430044..f60e208e86 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -80,7 +80,7 @@ Moving on, we come to the crunch --- the type object. :: 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ - 0, /* tp_reserved */ + 0, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ diff --git a/Doc/glossary.rst b/Doc/glossary.rst index c8af5f3090..daaeb953af 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -69,11 +69,34 @@ Glossary :ref:`the difference between arguments and parameters <faq-argument-vs-parameter>`, and :pep:`362`. + asynchronous context manager + An object which controls the environment seen in an + :keyword:`async with` statement by defining :meth:`__aenter__` and + :meth:`__aexit__` methods. Introduced by :pep:`492`. + + asynchronous iterable + An object, that can be used in an :keyword:`async for` statement. + Must return an :term:`awaitable` from its :meth:`__aiter__` method, + which should in turn be resolved in an :term:`asynchronous iterator` + object. Introduced by :pep:`492`. + + asynchronous iterator + An object that implements :meth:`__aiter__` and :meth:`__anext__` + methods, that must return :term:`awaitable` objects. + :keyword:`async for` resolves awaitable returned from asynchronous + iterator's :meth:`__anext__` method until it raises + :exc:`StopAsyncIteration` exception. Introduced by :pep:`492`. + attribute A value associated with an object which is referenced by name using dotted expressions. For example, if an object *o* has an attribute *a* it would be referenced as *o.a*. + awaitable + An object that can be used in an :keyword:`await` expression. Can be + a :term:`coroutine` or an object with an :meth:`__await__` method. + See also :pep:`492`. + BDFL Benevolent Dictator For Life, a.k.a. `Guido van Rossum <https://www.python.org/~guido/>`_, Python's creator. @@ -86,12 +109,21 @@ Glossary A :term:`text file` reads and writes :class:`str` objects. bytes-like object - An object that supports the :ref:`bufferobjects`, like :class:`bytes`, - :class:`bytearray` or :class:`memoryview`. Bytes-like objects can - be used for various operations that expect binary data, such as - compression, saving to a binary file or sending over a socket. - Some operations need the binary data to be mutable, in which case - not all bytes-like objects can apply. + An object that supports the :ref:`bufferobjects` and can + export a C-:term:`contiguous` buffer. This includes all :class:`bytes`, + :class:`bytearray`, and :class:`array.array` objects, as well as many + common :class:`memoryview` objects. Bytes-like objects can + be used for various operations that work with binary data; these include + compression, saving to a binary file, and sending over a socket. + + Some operations need the binary data to be mutable. The documentation + often refers to these as "read-write bytes-like objects". Example + mutable buffer objects include :class:`bytearray` and a + :class:`memoryview` of a :class:`bytearray`. + Other operations require the binary data to be stored in + immutable objects ("read-only bytes-like objects"); examples + of these include :class:`bytes` and a :class:`memoryview` + of a :class:`bytes` object. bytecode Python source code is compiled into bytecode, the internal representation @@ -139,6 +171,32 @@ Glossary statement by defining :meth:`__enter__` and :meth:`__exit__` methods. See :pep:`343`. + contiguous + .. index:: C-contiguous, Fortran contiguous + + A buffer is considered contiguous exactly if it is either + *C-contiguous* or *Fortran contiguous*. Zero-dimensional buffers are + C and Fortran contiguous. In one-dimensional arrays, the items + must be layed out in memory next to each other, in order of + increasing indexes starting from zero. In multidimensional + C-contiguous arrays, the last index varies the fastest when + visiting items in order of memory address. However, in + Fortran contiguous arrays, the first index varies the fastest. + + coroutine + Coroutines is a more generalized form of subroutines. Subroutines are + entered at one point and exited at another point. Coroutines can be + entered, exited, and resumed at many different points. They can be + implemented with the :keyword:`async def` statement. See also + :pep:`492`. + + coroutine function + A function which returns a :term:`coroutine` object. A coroutine + function may be defined with the :keyword:`async def` statement, + and may contain :keyword:`await`, :keyword:`async for`, and + :keyword:`async with` keywords. These were introduced + by :pep:`492`. + CPython The canonical implementation of the Python programming language, as distributed on `python.org <https://www.python.org>`_. The term "CPython" @@ -298,14 +356,23 @@ Glossary .. index:: single: generator generator - A function which returns an iterator. It looks like a normal function - except that it contains :keyword:`yield` statements for producing a series - of values usable in a for-loop or that can be retrieved one at a time with - the :func:`next` function. Each :keyword:`yield` temporarily suspends - processing, remembering the location execution state (including local - variables and pending try-statements). When the generator resumes, it - picks-up where it left-off (in contrast to functions which start fresh on - every invocation). + A function which returns a :term:`generator iterator`. It looks like a + normal function except that it contains :keyword:`yield` expressions + for producing a series of values usable in a for-loop or that can be + retrieved one at a time with the :func:`next` function. + + Usually refers to a generator function, but may refer to a + *generator iterator* in some contexts. In cases where the intended + meaning isn't clear, using the full terms avoids ambiguity. + + generator iterator + An object created by a :term:`generator` function. + + Each :keyword:`yield` temporarily suspends processing, remembering the + location execution state (including local variables and pending + try-statements). When the *generator iterator* resumes, it picks-up where + it left-off (in contrast to functions which start fresh on every + invocation). .. index:: single: generator expression @@ -410,6 +477,19 @@ Glossary than compiled ones, though their programs generally also run more slowly. See also :term:`interactive`. + interpreter shutdown + When asked to shut down, the Python interpreter enters a special phase + where it gradually releases all allocated resources, such as modules + and various critical internal structures. It also makes several calls + to the :term:`garbage collector <garbage collection>`. This can trigger + the execution of code in user-defined destructors or weakref callbacks. + Code executed during the shutdown phase can encounter various + exceptions as the resources it relies on may not function anymore + (common examples are library modules or the warnings machinery). + + The main reason for interpreter shutdown is that the ``__main__`` module + or the script being run has finished executing. + iterable An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as :class:`list`, :class:`str`, @@ -452,12 +532,13 @@ Glossary A number of tools in Python accept key functions to control how elements are ordered or grouped. They include :func:`min`, :func:`max`, - :func:`sorted`, :meth:`list.sort`, :func:`heapq.nsmallest`, - :func:`heapq.nlargest`, and :func:`itertools.groupby`. + :func:`sorted`, :meth:`list.sort`, :func:`heapq.merge`, + :func:`heapq.nsmallest`, :func:`heapq.nlargest`, and + :func:`itertools.groupby`. There are several ways to create a key function. For example. the :meth:`str.lower` method can serve as a key function for case insensitive - sorts. Alternatively, an ad-hoc key function can be built from a + sorts. Alternatively, a key function can be built from a :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``. Also, the :mod:`operator` module provides three key function constructors: :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index e362631db6..7524c4ad90 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -758,6 +758,14 @@ All Argument Clinic converters accept the following arguments: In addition, some converters accept additional arguments. Here is a list of these arguments, along with their meanings: + ``accept`` + A set of Python types (and possibly pseudo-types); + this restricts the allowable Python argument to values of these types. + (This is not a general-purpose facility; as a rule it only supports + specific lists of types as shown in the legacy converter table.) + + To accept ``None``, add ``NoneType`` to this set. + ``bitwise`` Only supported for unsigned integers. The native integer value of this Python argument will be written to the parameter without any range checking, @@ -772,39 +780,27 @@ of these arguments, along with their meanings: Only supported for strings. Specifies the encoding to use when converting this string from a Python str (Unicode) value into a C ``char *`` value. - ``length`` - Only supported for strings. If true, requests that the length of the - string be passed in to the impl function, just after the string parameter, - in a parameter named ``<parameter_name>_length``. - - ``nullable`` - Only supported for strings. If true, this parameter may also be set to - ``None``, in which case the C parameter will be set to ``NULL``. ``subclass_of`` Only supported for the ``object`` converter. Requires that the Python value be a subclass of a Python type, as expressed in C. - ``types`` - Only supported for the ``object`` (and ``self``) converter. Specifies + ``type`` + Only supported for the ``object`` and ``self`` converters. Specifies the C type that will be used to declare the variable. Default value is ``"PyObject *"``. - ``types`` - A string containing a list of Python types (and possibly pseudo-types); - this restricts the allowable Python argument to values of these types. - (This is not a general-purpose facility; as a rule it only supports - specific lists of types as shown in the legacy converter table.) - ``zeroes`` Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are - permitted inside the value. + permitted inside the value. The length of the string will be passed in + to the impl function, just after the string parameter, as a parameter named + ``<parameter_name>_length``. Please note, not every possible combination of arguments will work. -Often these arguments are implemented internally by specific ``PyArg_ParseTuple`` +Usually these arguments are implemented by specific ``PyArg_ParseTuple`` *format units*, with specific behavior. For example, currently you cannot -call ``str`` and pass in ``zeroes=True`` without also specifying an ``encoding``; -although it's perfectly reasonable to think this would work, these semantics don't +call ``unsigned_short`` without also specifying ``bitwise=True``. +Although it's perfectly reasonable to think this would work, these semantics don't map to any existing format unit. So Argument Clinic doesn't support it. (Or, at least, not yet.) @@ -816,13 +812,13 @@ on the right is the text you'd replace it with. ``'B'`` ``unsigned_char(bitwise=True)`` ``'b'`` ``unsigned_char`` ``'c'`` ``char`` -``'C'`` ``int(types='str')`` +``'C'`` ``int(accept={str})`` ``'d'`` ``double`` ``'D'`` ``Py_complex`` -``'es#'`` ``str(encoding='name_of_encoding', length=True, zeroes=True)`` ``'es'`` ``str(encoding='name_of_encoding')`` -``'et#'`` ``str(encoding='name_of_encoding', types='bytes bytearray str', length=True)`` -``'et'`` ``str(encoding='name_of_encoding', types='bytes bytearray str')`` +``'es#'`` ``str(encoding='name_of_encoding', zeroes=True)`` +``'et'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str})`` +``'et#'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str}, zeroes=True)`` ``'f'`` ``float`` ``'h'`` ``short`` ``'H'`` ``unsigned_short(bitwise=True)`` @@ -830,29 +826,30 @@ on the right is the text you'd replace it with. ``'I'`` ``unsigned_int(bitwise=True)`` ``'k'`` ``unsigned_long(bitwise=True)`` ``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` +``'l'`` ``long`` ``'L'`` ``PY_LONG_LONG`` ``'n'`` ``Py_ssize_t`` +``'O'`` ``object`` ``'O!'`` ``object(subclass_of='&PySomething_Type')`` ``'O&'`` ``object(converter='name_of_c_function')`` -``'O'`` ``object`` ``'p'`` ``bool`` -``'s#'`` ``str(length=True)`` ``'S'`` ``PyBytesObject`` ``'s'`` ``str`` -``'s*'`` ``Py_buffer(types='str bytes bytearray buffer')`` -``'u#'`` ``Py_UNICODE(length=True)`` -``'u'`` ``Py_UNICODE`` +``'s#'`` ``str(zeroes=True)`` +``'s*'`` ``Py_buffer(accept={buffer, str})`` ``'U'`` ``unicode`` -``'w*'`` ``Py_buffer(types='bytearray rwbuffer')`` -``'y#'`` ``str(types='bytes', length=True)`` +``'u'`` ``Py_UNICODE`` +``'u#'`` ``Py_UNICODE(zeroes=True)`` +``'w*'`` ``Py_buffer(accept={rwbuffer})`` ``'Y'`` ``PyByteArrayObject`` -``'y'`` ``str(types='bytes')`` +``'y'`` ``str(accept={bytes})`` +``'y#'`` ``str(accept={robuffer}, zeroes=True)`` ``'y*'`` ``Py_buffer`` -``'Z#'`` ``Py_UNICODE(nullable=True, length=True)`` -``'z#'`` ``str(nullable=True, length=True)`` -``'Z'`` ``Py_UNICODE(nullable=True)`` -``'z'`` ``str(nullable=True)`` -``'z*'`` ``Py_buffer(types='str bytes bytearray buffer', nullable=True)`` +``'Z'`` ``Py_UNICODE(accept={str, NoneType})`` +``'Z#'`` ``Py_UNICODE(accept={str, NoneType}, zeroes=True)`` +``'z'`` ``str(accept={str, NoneType})`` +``'z#'`` ``str(accept={str, NoneType}, zeroes=True)`` +``'z*'`` ``Py_buffer(accept={buffer, str, NoneType})`` ========= ================================================================================= As an example, here's our sample ``pickle.Pickler.dump`` using the proper diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 1969b32108..945a240666 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -481,10 +481,10 @@ Here's a sample usage of the ``generate_ints()`` generator: You could equally write ``for i in generate_ints(5)``, or ``a,b,c = generate_ints(3)``. -Inside a generator function, ``return value`` is semantically equivalent to -``raise StopIteration(value)``. If no value is returned or the bottom of the -function is reached, the procession of values ends and the generator cannot -return any further values. +Inside a generator function, ``return value`` causes ``StopIteration(value)`` +to be raised from the :meth:`~generator.__next__` method. Once this happens, or +the bottom of the function is reached, the procession of values ends and the +generator cannot yield any further values. You could achieve the effect of generators manually by writing your own class and storing all the local variables of the generator as instance variables. For diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index addaea5243..9c57fe77ed 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -325,6 +325,15 @@ which, when run, will produce:: MainThread: Look out! +.. versionchanged:: 3.5 + Prior to Python 3.5, the :class:`QueueListener` always passed every message + received from the queue to every handler it was initialized with. (This was + because it was assumed that level filtering was all done on the other side, + where the queue is filled.) From 3.5 onwards, this behaviour can be changed + by passing a keyword argument ``respect_handler_level=True`` to the + listener's constructor. When this is done, the listener compares the level + of each message with the handler's level, and only passes a message to a + handler if it's appropriate to do so. .. _network-logging: @@ -1680,7 +1689,7 @@ as in the following complete example:: def main(): logging.basicConfig(level=logging.INFO, format='%(message)s') - logging.info(_('message 1', set_value=set([1, 2, 3]), snowman='\u2603')) + logging.info(_('message 1', set_value={1, 2, 3}, snowman='\u2603')) if __name__ == '__main__': main() diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst index 5e875cdee2..a2aaf367c7 100644 --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -207,13 +207,12 @@ that's ``str``/``bytes`` in Python 2 and ``bytes`` in Python 3). The following table lists the **unique** methods of each data type across Python 2 & 3 (e.g., the ``decode()`` method is usable on the equivalent binary data type in either Python 2 or 3, but it can't be used by the text data type consistently -between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). +between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). Do +note that as of Python 3.5 the ``__mod__`` method was added to the bytes type. ======================== ===================== **Text data** **Binary data** ------------------------ --------------------- -__mod__ (``%`` operator) ------------------------- --------------------- \ decode ------------------------ --------------------- encode @@ -348,10 +347,12 @@ tox with your continuous integration system so that you never accidentally break Python 2 or 3 support. You may also want to use use the ``-bb`` flag with the Python 3 interpreter to -trigger an exception when you are comparing bytes to strings. Usually it's -simply ``False``, but if you made a mistake in your separation of text/binary -data handling you may be accidentally comparing text and binary data. This flag -will raise an exception when that occurs to help track down such cases. +trigger an exception when you are comparing bytes to strings or bytes to an int +(the latter is available starting in Python 3.5). By default type-differing +comparisons simply return ``False``, but if you made a mistake in your +separation of text/binary data handling or indexing on bytes you wouldn't easily +find the mistake. This flag will raise an exception when these kinds of +comparisons occur, making the mistake much easier to track down. And that's mostly it! At this point your code base is compatible with both Python 2 and 3 simultaneously. Your testing will also be set up so that you diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index 9ae04d718d..ad2c6ab7d5 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -1138,7 +1138,7 @@ Empty matches are replaced only when they're not adjacent to a previous match. If *replacement* is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a -carriage return, and so forth. Unknown escapes such as ``\j`` are left alone. +carriage return, and so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by the corresponding group in the RE. This lets you incorporate portions of the original text in the resulting replacement string. diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst index b49ac391ce..ee31a9c981 100644 --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -280,8 +280,9 @@ and optionally an *errors* argument. The *errors* argument specifies the response when the input string can't be converted according to the encoding's rules. Legal values for this argument are ``'strict'`` (raise a :exc:`UnicodeDecodeError` exception), ``'replace'`` (use -``U+FFFD``, ``REPLACEMENT CHARACTER``), or ``'ignore'`` (just leave the -character out of the Unicode result). +``U+FFFD``, ``REPLACEMENT CHARACTER``), ``'ignore'`` (just leave the +character out of the Unicode result), or ``'backslashreplace'`` (inserts a +``\xNN`` escape sequence). The following examples show the differences:: >>> b'\x80abc'.decode("utf-8", "strict") #doctest: +NORMALIZE_WHITESPACE @@ -291,6 +292,8 @@ The following examples show the differences:: invalid start byte >>> b'\x80abc'.decode("utf-8", "replace") '\ufffdabc' + >>> b'\x80abc'.decode("utf-8", "backslashreplace") + '\\x80abc' >>> b'\x80abc'.decode("utf-8", "ignore") 'abc' @@ -325,8 +328,9 @@ The *errors* parameter is the same as the parameter of the :meth:`~bytes.decode` method but supports a few more possible handlers. As well as ``'strict'``, ``'ignore'``, and ``'replace'`` (which in this case inserts a question mark instead of the unencodable character), there is -also ``'xmlcharrefreplace'`` (inserts an XML character reference) and -``backslashreplace`` (inserts a ``\uNNNN`` escape sequence). +also ``'xmlcharrefreplace'`` (inserts an XML character reference), +``backslashreplace`` (inserts a ``\uNNNN`` escape sequence) and +``namereplace`` (inserts a ``\N{...}`` escape sequence). The following example shows the different results:: @@ -346,6 +350,8 @@ The following example shows the different results:: b'ꀀabcd޴' >>> u.encode('ascii', 'backslashreplace') b'\\ua000abcd\\u07b4' + >>> u.encode('ascii', 'namereplace') + b'\\N{YI SYLLABLE IT}abcd\\u07b4' The low-level routines for registering and accessing the available encodings are found in the :mod:`codecs` module. Implementing new diff --git a/Doc/includes/noddy.c b/Doc/includes/noddy.c index 8f79fcf6bf..19a27a89e8 100644 --- a/Doc/includes/noddy.c +++ b/Doc/includes/noddy.c @@ -38,7 +38,7 @@ static PyModuleDef noddymodule = { }; PyMODINIT_FUNC -PyInit_noddy(void) +PyInit_noddy(void) { PyObject* m; diff --git a/Doc/includes/run-func.c b/Doc/includes/run-func.c index 1c9860d7a8..986d670319 100644 --- a/Doc/includes/run-func.c +++ b/Doc/includes/run-func.c @@ -13,7 +13,7 @@ main(int argc, char *argv[]) } Py_Initialize(); - pName = PyUnicode_FromString(argv[1]); + pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); diff --git a/Doc/includes/typestruct.h b/Doc/includes/typestruct.h index 726500980e..9f47899a19 100644 --- a/Doc/includes/typestruct.h +++ b/Doc/includes/typestruct.h @@ -9,7 +9,8 @@ typedef struct _typeobject { printfunc tp_print; getattrfunc tp_getattr; setattrfunc tp_setattr; - void *tp_reserved; /* formerly known as tp_compare */ + PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2) + or tp_reserved (Python 3) */ reprfunc tp_repr; /* Method suites for standard classes */ diff --git a/Doc/install/index.rst b/Doc/install/index.rst index 8f3ad7241a..876f350f90 100644 --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -361,7 +361,7 @@ And here are the values used on Windows: Type of file Installation directory =============== =========================================================== modules :file:`{userbase}\\Python{XY}\\site-packages` -scripts :file:`{userbase}\\Scripts` +scripts :file:`{userbase}\\Python{XY}\\Scripts` data :file:`{userbase}` C headers :file:`{userbase}\\Python{XY}\\Include\\{distname}` =============== =========================================================== diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst index 72f2963a2c..73d8b6b7e8 100644 --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -87,6 +87,9 @@ language using this mechanism: | unicode_literals | 2.6.0a2 | 3.0 | :pep:`3112`: | | | | | *Bytes literals in Python 3000* | +------------------+-------------+--------------+---------------------------------------------+ +| generator_stop | 3.5.0b1 | 3.7 | :pep:`479`: | +| | | | *StopIteration handling inside generators* | ++------------------+-------------+--------------+---------------------------------------------+ .. seealso:: diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 067fb8f748..72095a825f 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -135,7 +135,7 @@ ArgumentParser objects formatter_class=argparse.HelpFormatter, \ prefix_chars='-', fromfile_prefix_chars=None, \ argument_default=None, conflict_handler='error', \ - add_help=True) + add_help=True, allow_abbrev=True) Create a new :class:`ArgumentParser` object. All parameters should be passed as keyword arguments. Each parameter has its own more detailed description @@ -169,6 +169,12 @@ ArgumentParser objects * add_help_ - Add a -h/--help option to the parser (default: ``True``) + * allow_abbrev_ - Allows long options to be abbreviated if the + abbreviation is unambiguous. (default: ``True``) + + .. versionchanged:: 3.5 + *allow_abbrev* parameter was added. + The following sections describe how each of these are used. @@ -518,6 +524,26 @@ calls, we supply ``argument_default=SUPPRESS``:: >>> parser.parse_args([]) Namespace() +.. _allow_abbrev: + +allow_abbrev +^^^^^^^^^^^^ + +Normally, when you pass an argument list to the +:meth:`~ArgumentParser.parse_args` method of a :class:`ArgumentParser`, +it :ref:`recognizes abbreviations <prefix-matching>` of long options. + +This feature can be disabled by setting ``allow_abbrev`` to ``False``:: + + >>> parser = argparse.ArgumentParser(prog='PROG', allow_abbrev=False) + >>> parser.add_argument('--foobar', action='store_true') + >>> parser.add_argument('--foonley', action='store_false') + >>> parser.parse_args(['--foon']) + usage: PROG [-h] [--foobar] [--foonley] + PROG: error: unrecognized arguments: --foon + +.. versionadded:: 3.5 + conflict_handler ^^^^^^^^^^^^^^^^ @@ -1410,9 +1436,9 @@ argument:: Argument abbreviations (prefix matching) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The :meth:`~ArgumentParser.parse_args` method allows long options to be -abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches -a unique option):: +The :meth:`~ArgumentParser.parse_args` method :ref:`by default <allow_abbrev>` +allows long options to be abbreviated to a prefix, if the abbreviation is +unambiguous (the prefix matches a unique option):: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') @@ -1426,6 +1452,7 @@ a unique option):: PROG: error: ambiguous option: -ba could match -badger, -bacon An error is produced for arguments that could produce more than one options. +This feature can be disabled by setting :ref:`allow_abbrev` to ``False``. Beyond ``sys.argv`` diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index c6fa06162e..794da8cced 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -147,40 +147,6 @@ connection requests. by the channel after :meth:`found_terminator` is called. -asynchat - Auxiliary Classes ------------------------------------------- - -.. class:: fifo(list=None) - - A :class:`fifo` holding data which has been pushed by the application but - not yet popped for writing to the channel. A :class:`fifo` is a list used - to hold data and/or producers until they are required. If the *list* - argument is provided then it should contain producers or data items to be - written to the channel. - - - .. method:: is_empty() - - Returns ``True`` if and only if the fifo is empty. - - - .. method:: first() - - Returns the least-recently :meth:`push`\ ed item from the fifo. - - - .. method:: push(data) - - Adds the given data (which may be a string or a producer object) to the - producer fifo. - - - .. method:: pop() - - If the fifo is not empty, returns ``True, first()``, deleting the popped - item. Returns ``False, None`` for an empty fifo. - - .. _asynchat-example: asynchat Example diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst index 1d1f795294..156c5c06ac 100644 --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -106,6 +106,9 @@ directly its :meth:`Future.cancel` method, but:: loop.call_soon_threadsafe(fut.cancel) +To handle signals and to execute subprocesses, the event loop must be run in +the main thread. + To schedule a coroutine object from a different thread, the :func:`run_coroutine_threadsafe` function should be used. It returns a :class:`concurrent.futures.Future` to access the result:: @@ -113,9 +116,6 @@ To schedule a coroutine object from a different thread, the future = asyncio.run_coroutine_threadsafe(coro_func(), loop) result = future.result(timeout) # Wait for the result with a timeout -To handle signals and to execute subprocesses, the event loop must be run in -the main thread. - The :meth:`BaseEventLoop.run_in_executor` method can be used with a thread pool executor to execute a callback in different thread to not block the thread of the event loop. @@ -168,10 +168,10 @@ Detect coroutine objects never scheduled ---------------------------------------- When a coroutine function is called and its result is not passed to -:func:`async` or to the :meth:`BaseEventLoop.create_task` method, the execution -of the coroutine object will never be scheduled which is probably a bug. -:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to :ref:`log a -warning <asyncio-logger>` to detect it. +:func:`ensure_future` or to the :meth:`BaseEventLoop.create_task` method, +the execution of the coroutine object will never be scheduled which is +probably a bug. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` +to :ref:`log a warning <asyncio-logger>` to detect it. Example with the bug:: @@ -190,7 +190,7 @@ Output in debug mode:: File "test.py", line 7, in <module> test() -The fix is to call the :func:`async` function or the +The fix is to call the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method with the coroutine object. .. seealso:: diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index e867c80d4e..279bc29636 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -36,7 +36,7 @@ Run an event loop Run until the :class:`Future` is done. If the argument is a :ref:`coroutine object <coroutine>`, it is wrapped by - :func:`async`. + :func:`ensure_future`. Return the Future's result, or raise its exception. @@ -275,7 +275,9 @@ Creating connections to bind the socket to locally. The *local_host* and *local_port* are looked up using getaddrinfo(), similarly to *host* and *port*. - On Windows with :class:`ProactorEventLoop`, SSL/TLS is not supported. + .. versionchanged:: 3.5 + + On Windows with :class:`ProactorEventLoop`, SSL/TLS is now supported. .. seealso:: @@ -399,14 +401,16 @@ Creating listening connections This method is a :ref:`coroutine <coroutine>`. - On Windows with :class:`ProactorEventLoop`, SSL/TLS is not supported. + .. versionchanged:: 3.5 + + On Windows with :class:`ProactorEventLoop`, SSL/TLS is now supported. .. seealso:: The function :func:`start_server` creates a (:class:`StreamReader`, :class:`StreamWriter`) pair and calls back a function with this pair. - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5.1 The *host* parameter can now be a sequence of strings. diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst index afb8b9f35b..ae3bf90298 100644 --- a/Doc/library/asyncio-eventloops.rst +++ b/Doc/library/asyncio-eventloops.rst @@ -100,8 +100,6 @@ Common limits of Windows event loops: :class:`ProactorEventLoop` specific limits: -- SSL is not supported: :meth:`~BaseEventLoop.create_connection` and - :meth:`~BaseEventLoop.create_server` cannot be used with SSL for example - :meth:`~BaseEventLoop.create_datagram_endpoint` (UDP) is not supported - :meth:`~BaseEventLoop.add_reader` and :meth:`~BaseEventLoop.add_writer` are not supported @@ -112,6 +110,10 @@ The best resolution is 0.5 msec. The resolution depends on the hardware <http://fr.wikipedia.org/wiki/High_Precision_Event_Timer>`_) and on the Windows configuration. See :ref:`asyncio delayed calls <asyncio-delayed-calls>`. +.. versionchanged:: 3.5 + + :class:`ProactorEventLoop` now supports SSL. + Mac OS X ^^^^^^^^ diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst index 656816f75b..18aeafa81f 100644 --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -82,7 +82,7 @@ BaseTransport - ``'subprocess'``: :class:`subprocess.Popen` instance - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5.1 ``'ssl_object'`` info was added to SSL sockets. @@ -453,9 +453,9 @@ buffer size reaches the low-water mark. Coroutines and protocols ------------------------ -Coroutines can be scheduled in a protocol method using :func:`async`, but there -is no guarantee made about the execution order. Protocols are not aware of -coroutines created in protocol methods and so will not wait for them. +Coroutines can be scheduled in a protocol method using :func:`ensure_future`, +but there is no guarantee made about the execution order. Protocols are not +aware of coroutines created in protocol methods and so will not wait for them. To have a reliable execution order, use :ref:`stream objects <asyncio-streams>` in a coroutine with ``yield from``. For example, the :meth:`StreamWriter.drain` diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 33706726a3..f11c09ac29 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -8,7 +8,6 @@ Queues: * :class:`Queue` * :class:`PriorityQueue` * :class:`LifoQueue` -* :class:`JoinableQueue` asyncio queue API was designed to be close to classes of the :mod:`queue` module (:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, @@ -144,16 +143,6 @@ LifoQueue first. -JoinableQueue -^^^^^^^^^^^^^ - -.. class:: JoinableQueue - - Deprecated alias for :class:`Queue`. - - .. deprecated:: 3.4.4 - - Exceptions ^^^^^^^^^^ diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index 41bde257e1..08c5873d8b 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -8,17 +8,23 @@ Tasks and coroutines Coroutines ---------- -A coroutine is a generator that follows certain conventions. For -documentation purposes, all coroutines should be decorated with -``@asyncio.coroutine``, but this cannot be strictly enforced. - -Coroutines use the ``yield from`` syntax introduced in :pep:`380`, +Coroutines used with :mod:`asyncio` may be implemented using the +:keyword:`async def` statement, or by using :term:`generators <generator>`. +The :keyword:`async def` type of coroutine was added in Python 3.5, and +is recommended if there is no need to support older Python versions. + +Generator-based coroutines should be decorated with :func:`@asyncio.coroutine +<asyncio.coroutine>`, although this is not strictly enforced. +The decorator enables compatibility with :keyword:`async def` coroutines, +and also serves as documentation. Generator-based +coroutines use the ``yield from`` syntax introduced in :pep:`380`, instead of the original ``yield`` syntax. The word "coroutine", like the word "generator", is used for two different (though related) concepts: -- The function that defines a coroutine (a function definition +- The function that defines a coroutine + (a function definition using :keyword:`async def` or decorated with ``@asyncio.coroutine``). If disambiguation is needed we will call this a *coroutine function* (:func:`iscoroutinefunction` returns ``True``). @@ -30,29 +36,30 @@ different (though related) concepts: Things a coroutine can do: -- ``result = yield from future`` -- suspends the coroutine until the +- ``result = await future`` or ``result = yield from future`` -- + suspends the coroutine until the future is done, then returns the future's result, or raises an exception, which will be propagated. (If the future is cancelled, it will raise a ``CancelledError`` exception.) Note that tasks are futures, and everything said about futures also applies to tasks. -- ``result = yield from coroutine`` -- wait for another coroutine to +- ``result = await coroutine`` or ``result = yield from coroutine`` -- + wait for another coroutine to produce a result (or raise an exception, which will be propagated). The ``coroutine`` expression must be a *call* to another coroutine. - ``return expression`` -- produce a result to the coroutine that is - waiting for this one using ``yield from``. + waiting for this one using :keyword:`await` or ``yield from``. - ``raise exception`` -- raise an exception in the coroutine that is - waiting for this one using ``yield from``. + waiting for this one using :keyword:`await` or ``yield from``. -Calling a coroutine does not start its code running -- it is just a -generator, and the coroutine object returned by the call is really a -generator object, which doesn't do anything until you iterate over it. -In the case of a coroutine object, there are two basic ways to start -it running: call ``yield from coroutine`` from another coroutine +Calling a coroutine does not start its code running -- +the coroutine object returned by the call doesn't do anything until you +schedule its execution. There are two basic ways to start it running: +call ``await coroutine`` or ``yield from coroutine`` from another coroutine (assuming the other coroutine is already running!), or schedule its execution -using the :func:`async` function or the :meth:`BaseEventLoop.create_task` +using the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method. @@ -60,9 +67,15 @@ Coroutines (and tasks) can only run when the event loop is running. .. decorator:: coroutine - Decorator to mark coroutines. + Decorator to mark generator-based coroutines. This enables + the generator use :keyword:`!yield from` to call :keyword:`async + def` coroutines, and also enables the generator to be called by + :keyword:`async def` coroutines, for instance using an + :keyword:`await` expression. + + There is no need to decorate :keyword:`async def` coroutines themselves. - If the coroutine is not yielded from before it is destroyed, an error + If the generator is not yielded from before it is destroyed, an error message is logged. See :ref:`Detect coroutines never scheduled <asyncio-coroutine-not-scheduled>`. @@ -72,7 +85,7 @@ Coroutines (and tasks) can only run when the event loop is running. even if they are plain Python functions returning a :class:`Future`. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be - used in a callback-style code, wrap its result with :func:`async`. + used in a callback-style code, wrap its result with :func:`ensure_future`. .. _asyncio-hello-world-coroutine: @@ -84,8 +97,7 @@ Example of coroutine displaying ``"Hello World"``:: import asyncio - @asyncio.coroutine - def hello_world(): + async def hello_world(): print("Hello World!") loop = asyncio.get_event_loop() @@ -111,20 +123,30 @@ using the :meth:`sleep` function:: import asyncio import datetime - @asyncio.coroutine - def display_date(loop): + async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break - yield from asyncio.sleep(1) + await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() coroutine is done loop.run_until_complete(display_date(loop)) loop.close() +The same coroutine implemented using a generator:: + + @asyncio.coroutine + def display_date(loop): + end_time = loop.time() + 5.0 + while True: + print(datetime.datetime.now()) + if (loop.time() + 1.0) >= end_time: + break + yield from asyncio.sleep(1) + .. seealso:: The :ref:`display the current date with call_later() @@ -139,15 +161,13 @@ Example chaining coroutines:: import asyncio - @asyncio.coroutine - def compute(x, y): + async def compute(x, y): print("Compute %s + %s ..." % (x, y)) - yield from asyncio.sleep(1.0) + await asyncio.sleep(1.0) return x + y - @asyncio.coroutine - def print_sum(x, y): - result = yield from compute(x, y) + async def print_sum(x, y): + result = await compute(x, y) print("%s + %s = %s" % (x, y, result)) loop = asyncio.get_event_loop() @@ -374,7 +394,7 @@ Task <coroutine>` did not complete. It is probably a bug and a warning is logged: see :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`. - Don't directly create :class:`Task` instances: use the :func:`async` + Don't directly create :class:`Task` instances: use the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method. This class is :ref:`not thread safe <asyncio-multithreading>`. @@ -550,12 +570,14 @@ Task functions .. function:: iscoroutine(obj) - Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`. + Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`, + which may be based on a generator or an :keyword:`async def` coroutine. -.. function:: iscoroutinefunction(obj) +.. function:: iscoroutinefunction(func) - Return ``True`` if *func* is a decorated :ref:`coroutine function - <coroutine>`. + Return ``True`` if *func* is determined to be a :ref:`coroutine function + <coroutine>`, which may be a decorated generator function or an + :keyword:`async def` function. .. coroutinefunction:: sleep(delay, result=None, \*, loop=None) diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 488cda5968..1b8d9cffc6 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -120,6 +120,10 @@ All of the classes in this module may safely be accessed from multiple threads. .. versionchanged:: 3.4 The ``'x'`` (exclusive creation) mode was added. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Incremental (de)compression --------------------------- @@ -162,15 +166,32 @@ Incremental (de)compression you need to decompress a multi-stream input with :class:`BZ2Decompressor`, you must use a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. - Provide data to the decompressor object. Returns a chunk of decompressed - data if possible, or an empty byte string otherwise. + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. - Attempting to decompress data after the end of the current stream is - reached raises an :exc:`EOFError`. If any data is found after the end of - the stream, it is ignored and saved in the :attr:`unused_data` attribute. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: eof @@ -186,6 +207,13 @@ Incremental (de)compression If this attribute is accessed before the end of the stream has been reached, its value will be ``b''``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 + One-shot (de)compression ------------------------ diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index 74abed5c55..7e496ca361 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -157,6 +157,9 @@ return bytes):: if not line: break linecount = linecount + 1 +:class:`FieldStorage` objects also support being used in a :keyword:`with` +statement, which will automatically close them when done. + If an error is encountered when obtaining the contents of an uploaded file (for example, when the user interrupts the form submission by clicking on a Back or Cancel button) the :attr:`~FieldStorage.done` attribute of the @@ -182,6 +185,10 @@ A form submitted via POST that also has a query string will contain both The :attr:`~FieldStorage.file` attribute is automatically closed upon the garbage collection of the creating :class:`FieldStorage` instance. +.. versionchanged:: 3.5 + Added support for the context management protocol to the + :class:`FieldStorage` class. + Higher Level Interface ---------------------- diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst index a981d94a11..ab619a082a 100644 --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -207,6 +207,38 @@ Classification functions and ``False`` otherwise. +.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) + + Return ``True`` if the values *a* and *b* are close to each other and + ``False`` otherwise. + + Whether or not two values are considered close is determined according to + given absolute and relative tolerances. + + *rel_tol* is the relative tolerance -- it is the maximum allowed difference + between *a* and *b*, relative to the larger absolute value of *a* or *b*. + For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default + tolerance is ``1e-09``, which assures that the two values are the same + within about 9 decimal digits. *rel_tol* must be greater than zero. + + *abs_tol* is the minimum absolute tolerance -- useful for comparisons near + zero. *abs_tol* must be at least zero. + + If no errors occur, the result will be: + ``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``. + + The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be + handled according to IEEE rules. Specifically, ``NaN`` is not considered + close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only + considered close to themselves. + + .. versionadded:: 3.5 + + .. seealso:: + + :pep:`485` -- A function for testing approximate equality + + Constants --------- diff --git a/Doc/library/code.rst b/Doc/library/code.rst index 5b5d7cc8c1..275201c69b 100644 --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -4,6 +4,7 @@ .. module:: code :synopsis: Facilities to implement read-eval-print loops. +**Source code:** :source:`Lib/code.py` The ``code`` module provides facilities to implement read-eval-print loops in Python. Two classes and convenience functions are included which can be used to @@ -113,6 +114,9 @@ Interactive Interpreter Objects because it is within the interpreter object implementation. The output is written by the :meth:`write` method. + .. versionchanged:: 3.5 The full chained traceback is displayed instead + of just the primary traceback. + .. method:: InteractiveInterpreter.write(data) @@ -165,4 +169,3 @@ interpreter objects as well as the following additions. newline. When the user enters the EOF key sequence, :exc:`EOFError` is raised. The base implementation reads from ``sys.stdin``; a subclass may replace this with a different implementation. - diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 628969c65d..46d72b5e9b 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -7,6 +7,7 @@ .. sectionauthor:: Marc-André Lemburg <mal@lemburg.com> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/codecs.py` .. index:: single: Unicode @@ -29,10 +30,9 @@ module features are restricted to use specifically with The module defines the following functions for encoding and decoding with any codec: -.. function:: encode(obj, [encoding[, errors]]) +.. function:: encode(obj, encoding='utf-8', errors='strict') - Encodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Encodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that encoding errors raise @@ -40,10 +40,9 @@ any codec: :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. -.. function:: decode(obj, [encoding[, errors]]) +.. function:: decode(obj, encoding='utf-8', errors='strict') - Decodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Decodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that decoding errors raise @@ -104,7 +103,6 @@ The full details for each codec can also be looked up directly: To simplify access to the various codec components, the module provides these additional functions which use :func:`lookup` for the codec lookup: - .. function:: getencoder(encoding) Look up the codec for the given encoding and return its encoder function. @@ -274,6 +272,7 @@ implement the file protocols. Codec authors also need to define how the codec will handle encoding and decoding errors. +.. _surrogateescape: .. _error-handlers: Error Handlers @@ -315,10 +314,14 @@ The following error handlers are only applicable to | | reference (only for encoding). Implemented | | | in :func:`xmlcharrefreplace_errors`. | +-------------------------+-----------------------------------------------+ -| ``'backslashreplace'`` | Replace with backslashed escape sequences | -| | (only for encoding). Implemented in | +| ``'backslashreplace'`` | Replace with backslashed escape sequences. | +| | Implemented in | | | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ +| ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | +| | (only for encoding). Implemented in | +| | :func:`namereplace_errors`. | ++-------------------------+-----------------------------------------------+ | ``'surrogateescape'`` | On decoding, replace byte with individual | | | surrogate code ranging from ``U+DC80`` to | | | ``U+DCFF``. This code will then be turned | @@ -344,6 +347,13 @@ In addition, the following error handler is specific to the given codecs: .. versionchanged:: 3.4 The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs. +.. versionadded:: 3.5 + The ``'namereplace'`` error handler. + +.. versionchanged:: 3.5 + The ``'backslashreplace'`` error handlers now works with decoding and + translating. + The set of allowed values can be extended by registering a new named error handler: @@ -411,9 +421,17 @@ functions: .. function:: backslashreplace_errors(exception) - Implements the ``'backslashreplace'`` error handling (for encoding with + Implements the ``'backslashreplace'`` error handling (for + :term:`text encodings <text encoding>` only): malformed data is + replaced by a backslashed escape sequence. + +.. function:: namereplace_errors(exception) + + Implements the ``'namereplace'`` error handling (for encoding with :term:`text encodings <text encoding>` only): the - unencodable character is replaced by a backslashed escape sequence. + unencodable character is replaced by a ``\N{...}`` escape sequence. + + .. versionadded:: 3.5 .. _codec-objects: @@ -1144,8 +1162,16 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | koi8_r | | Russian | +-----------------+--------------------------------+--------------------------------+ +| koi8_t | | Tajik | +| | | | +| | | .. versionadded:: 3.5 | ++-----------------+--------------------------------+--------------------------------+ | koi8_u | | Ukrainian | +-----------------+--------------------------------+--------------------------------+ +| kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh | +| | | | +| | | .. versionadded:: 3.5 | ++-----------------+--------------------------------+--------------------------------+ | mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, | | | | Macedonian, Russian, Serbian | +-----------------+--------------------------------+--------------------------------+ @@ -1446,4 +1472,3 @@ This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this is only done once (on the first write to the byte stream). For decoding an optional UTF-8 encoded BOM at the start of the data will be skipped. - diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index d73f05ae56..d9b93ad262 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -33,13 +33,14 @@ The collections module offers the following :term:`ABCs <abstract base class>`: .. tabularcolumns:: |l|L|L|L| -========================= ===================== ====================== ==================================================== +========================== ====================== ======================= ==================================================== ABC Inherits from Abstract Methods Mixin Methods -========================= ===================== ====================== ==================================================== +========================== ====================== ======================= ==================================================== :class:`Container` ``__contains__`` :class:`Hashable` ``__hash__`` :class:`Iterable` ``__iter__`` :class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__`` +:class:`Generator` :class:`Iterator` ``send``, ``throw`` ``close``, ``__iter__``, ``__next__`` :class:`Sized` ``__len__`` :class:`Callable` ``__call__`` @@ -80,7 +81,11 @@ ABC Inherits from Abstract Methods Mixin :class:`KeysView` :class:`MappingView`, ``__contains__``, :class:`Set` ``__iter__`` :class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` -========================= ===================== ====================== ==================================================== +:class:`Awaitable` ``__await__`` +:class:`Coroutine` :class:`Awaitable` ``send``, ``throw`` ``close`` +:class:`AsyncIterable` ``__aiter__`` +:class:`AsyncIterator` :class:`AsyncIterable` ``__anext__`` ``__aiter__`` +========================== ====================== ======================= ==================================================== .. class:: Container @@ -102,11 +107,34 @@ ABC Inherits from Abstract Methods Mixin :meth:`~iterator.__next__` methods. See also the definition of :term:`iterator`. +.. class:: Generator + + ABC for generator classes that implement the protocol defined in + :pep:`342` that extends iterators with the :meth:`~generator.send`, + :meth:`~generator.throw` and :meth:`~generator.close` methods. + See also the definition of :term:`generator`. + + .. versionadded:: 3.5 + .. class:: Sequence MutableSequence ABCs for read-only and mutable :term:`sequences <sequence>`. + Implementation note: Some of the mixin methods, such as + :meth:`__iter__`, :meth:`__reversed__` and :meth:`index`, make + repeated calls to the underlying :meth:`__getitem__` method. + Consequently, if :meth:`__getitem__` is implemented with constant + access speed, the mixin methods will have linear performance; + however, if the underlying method is linear (as it would be with a + linked list), the mixins will have quadratic performance and will + likely need to be overridden. + + .. versionchanged:: 3.5 + The index() method added support for *stop* and *start* + arguments. + + .. class:: Set MutableSet @@ -124,6 +152,56 @@ ABC Inherits from Abstract Methods Mixin ABCs for mapping, items, keys, and values :term:`views <dictionary view>`. +.. class:: Awaitable + + ABC for :term:`awaitable` objects, which can be used in :keyword:`await` + expressions. Custom implementations must provide the :meth:`__await__` + method. + + :term:`Coroutine` objects and instances of the + :class:`~collections.abc.Coroutine` ABC are all instances of this ABC. + + .. note:: + In CPython, generator-based coroutines (generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine`) are + *awaitables*, even though they do not have an :meth:`__await__` method. + Using ``isinstance(gencoro, Awaitable)`` for them will return ``False``. + Use :func:`inspect.isawaitable` to detect them. + + .. versionadded:: 3.5 + +.. class:: Coroutine + + ABC for coroutine compatible classes. These implement the + following methods, defined in :ref:`coroutine-objects`: + :meth:`~coroutine.send`, :meth:`~coroutine.throw`, and + :meth:`~coroutine.close`. Custom implementations must also implement + :meth:`__await__`. All :class:`Coroutine` instances are also instances of + :class:`Awaitable`. See also the definition of :term:`coroutine`. + + .. note:: + In CPython, generator-based coroutines (generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine`) are + *awaitables*, even though they do not have an :meth:`__await__` method. + Using ``isinstance(gencoro, Coroutine)`` for them will return ``False``. + Use :func:`inspect.isawaitable` to detect them. + + .. versionadded:: 3.5 + +.. class:: AsyncIterable + + ABC for classes that provide ``__aiter__`` method. See also the + definition of :term:`asynchronous iterable`. + + .. versionadded:: 3.5 + +.. class:: AsyncIterator + + ABC for classes that provide ``__aiter__`` and ``__anext__`` + methods. See also the definition of :term:`asynchronous iterator`. + + .. versionadded:: 3.5 + These ABCs allow us to ask classes or instances if they provide particular functionality, for example:: diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 09718254eb..0e5ff20008 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -387,7 +387,7 @@ or subtracting from an empty counter. Section 4.6.3, Exercise 19*. * To enumerate all distinct multisets of a given size over a given set of - elements, see :func:`itertools.combinations_with_replacement`. + elements, see :func:`itertools.combinations_with_replacement`: map(Counter, combinations_with_replacement('ABC', 2)) --> AA AB AC BB BC CC @@ -437,6 +437,13 @@ or subtracting from an empty counter. Remove all elements from the deque leaving it with length 0. + .. method:: copy() + + Create a shallow copy of the deque. + + .. versionadded:: 3.5 + + .. method:: count(x) Count the number of deque elements equal to *x*. @@ -457,6 +464,22 @@ or subtracting from an empty counter. elements in the iterable argument. + .. method:: index(x[, start[, stop]]) + + Return the position of *x* in the deque (at or after index *start* + and before index *stop*). Returns the first match or raises + :exc:`ValueError` if not found. + + .. versionadded:: 3.5 + + + .. method:: insert(i, x) + + Insert *x* into the deque at position *i*. + + .. versionadded:: 3.5 + + .. method:: pop() Remove and return an element from the right side of the deque. If no @@ -471,7 +494,7 @@ or subtracting from an empty counter. .. method:: remove(value) - Removed the first occurrence of *value*. If not found, raises a + Remove the first occurrence of *value*. If not found, raises a :exc:`ValueError`. @@ -504,6 +527,9 @@ the :keyword:`in` operator, and subscript references such as ``d[-1]``. Indexed access is O(1) at both ends but slows to O(n) in the middle. For fast random access, use lists instead. +Starting in version 3.5, deques support ``__add__()``, ``__mul__()``, +and ``__imul__()``. + Example: .. doctest:: @@ -899,6 +925,15 @@ create a new named tuple type from the :attr:`_fields` attribute: >>> Point3D = namedtuple('Point3D', Point._fields + ('z',)) +Docstrings can be customized by making direct assignments to the ``__doc__`` +fields: + + >>> Book = namedtuple('Book', ['id', 'title', 'authors']) + >>> Book.__doc__ = 'Hardcover book in active collection' + >>> Book.id.__doc__ = '13-digit ISBN' + >>> Book.title.__doc__ = 'Title of first printing' + >>> Book.author.__doc__ = 'List of authors sorted by last name' + Default values can be implemented by using :meth:`_replace` to customize a prototype instance: @@ -982,6 +1017,9 @@ The :class:`OrderedDict` constructor and :meth:`update` method both accept keyword arguments, but their order is lost because Python's function call semantics pass in keyword arguments using a regular unordered dictionary. +.. versionchanged:: 3.5 + The items, keys, and values :term:`views <dictionary view>` + of :class:`OrderedDict` now support reverse iteration using :func:`reversed`. :class:`OrderedDict` Examples and Recipes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -1119,3 +1157,7 @@ attribute. be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a subclass) or an arbitrary sequence which can be converted into a string using the built-in :func:`str` function. + + .. versionchanged:: 3.5 + New methods ``__getnewargs__``, ``__rmod__``, ``casefold``, + ``format_map``, ``isprintable``, and ``maketrans``. diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 9712de2fef..c5736f2043 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -42,7 +42,8 @@ compile Python sources. .. cmdoption:: -q - Do not print the list of files compiled, print only error messages. + Do not print the list of files compiled. If passed once, error messages will + still be printed. If passed twice (``-qq``), all output is suppressed. .. cmdoption:: -d destdir @@ -70,9 +71,28 @@ compile Python sources. is to write files to their :pep:`3147` locations and names, which allows byte-code files from multiple versions of Python to coexist. +.. cmdoption:: -r + + Control the maximum recursion level for subdirectories. + If this is given, then ``-l`` option will not be taken into account. + :program:`python -m compileall <directory> -r 0` is equivalent to + :program:`python -m compileall <directory> -l`. + +.. cmdoption:: -j N + + Use *N* workers to compile the files within the given directory. + If ``0`` is used, then the result of :func:`os.cpu_count()` + will be used. + .. versionchanged:: 3.2 Added the ``-i``, ``-b`` and ``-h`` options. +.. versionchanged:: 3.5 + Added the ``-j``, ``-r``, and ``-qq`` options. ``-q`` option + was changed to a multilevel value. ``-b`` will always produce a + byte-code file ending in ``.pyc``, never ``.pyo``. + + There is no command-line option to control the optimization level used by the :func:`compile` function, because the Python interpreter itself already provides the option: :program:`python -O -m compileall`. @@ -80,7 +100,7 @@ provides the option: :program:`python -O -m compileall`. Public functions ---------------- -.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) +.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` files along the way. @@ -101,8 +121,9 @@ Public functions file considered for compilation, and if it returns a true value, the file is skipped. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -113,11 +134,26 @@ Public functions *optimize* specifies the optimization level for the compiler. It is passed to the built-in :func:`compile` function. + The argument *workers* specifies how many workers are used to + compile files in parallel. The default is to not use multiple workers. + If the platform can't use multiple workers and *workers* argument is given, + then sequential compilation will be used as a fallback. If *workers* is + lower than ``0``, a :exc:`ValueError` will be raised. + .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + Added the *workers* parameter. + + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. -.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) +.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1) Compile the file with path *fullname*. @@ -131,8 +167,9 @@ Public functions file being compiled, and if it returns a true value, the file is not compiled and ``True`` is returned. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -145,8 +182,14 @@ Public functions .. versionadded:: 3.2 + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. -.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False, optimize=-1) + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. + +.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1) Byte-compile all the :file:`.py` files found along ``sys.path``. If *skip_curdir* is true (the default), the current directory is not included @@ -157,6 +200,12 @@ Public functions .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. To force a recompile of all the :file:`.py` files in the :file:`Lib/` subdirectory and all its subdirectories:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index cc8b335b8f..9ac617117f 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -38,7 +38,7 @@ Executor Objects future = executor.submit(pow, 323, 1235) print(future.result()) - .. method:: map(func, *iterables, timeout=None) + .. method:: map(func, *iterables, timeout=None, chunksize=1) Equivalent to :func:`map(func, *iterables) <map>` except *func* is executed asynchronously and several calls to *func* may be made concurrently. The @@ -48,7 +48,16 @@ Executor Objects *timeout* can be an int or a float. If *timeout* is not specified or ``None``, there is no limit to the wait time. If a call raises an exception, then that exception will be raised when its value is - retrieved from the iterator. + retrieved from the iterator. When using :class:`ProcessPoolExecutor`, this + method chops *iterables* into a number of chunks which it submits to the + pool as separate tasks. The (approximate) size of these chunks can be + specified by setting *chunksize* to a positive integer. For very long + iterables, using a large value for *chunksize* can significantly improve + performance compared to the default size of 1. With :class:`ThreadPoolExecutor`, + *chunksize* has no effect. + + .. versionchanged:: 3.5 + Added the *chunksize* argument. .. method:: shutdown(wait=True) @@ -115,11 +124,19 @@ And:: executor.submit(wait_on_future) -.. class:: ThreadPoolExecutor(max_workers) +.. class:: ThreadPoolExecutor(max_workers=None) An :class:`Executor` subclass that uses a pool of at most *max_workers* threads to execute calls asynchronously. + .. versionchanged:: 3.5 + If *max_workers* is ``None`` or + not given, it will default to the number of processors on the machine, + multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often + used to overlap I/O instead of CPU work and the number of workers + should be higher than the number of workers + for :class:`ProcessPoolExecutor`. + .. _threadpoolexecutor-example: @@ -175,6 +192,8 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. An :class:`Executor` subclass that executes calls asynchronously using a pool of at most *max_workers* processes. If *max_workers* is ``None`` or not given, it will default to the number of processors on the machine. + If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError` + will be raised. .. versionchanged:: 3.3 When one of the worker processes terminates abruptly, a diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 92551bc3bd..c9187a3441 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -11,6 +11,8 @@ .. sectionauthor:: Christopher G. Petrilli <petrilli@amber.org> .. sectionauthor:: Łukasz Langa <lukasz@langa.pl> +**Source code:** :source:`Lib/configparser.py` + .. index:: pair: .ini; file pair: configuration; file @@ -142,12 +144,13 @@ datatypes, you should convert on your own: >>> float(topsecret['CompressionLevel']) 9.0 -Extracting Boolean values is not that simple, though. Passing the value -to ``bool()`` would do no good since ``bool('False')`` is still -``True``. This is why config parsers also provide :meth:`getboolean`. -This method is case-insensitive and recognizes Boolean values from -``'yes'``/``'no'``, ``'on'``/``'off'`` and ``'1'``/``'0'`` [1]_. -For example: +Since this task is so common, config parsers provide a range of handy getter +methods to handle integers, floats and booleans. The last one is the most +interesting because simply passing the value to ``bool()`` would do no good +since ``bool('False')`` is still ``True``. This is why config parsers also +provide :meth:`getboolean`. This method is case-insensitive and recognizes +Boolean values from ``'yes'``/``'no'``, ``'on'``/``'off'``, +``'true'``/``'false'`` and ``'1'``/``'0'`` [1]_. For example: .. doctest:: @@ -159,10 +162,8 @@ For example: True Apart from :meth:`getboolean`, config parsers also provide equivalent -:meth:`getint` and :meth:`getfloat` methods, but these are far less -useful since conversion using :func:`int` and :func:`float` is -sufficient for these types. - +:meth:`getint` and :meth:`getfloat` methods. You can register your own +converters and customize the provided ones. [1]_ Fallback Values --------------- @@ -317,11 +318,11 @@ from ``get()`` calls. .. class:: ExtendedInterpolation() An alternative handler for interpolation which implements a more advanced - syntax, used for instance in ``zc.buildout``. Extended interpolation is + syntax, used for instance in ``zc.buildout``. Extended interpolation is using ``${section:option}`` to denote a value from a foreign section. - Interpolation can span multiple levels. For convenience, if the ``section:`` - part is omitted, interpolation defaults to the current section (and possibly - the default values from the special section). + Interpolation can span multiple levels. For convenience, if the + ``section:`` part is omitted, interpolation defaults to the current section + (and possibly the default values from the special section). For example, the configuration specified above with basic interpolation, would look like this with extended interpolation: @@ -399,13 +400,13 @@ However, there are a few differences that should be taken into account: * ``parser.popitem()`` never returns it. * ``parser.get(section, option, **kwargs)`` - the second argument is **not** - a fallback value. Note however that the section-level ``get()`` methods are + a fallback value. Note however that the section-level ``get()`` methods are compatible both with the mapping protocol and the classic configparser API. * ``parser.items()`` is compatible with the mapping protocol (returns a list of *section_name*, *section_proxy* pairs including the DEFAULTSECT). However, this method can also be invoked with arguments: ``parser.items(section, raw, - vars)``. The latter call returns a list of *option*, *value* pairs for + vars)``. The latter call returns a list of *option*, *value* pairs for a specified ``section``, with all interpolations expanded (unless ``raw=True`` is provided). @@ -539,9 +540,9 @@ the :meth:`__init__` options: * *delimiters*, default value: ``('=', ':')`` - Delimiters are substrings that delimit keys from values within a section. The - first occurrence of a delimiting substring on a line is considered a delimiter. - This means values (but not keys) can contain the delimiters. + Delimiters are substrings that delimit keys from values within a section. + The first occurrence of a delimiting substring on a line is considered + a delimiter. This means values (but not keys) can contain the delimiters. See also the *space_around_delimiters* argument to :meth:`ConfigParser.write`. @@ -553,7 +554,7 @@ the :meth:`__init__` options: Comment prefixes are strings that indicate the start of a valid comment within a config file. *comment_prefixes* are used only on otherwise empty lines (optionally indented) whereas *inline_comment_prefixes* can be used after - every valid value (e.g. section names, options and empty lines as well). By + every valid value (e.g. section names, options and empty lines as well). By default inline comments are disabled and ``'#'`` and ``';'`` are used as prefixes for whole line comments. @@ -563,10 +564,10 @@ the :meth:`__init__` options: Please note that config parsers don't support escaping of comment prefixes so using *inline_comment_prefixes* may prevent users from specifying option - values with characters used as comment prefixes. When in doubt, avoid setting - *inline_comment_prefixes*. In any circumstances, the only way of storing - comment prefix characters at the beginning of a line in multiline values is to - interpolate the prefix, for example:: + values with characters used as comment prefixes. When in doubt, avoid + setting *inline_comment_prefixes*. In any circumstances, the only way of + storing comment prefix characters at the beginning of a line in multiline + values is to interpolate the prefix, for example:: >>> from configparser import ConfigParser, ExtendedInterpolation >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) @@ -611,7 +612,7 @@ the :meth:`__init__` options: When set to ``True``, the parser will not allow for any section or option duplicates while reading from a single source (using :meth:`read_file`, - :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict + :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict parsers in new applications. .. versionchanged:: 3.2 @@ -646,12 +647,12 @@ the :meth:`__init__` options: The convention of allowing a special section of default values for other sections or interpolation purposes is a powerful concept of this library, - letting users create complex declarative configurations. This section is + letting users create complex declarative configurations. This section is normally called ``"DEFAULT"`` but this can be customized to point to any - other valid section name. Some typical values include: ``"general"`` or - ``"common"``. The name provided is used for recognizing default sections when - reading from any source and is used when writing configuration back to - a file. Its current value can be retrieved using the + other valid section name. Some typical values include: ``"general"`` or + ``"common"``. The name provided is used for recognizing default sections + when reading from any source and is used when writing configuration back to + a file. Its current value can be retrieved using the ``parser_instance.default_section`` attribute and may be modified at runtime (i.e. to convert files from one format to another). @@ -660,14 +661,30 @@ the :meth:`__init__` options: Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. :class:`RawConfigParser` has a default value of ``None``. +* *converters*, default value: not set + + Config parsers provide option value getters that perform type conversion. By + default :meth:`getint`, :meth:`getfloat`, and :meth:`getboolean` are + implemented. Should other getters be desirable, users may define them in + a subclass or pass a dictionary where each key is a name of the converter and + each value is a callable implementing said conversion. For instance, passing + ``{'decimal': decimal.Decimal}`` would add :meth:`getdecimal` on both the + parser object and all section proxies. In other words, it will be possible + to write both ``parser_instance.getdecimal('section', 'key', fallback=0)`` + and ``parser_instance['section'].getdecimal('key', 0)``. + + If the converter needs to access the state of the parser, it can be + implemented as a method on a config parser subclass. If the name of this + method starts with ``get``, it will be available on all section proxies, in + the dict-compatible form (see the ``getdecimal()`` example above). More advanced customization may be achieved by overriding default values of -these parser attributes. The defaults are defined on the classes, so they -may be overridden by subclasses or by attribute assignment. +these parser attributes. The defaults are defined on the classes, so they may +be overridden by subclasses or by attribute assignment. .. attribute:: BOOLEAN_STATES @@ -725,10 +742,11 @@ may be overridden by subclasses or by attribute assignment. .. attribute:: SECTCRE - A compiled regular expression used to parse section headers. The default - matches ``[section]`` to the name ``"section"``. Whitespace is considered part - of the section name, thus ``[ larch ]`` will be read as a section of name - ``" larch "``. Override this attribute if that's unsuitable. For example: + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered + part of the section name, thus ``[ larch ]`` will be read as a section of + name ``" larch "``. Override this attribute if that's unsuitable. For + example: .. doctest:: @@ -859,7 +877,7 @@ interpolation if an option used is not defined elsewhere. :: ConfigParser Objects -------------------- -.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation()) +.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation(), converters={}) The main configuration parser. When *defaults* is given, it is initialized into the dictionary of intrinsic defaults. When *dict_type* is given, it @@ -869,8 +887,8 @@ ConfigParser Objects When *delimiters* is given, it is used as the set of substrings that divide keys from values. When *comment_prefixes* is given, it will be used as the set of substrings that prefix comments in otherwise empty lines. - Comments can be indented. When *inline_comment_prefixes* is given, it will be - used as the set of substrings that prefix comments in non-empty lines. + Comments can be indented. When *inline_comment_prefixes* is given, it will + be used as the set of substrings that prefix comments in non-empty lines. When *strict* is ``True`` (the default), the parser won't allow for any section or option duplicates while reading from a single source (file, @@ -884,13 +902,13 @@ ConfigParser Objects When *default_section* is given, it specifies the name for the special section holding default values for other sections and interpolation purposes - (normally named ``"DEFAULT"``). This value can be retrieved and changed on + (normally named ``"DEFAULT"``). This value can be retrieved and changed on runtime using the ``default_section`` instance attribute. Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. All option names used in interpolation will be passed through the @@ -899,6 +917,12 @@ ConfigParser Objects converts option names to lower case), the values ``foo %(bar)s`` and ``foo %(BAR)s`` are equivalent. + When *converters* is given, it should be a dictionary where each key + represents the name of a type converter and each value is a callable + implementing the conversion from string to the desired datatype. Every + converter gets its own corresponding :meth:`get*()` method on the parser + object and section proxies. + .. versionchanged:: 3.1 The default *dict_type* is :class:`collections.OrderedDict`. @@ -907,6 +931,9 @@ ConfigParser Objects *empty_lines_in_values*, *default_section* and *interpolation* were added. + .. versionchanged:: 3.5 + The *converters* argument was added. + .. method:: defaults() @@ -944,7 +971,7 @@ ConfigParser Objects .. method:: has_option(section, option) If the given *section* exists, and contains the given *option*, return - :const:`True`; otherwise return :const:`False`. If the specified + :const:`True`; otherwise return :const:`False`. If the specified *section* is :const:`None` or an empty string, DEFAULT is assumed. @@ -1069,7 +1096,7 @@ ConfigParser Objects :meth:`get` method. .. versionchanged:: 3.2 - Items present in *vars* no longer appear in the result. The previous + Items present in *vars* no longer appear in the result. The previous behaviour mixed actual parser options with variables provided for interpolation. @@ -1170,7 +1197,7 @@ RawConfigParser Objects .. note:: Consider using :class:`ConfigParser` instead which checks types of - the values to be stored internally. If you don't want interpolation, you + the values to be stored internally. If you don't want interpolation, you can use ``ConfigParser(interpolation=None)``. @@ -1181,7 +1208,7 @@ RawConfigParser Objects *default section* name is passed, :exc:`ValueError` is raised. Type of *section* is not checked which lets users create non-string named - sections. This behaviour is unsupported and may cause internal errors. + sections. This behaviour is unsupported and may cause internal errors. .. method:: set(section, option, value) @@ -1282,3 +1309,4 @@ Exceptions .. [1] Config parsers allow for heavy customization. If you are interested in changing the behaviour outlined by the footnote reference, consult the `Customizing Parser Behaviour`_ section. + diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index 42b5af23a5..d5a0f09173 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -45,7 +45,6 @@ A small number of constants live in the built-in namespace. They are: for more details. - .. data:: Ellipsis The same as ``...``. Special value used mostly in conjunction with extended diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index 6f36864ac2..550b3476bd 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -172,6 +172,16 @@ Functions and classes provided: .. versionadded:: 3.4 +.. function:: redirect_stderr(new_target) + + Similar to :func:`~contextlib.redirect_stdout` but redirecting + :data:`sys.stderr` to another file or file-like object. + + This context manager is :ref:`reentrant <reentrant-cms>`. + + .. versionadded:: 3.5 + + .. class:: ContextDecorator() A base class that enables a context manager to also be used as a decorator. diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index e9a9cb5ddb..4fcfaef258 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -5,6 +5,7 @@ :synopsis: Write and read tabular data to and from delimited files. .. sectionauthor:: Skip Montanaro <skip@pobox.com> +**Source code:** :source:`Lib/csv.py` .. index:: single: csv @@ -418,7 +419,7 @@ Writer Objects :class:`Writer` objects (:class:`DictWriter` instances and objects returned by the :func:`writer` function) have the following public methods. A *row* must be -a sequence of strings or numbers for :class:`Writer` objects and a dictionary +an iterable of strings or numbers for :class:`Writer` objects and a dictionary mapping fieldnames to strings or numbers (by passing them through :func:`str` first) for :class:`DictWriter` objects. Note that complex numbers are written out surrounded by parens. This may cause some problems for other programs which @@ -430,6 +431,8 @@ read CSV files (assuming they support complex numbers at all). Write the *row* parameter to the writer's file object, formatted according to the current dialect. + .. versionchanged:: 3.5 + Added support of arbitrary iterables. .. method:: csvwriter.writerows(rows) diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index f3e60b4be9..e8dfd833e4 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -599,6 +599,13 @@ The module :mod:`curses` defines the following functions: Only one *ch* can be pushed before :meth:`getch` is called. +.. function:: update_lines_cols() + + Update :envvar:`LINES` and :envvar:`COLS`. Useful for detecting manual screen resize. + + .. versionadded:: 3.5 + + .. function:: unget_wch(ch) Push *ch* so the next :meth:`get_wch` will return it. diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 88f3f6eea7..976cd49000 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters <tim@zope.com> .. sectionauthor:: A.M. Kuchling <amk@amk.ca> +**Source code:** :source:`Lib/datetime.py` + .. XXX what order should the types be discussed in? The :mod:`datetime` module supplies classes for manipulating dates and times in @@ -757,13 +759,19 @@ Other constructors, all class methods: :attr:`tzinfo` ``None``. This may raise :exc:`OverflowError`, if the timestamp is out of the range of values supported by the platform C :c:func:`gmtime` function, and :exc:`OSError` on :c:func:`gmtime` failure. - It's common for this to be restricted to years in 1970 through 2038. See also - :meth:`fromtimestamp`. + It's common for this to be restricted to years in 1970 through 2038. + + To get an aware :class:`.datetime` object, call :meth:`fromtimestamp`:: + + datetime.fromtimestamp(timestamp, timezone.utc) + + On the POSIX compliant platforms, it is equivalent to the following + expression:: - On the POSIX compliant platforms, ``utcfromtimestamp(timestamp)`` - is equivalent to the following expression:: + datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp) - datetime(1970, 1, 1) + timedelta(seconds=timestamp) + except the latter formula always supports the full years range: between + :const:`MINYEAR` and :const:`MAXYEAR` inclusive. .. versionchanged:: 3.3 Raise :exc:`OverflowError` instead of :exc:`ValueError` if the timestamp @@ -1376,10 +1384,13 @@ Supported operations: * efficient pickling -* in Boolean contexts, a :class:`.time` object is considered to be true if and - only if, after converting it to minutes and subtracting :meth:`utcoffset` (or - ``0`` if that's ``None``), the result is non-zero. +In boolean contexts, a :class:`.time` object is always considered to be true. +.. versionchanged:: 3.5 + Before Python 3.5, a :class:`.time` object was considered to be false if it + represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. Instance methods: diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index e6a82d67ce..3f3c43d438 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -325,13 +325,18 @@ The module defines the following: dumbdbm database is created, files with :file:`.dat` and :file:`.dir` extensions are created. - The optional *flag* argument is currently ignored; the database is always opened - for update, and will be created if it does not exist. + The optional *flag* argument supports only the semantics of ``'c'`` + and ``'n'`` values. Other values will default to database being always + opened for update, and will be created if it does not exist. The optional *mode* argument is the Unix mode of the file, used only when the database has to be created. It defaults to octal ``0o666`` (and will be modified by the prevailing umask). + .. versionchanged:: 3.5 + :func:`.open` always creates a new database when the flag has the value + ``'n'``. + In addition to the methods provided by the :class:`collections.abc.MutableMapping` class, :class:`dumbdbm` objects provide the following methods: diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index 759be70c9f..5d7ffffbe9 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -12,6 +12,8 @@ .. moduleauthor:: Stefan Krah <skrah at bytereef.org> .. sectionauthor:: Raymond D. Hettinger <python at rcn.com> +**Source code:** :source:`Lib/decimal.py` + .. import modules for testing inline doctests with the Sphinx doctest builder .. testsetup:: * @@ -742,7 +744,7 @@ Decimal objects * ``"NaN"``, indicating that the operand is a quiet NaN (Not a Number). * ``"sNaN"``, indicating that the operand is a signaling NaN. - .. method:: quantize(exp, rounding=None, context=None, watchexp=True) + .. method:: quantize(exp, rounding=None, context=None) Return a value equal to the first operand after rounding and having the exponent of the second operand. @@ -765,14 +767,8 @@ Decimal objects ``context`` argument; if neither argument is given the rounding mode of the current thread's context is used. - If *watchexp* is set (default), then an error is returned whenever the - resulting exponent is greater than :attr:`Emax` or less than - :attr:`Etiny`. - - .. deprecated:: 3.3 - *watchexp* is an implementation detail from the pure Python version - and is not present in the C version. It will be removed in version - 3.4, where it defaults to ``True``. + An error is returned whenever the resulting exponent is greater than + :attr:`Emax` or less than :attr:`Etiny`. .. method:: radix() @@ -2092,4 +2088,3 @@ Alternatively, inputs can be rounded upon creation using the >>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678') Decimal('1.2345') - diff --git a/Doc/library/development.rst b/Doc/library/development.rst index 06e7048a04..d2b5fa2aa4 100644 --- a/Doc/library/development.rst +++ b/Doc/library/development.rst @@ -16,6 +16,7 @@ The list of modules described in this chapter is: .. toctree:: + typing.rst pydoc.rst doctest.rst unittest.rst diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 5f72ea6233..38245e776a 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters <tim_one@users.sourceforge.net> .. Markup by Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/difflib.py` + .. testsetup:: import sys @@ -25,7 +27,9 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching." The idea is to find the longest contiguous matching subsequence that contains no "junk" - elements (the Ratcliff and Obershelp algorithm doesn't address junk). The same + elements; these "junk" elements are ones that are uninteresting in some + sense, such as blank lines or whitespace. (Handling junk is an + extension to the Ratcliff and Obershelp algorithm.) The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people. @@ -100,7 +104,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The following methods are public: - .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) + .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, \ + numlines=5, *, charset='utf-8') Compares *fromlines* and *tolines* (lists of strings) and returns a string which is a complete HTML file containing a table showing line by line differences with @@ -119,6 +124,10 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. the next difference highlight at the top of the browser without any leading context). + .. versionchanged:: 3.5 + *charset* keyword-only argument was added. The default charset of + HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. + .. method:: make_table(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) Compares *fromlines* and *tolines* (lists of strings) and returns a string which @@ -208,7 +217,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. Compare *a* and *b* (lists of strings); return a :class:`Differ`\ -style delta (a :term:`generator` generating the delta lines). - Optional keyword parameters *linejunk* and *charjunk* are for filter functions + Optional keyword parameters *linejunk* and *charjunk* are filtering functions (or ``None``): *linejunk*: A function that accepts a single string argument, and returns @@ -222,7 +231,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. *charjunk*: A function that accepts a character (a string of length 1), and returns if the character is junk, or false if not. The default is module-level function :func:`IS_CHARACTER_JUNK`, which filters out whitespace characters (a - blank or tab; note: bad idea to include newline in this!). + blank or tab; it's a bad idea to include newline in this!). :file:`Tools/scripts/ndiff.py` is a command-line front-end to this function. @@ -306,6 +315,21 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. +.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n') + + Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a + sequence of delta lines (also bytes) in the format returned by *dfunc*. + *dfunc* must be a callable, typically either :func:`unified_diff` or + :func:`context_diff`. + + Allows you to compare data with unknown or inconsistent encoding. All + inputs except *n* must be bytes objects, not str. Works by losslessly + converting all inputs (except *n*) to str, and calling ``dfunc(a, b, + fromfile, tofile, fromfiledate, tofiledate, n, lineterm)``. The output of + *dfunc* is then converted back to bytes, so the delta lines that you + receive have the same unknown/inconsistent encodings as *a* and *b*. + + .. versionadded:: 3.5 .. function:: IS_LINE_JUNK(line) @@ -622,6 +646,12 @@ The :class:`Differ` class has this constructor: length 1), and returns true if the character is junk. The default is ``None``, meaning that no character is considered junk. + These junk-filtering functions speed up matching to find + differences and do not cause any differing lines or characters to + be ignored. Read the description of the + :meth:`~SequenceMatcher.find_longest_match` method's *isjunk* + parameter for an explanation. + :class:`Differ` objects are used (deltas generated) via a single method: @@ -713,65 +743,4 @@ This example shows how to use difflib to create a ``diff``-like utility. It is also contained in the Python source distribution, as :file:`Tools/scripts/diff.py`. -.. testcode:: - - """ Command line interface to difflib.py providing diffs in four formats: - - * ndiff: lists every line and highlights interline changes. - * context: highlights clusters of changes in a before/after format. - * unified: highlights clusters of changes in an inline format. - * html: generates side by side comparison with change highlights. - - """ - - import sys, os, time, difflib, optparse - - def main(): - # Configure the option parser - usage = "usage: %prog [options] fromfile tofile" - parser = optparse.OptionParser(usage) - parser.add_option("-c", action="store_true", default=False, - help='Produce a context format diff (default)') - parser.add_option("-u", action="store_true", default=False, - help='Produce a unified format diff') - hlp = 'Produce HTML side by side diff (can use -c and -l in conjunction)' - parser.add_option("-m", action="store_true", default=False, help=hlp) - parser.add_option("-n", action="store_true", default=False, - help='Produce a ndiff format diff') - parser.add_option("-l", "--lines", type="int", default=3, - help='Set number of context lines (default 3)') - (options, args) = parser.parse_args() - - if len(args) == 0: - parser.print_help() - sys.exit(1) - if len(args) != 2: - parser.error("need to specify both a fromfile and tofile") - - n = options.lines - fromfile, tofile = args # as specified in the usage string - - # we're passing these as arguments to the diff function - fromdate = time.ctime(os.stat(fromfile).st_mtime) - todate = time.ctime(os.stat(tofile).st_mtime) - with open(fromfile) as fromf, open(tofile) as tof: - fromlines, tolines = list(fromf), list(tof) - - if options.u: - diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, - fromdate, todate, n=n) - elif options.n: - diff = difflib.ndiff(fromlines, tolines) - elif options.m: - diff = difflib.HtmlDiff().make_file(fromlines, tolines, fromfile, - tofile, context=options.c, - numlines=n) - else: - diff = difflib.context_diff(fromlines, tolines, fromfile, tofile, - fromdate, todate, n=n) - - # we're using writelines because diff is a generator - sys.stdout.writelines(diff) - - if __name__ == '__main__': - main() +.. literalinclude:: ../../Tools/scripts/diff.py diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 273fb20634..1bcb3a4a07 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -48,8 +48,9 @@ code. .. class:: Bytecode(x, *, first_line=None, current_offset=None) - Analyse the bytecode corresponding to a function, method, string of source - code, or a code object (as returned by :func:`compile`). + + Analyse the bytecode corresponding to a function, generator, method, string + of source code, or a code object (as returned by :func:`compile`). This is a convenience wrapper around many of the functions listed below, most notably :func:`get_instructions`, as iterating over a :class:`Bytecode` @@ -109,7 +110,7 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: code_info(x) Return a formatted multi-line string with detailed code object information - for the supplied function, method, source code string or code object. + for the supplied function, generator, method, source code string or code object. Note that the exact contents of code info strings are highly implementation dependent and they may change arbitrarily across Python VMs or Python @@ -136,11 +137,11 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: dis(x=None, *, file=None) Disassemble the *x* object. *x* can denote either a module, a class, a - method, a function, a code object, a string of source code or a byte sequence - of raw bytecode. For a module, it disassembles all functions. For a class, - it disassembles all methods. For a code object or sequence of raw bytecode, - it prints one line per bytecode instruction. Strings are first compiled to - code objects with the :func:`compile` built-in function before being + method, a function, a generator, a code object, a string of source code or + a byte sequence of raw bytecode. For a module, it disassembles all functions. + For a class, it disassembles all methods. For a code object or sequence of + raw bytecode, it prints one line per bytecode instruction. Strings are first + compiled to code objects with the :func:`compile` built-in function before being disassembled. If no object is provided, this function disassembles the last traceback. @@ -345,6 +346,14 @@ result back on the stack. Implements ``TOS = iter(TOS)``. +.. opcode:: GET_YIELD_FROM_ITER + + If ``TOS`` is a :term:`generator iterator` or :term:`coroutine` object + it is left as is. Otherwise, implements ``TOS = iter(TOS)``. + + .. versionadded:: 3.5 + + **Binary operations** Binary operations remove the top of the stack (TOS) and the second top-most @@ -361,6 +370,13 @@ result back on the stack. Implements ``TOS = TOS1 * TOS``. +.. opcode:: BINARY_MATRIX_MULTIPLY + + Implements ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: BINARY_FLOOR_DIVIDE Implements ``TOS = TOS1 // TOS``. @@ -433,6 +449,13 @@ the original TOS1. Implements in-place ``TOS = TOS1 * TOS``. +.. opcode:: INPLACE_MATRIX_MULTIPLY + + Implements in-place ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: INPLACE_FLOOR_DIVIDE Implements in-place ``TOS = TOS1 // TOS``. @@ -493,6 +516,40 @@ the original TOS1. Implements ``del TOS1[TOS]``. +**Coroutine opcodes** + +.. opcode:: GET_AWAITABLE + + Implements ``TOS = get_awaitable(TOS)``, where ``get_awaitable(o)`` + returns ``o`` if ``o`` is a coroutine object or a generator object with + the CO_ITERABLE_COROUTINE flag, or resolves + ``o.__await__``. + + +.. opcode:: GET_AITER + + Implements ``TOS = get_awaitable(TOS.__aiter__())``. See ``GET_AWAITABLE`` + for details about ``get_awaitable`` + + +.. opcode:: GET_ANEXT + + Implements ``PUSH(get_awaitable(TOS.__anext__()))``. See ``GET_AWAITABLE`` + for details about ``get_awaitable`` + + +.. opcode:: BEFORE_ASYNC_WITH + + Resolves ``__aenter__`` and ``__aexit__`` from the object on top of the + stack. Pushes ``__aexit__`` and result of ``__aenter__()`` to the stack. + + +.. opcode:: SETUP_ASYNC_WITH + + Creates a new frame object. + + + **Miscellaneous opcodes** .. opcode:: PRINT_EXPR @@ -597,7 +654,7 @@ iterations of the loop. :opcode:`UNPACK_SEQUENCE`). -.. opcode:: WITH_CLEANUP +.. opcode:: WITH_CLEANUP_START Cleans up the stack when a :keyword:`with` statement block exits. TOS is the context manager's :meth:`__exit__` bound method. Below TOS are 1--3 values @@ -609,7 +666,13 @@ iterations of the loop. * (SECOND, THIRD, FOURTH) = exc_info() In the last case, ``TOS(SECOND, THIRD, FOURTH)`` is called, otherwise - ``TOS(None, None, None)``. In addition, TOS is removed from the stack. + ``TOS(None, None, None)``. Pushes SECOND and result of the call + to the stack. + + +.. opcode:: WITH_CLEANUP_FINISH + + Pops exception type and result of 'exit' function call from the stack. If the stack represents an exception, *and* the function call returns a 'true' value, this information is "zapped" and replaced with a single @@ -795,10 +858,6 @@ the more significant byte last. Pushes a try block from a try-except clause onto the block stack. *delta* points to the finally block. -.. opcode:: STORE_MAP - - Store a key and value pair in a dictionary. Pops the key and value while - leaving the dictionary on the stack. .. opcode:: LOAD_FAST (var_num) diff --git a/Doc/library/distribution.rst b/Doc/library/distribution.rst index c4954d1b4a..3e6e84b42a 100644 --- a/Doc/library/distribution.rst +++ b/Doc/library/distribution.rst @@ -12,3 +12,4 @@ with a local index server, or without any index server at all. distutils.rst ensurepip.rst venv.rst + zipapp.rst diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index ea6e1b5481..9f7d12c626 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -1053,15 +1053,9 @@ from text files and modules with doctests: This function uses the same search technique as :func:`testmod`. - .. note:: - Unlike :func:`testmod` and :class:`DocTestFinder`, this function raises - a :exc:`ValueError` if *module* contains no docstrings. You can prevent - this error by passing a :class:`DocTestFinder` instance as the - *test_finder* argument with its *exclude_empty* keyword argument set - to ``False``:: - - >>> finder = doctest.DocTestFinder(exclude_empty=False) - >>> suite = doctest.DocTestSuite(test_finder=finder) + .. versionchanged:: 3.5 + :func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module* + contains no docstrings instead of raising :exc:`ValueError`. Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index aeea94221d..b91f26d120 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -578,6 +578,15 @@ Here are the methods of the :class:`Message` class: will be *failobj*. + .. method:: get_content_disposition() + + Return the lowercased value (without parameters) of the message's + :mailheader:`Content-Disposition` header if it has one, or ``None``. The + possible values for this method are *inline*, *attachment* or ``None`` + if the message follows :rfc:`2183`. + + .. versionadded:: 3.5 + .. method:: walk() The :meth:`walk` method is an all-purpose generator which can be used to diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index 1d70225fc8..67d0a67954 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -195,7 +195,8 @@ Here are the classes: set of the text and is passed as an argument to the :class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults to ``us-ascii`` if the string contains only ``ascii`` code points, and - ``utf-8`` otherwise. + ``utf-8`` otherwise. The *_charset* parameter accepts either a string or a + :class:`~email.charset.Charset` instance. Unless the *_charset* argument is explicitly set to ``None``, the MIMEText object created will have both a :mailheader:`Content-Type` header @@ -206,3 +207,6 @@ Here are the classes: ``Content-Transfer-Encoding`` header, after which a ``set_payload`` call will automatically encode the new payload (and add a new :mailheader:`Content-Transfer-Encoding` header). + + .. versionchanged:: 3.5 + *_charset* also accepts :class:`~email.charset.Charset` instances. diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index d4e3fc186a..045b119303 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -187,6 +187,18 @@ added matters. To illustrate:: :const:`False` (the default), defects will be passed to the :meth:`register_defect` method. + + + .. attribute:: mangle_from\_ + + If :const:`True`, lines starting with *"From "* in the body are + escaped by putting a ``>`` in front of them. This parameter is used when + the message is being serialized by a generator. + Default: :const:`False`. + + .. versionadded:: 3.5 + The *mangle_from_* parameter. + The following :class:`Policy` method is intended to be called by code using the email library to create policy instances with custom settings: @@ -319,6 +331,13 @@ added matters. To illustrate:: :const:`compat32`, that is used as the default policy. Thus the default behavior of the email package is to maintain compatibility with Python 3.2. + The following attributes have values that are different from the + :class:`Policy` default: + + .. attribute:: mangle_from_ + + The default is ``True``. + The class provides the following concrete implementations of the abstract methods of :class:`Policy`: @@ -356,6 +375,14 @@ added matters. To illustrate:: line breaks and any (RFC invalid) binary data it may contain. +An instance of :class:`Compat32` is provided as a module constant: + +.. data:: compat32 + + An instance of :class:`Compat32`, providing backward compatibility with the + behavior of the email package in Python 3.2. + + .. note:: The documentation below describes new policies that are included in the @@ -378,6 +405,14 @@ added matters. To illustrate:: In addition to the settable attributes listed above that apply to all policies, this policy adds the following additional attributes: + .. attribute:: utf8 + + If ``False``, follow :rfc:`5322`, supporting non-ASCII characters in + headers by encoding them as "encoded words". If ``True``, follow + :rfc:`6532` and use ``utf-8`` encoding for headers. Messages + formatted in this way may be passed to SMTP servers that support + the ``SMTPUTF8`` extension (:rfc:`6531`). + .. attribute:: refold_source If the value for a header in the ``Message`` object originated from a @@ -499,6 +534,14 @@ more closely to the RFCs relevant to their domains. Like ``default``, but with ``linesep`` set to ``\r\n``, which is RFC compliant. +.. data:: SMTPUTF8 + + The same as ``SMTP`` except that :attr:`~EmailPolicy.utf8` is ``True``. + Useful for serializing messages to a message store without using encoded + words in the headers. Should only be used for SMTP trasmission if the + sender or recipient addresses have non-ASCII characters (the + :meth:`smtplib.SMTP.send_message` method handles this automatically). + .. data:: HTTP Suitable for serializing headers with for use in HTTP traffic. Like diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index cf095592f6..9b4f9b419f 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -314,11 +314,11 @@ Then:: >>> str(Mood.funky) 'my custom str! 1' -The rules for what is allowed are as follows: _sunder_ names (starting and -ending with a single underscore) are reserved by enum and cannot be used; -all other attributes defined within an enumeration will become members of this -enumeration, with the exception of *__dunder__* names and descriptors (methods -are also descriptors). +The rules for what is allowed are as follows: names that start and end with a +with a single underscore are reserved by enum and cannot be used; all other +attributes defined within an enumeration will become members of this +enumeration, with the exception of special methods (:meth:`__str__`, +:meth:`__add__`, etc.) and descriptors (methods are also descriptors). Note: if your enumeration defines :meth:`__new__` and/or :meth:`__init__` then whatever value(s) were given to the enum member will be passed into those @@ -400,7 +400,8 @@ The second argument is the *source* of enumeration member names. It can be a whitespace-separated string of names, a sequence of names, a sequence of 2-tuples with key/value pairs, or a mapping (e.g. dictionary) of names to values. The last two options enable assigning arbitrary values to -enumerations; the others auto-assign increasing integers starting with 1. A +enumerations; the others auto-assign increasing integers starting with 1 (use +the ``start`` parameter to specify a different starting value). A new class derived from :class:`Enum` is returned. In other words, the above assignment to :class:`Animal` is equivalent to:: @@ -438,12 +439,12 @@ SomeData in the global scope:: The complete signature is:: - Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>) + Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>, start=1) :value: What the new Enum class will record as its name. :names: The Enum members. This can be a whitespace or comma separated string - (values will start at 1):: + (values will start at 1 unless otherwise specified):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -465,6 +466,11 @@ The complete signature is:: :type: type to mix in to new Enum class. +:start: number to start counting at if only names are passed in. + +.. versionchanged:: 3.5 + The *start* parameter was added. + Derived Enumerations -------------------- diff --git a/Doc/library/errno.rst b/Doc/library/errno.rst index d2163b6258..22a5cbc450 100644 --- a/Doc/library/errno.rst +++ b/Doc/library/errno.rst @@ -41,7 +41,10 @@ defined by the module. The specific list of defined symbols is available as .. data:: EINTR - Interrupted system call + Interrupted system call. + + .. seealso:: + This error is mapped to the exception :exc:`InterruptedError`. .. data:: EIO diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 271a5c8219..0a422b238d 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -162,7 +162,8 @@ The following exceptions are the exceptions that are usually raised. .. exception:: GeneratorExit - Raised when a :term:`generator`\'s :meth:`close` method is called. It + Raised when a :term:`generator` or :term:`coroutine` is closed; + see :meth:`generator.close` and :meth:`coroutine.close`. It directly inherits from :exc:`BaseException` instead of :exc:`Exception` since it is technically not an error. @@ -281,6 +282,16 @@ The following exceptions are the exceptions that are usually raised. handling in C, most floating point operations are not checked. +.. exception:: RecursionError + + This exception is derived from :exc:`RuntimeError`. It is raised when the + interpreter detects that the maximum recursion depth (see + :func:`sys.getrecursionlimit`) is exceeded. + + .. versionadded:: 3.5 + Previously, a plain :exc:`RuntimeError` was raised. + + .. exception:: ReferenceError This exception is raised when a weak reference proxy, created by the @@ -306,14 +317,30 @@ The following exceptions are the exceptions that are usually raised. given as an argument when constructing the exception, and defaults to :const:`None`. - When a generator function returns, a new :exc:`StopIteration` instance is + When a :term:`generator` or :term:`coroutine` function + returns, a new :exc:`StopIteration` instance is raised, and the value returned by the function is used as the :attr:`value` parameter to the constructor of the exception. + If a generator function defined in the presence of a ``from __future__ + import generator_stop`` directive raises :exc:`StopIteration`, it will be + converted into a :exc:`RuntimeError` (retaining the :exc:`StopIteration` + as the new exception's cause). + .. versionchanged:: 3.3 Added ``value`` attribute and the ability for generator functions to use it to return a value. + .. versionchanged:: 3.5 + Introduced the RuntimeError transformation. + +.. exception:: StopAsyncIteration + + Must be raised by :meth:`__anext__` method of an + :term:`asynchronous iterator` object to stop the iteration. + + .. versionadded:: 3.5 + .. exception:: SyntaxError Raised when the parser encounters a syntax error. This may occur in an @@ -536,7 +563,12 @@ depending on the system error code. .. exception:: InterruptedError Raised when a system call is interrupted by an incoming signal. - Corresponds to :c:data:`errno` ``EINTR``. + Corresponds to :c:data:`errno` :py:data:`~errno.EINTR`. + + .. versionchanged:: 3.5 + Python now retries system calls when a syscall is interrupted by a + signal, except if the signal handler raises an exception (see :pep:`475` + for the rationale), instead of raising :exc:`InterruptedError`. .. exception:: IsADirectoryError diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst index eb2016a7b9..3a5badd0ff 100644 --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -47,6 +47,9 @@ Dumping the traceback Dump the tracebacks of all threads into *file*. If *all_threads* is ``False``, dump only the current thread. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + Fault handler state ------------------- @@ -59,6 +62,12 @@ Fault handler state produce tracebacks for every running thread. Otherwise, dump only the current thread. + The *file* must be kept open until the fault handler is disabled: see + :ref:`issue with file descriptors <faulthandler-fd>`. + + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: disable() Disable the fault handler: uninstall the signal handlers installed by @@ -82,9 +91,16 @@ Dumping the tracebacks after a timeout call replaces previous parameters and resets the timeout. The timer has a sub-second resolution. + The *file* must be kept open until the traceback is dumped or + :func:`cancel_dump_traceback_later` is called: see :ref:`issue with file + descriptors <faulthandler-fd>`. + This function is implemented using a watchdog thread and therefore is not available if Python is compiled with threads disabled. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: cancel_dump_traceback_later() Cancel the last call to :func:`dump_traceback_later`. @@ -99,8 +115,14 @@ Dumping the traceback on a user signal the traceback of all threads, or of the current thread if *all_threads* is ``False``, into *file*. Call the previous handler if chain is ``True``. + The *file* must be kept open until the signal is unregistered by + :func:`unregister`: see :ref:`issue with file descriptors <faulthandler-fd>`. + Not available on Windows. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: unregister(signum) Unregister a user signal: uninstall the handler of the *signum* signal @@ -110,6 +132,8 @@ Dumping the traceback on a user signal Not available on Windows. +.. _faulthandler-fd: + Issue with file descriptors --------------------------- diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index 8e932fb954..432140f279 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -28,41 +28,41 @@ descriptor. The module defines the following functions: -.. function:: fcntl(fd, op[, arg]) +.. function:: fcntl(fd, cmd, arg=0) - Perform the operation *op* on file descriptor *fd* (file objects providing + Perform the operation *cmd* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). The values used - for *op* are operating system dependent, and are available as constants + for *cmd* are operating system dependent, and are available as constants in the :mod:`fcntl` module, using the same names as used in the relevant C - header files. The argument *arg* is optional, and defaults to the integer - value ``0``. When present, it can either be an integer value, or a string. - With the argument missing or an integer value, the return value of this function - is the integer return value of the C :c:func:`fcntl` call. When the argument is - a string it represents a binary structure, e.g. created by :func:`struct.pack`. - The binary data is copied to a buffer whose address is passed to the C - :c:func:`fcntl` call. The return value after a successful call is the contents - of the buffer, converted to a string object. The length of the returned string - will be the same as the length of the *arg* argument. This is limited to 1024 - bytes. If the information returned in the buffer by the operating system is - larger than 1024 bytes, this is most likely to result in a segmentation - violation or a more subtle data corruption. + header files. The argument *arg* can either be an integer value, or a + :class:`bytes` object. With an integer value, the return value of this + function is the integer return value of the C :c:func:`fcntl` call. When + the argument is bytes it represents a binary structure, e.g. created by + :func:`struct.pack`. The binary data is copied to a buffer whose address is + passed to the C :c:func:`fcntl` call. The return value after a successful + call is the contents of the buffer, converted to a :class:`bytes` object. + The length of the returned object will be the same as the length of the + *arg* argument. This is limited to 1024 bytes. If the information returned + in the buffer by the operating system is larger than 1024 bytes, this is + most likely to result in a segmentation violation or a more subtle data + corruption. If the :c:func:`fcntl` fails, an :exc:`OSError` is raised. -.. function:: ioctl(fd, op[, arg[, mutate_flag]]) +.. function:: ioctl(fd, request, arg=0, mutate_flag=True) This function is identical to the :func:`~fcntl.fcntl` function, except that the argument handling is even more complicated. - The op parameter is limited to values that can fit in 32-bits. - Additional constants of interest for use as the *op* argument can be + The *request* parameter is limited to values that can fit in 32-bits. + Additional constants of interest for use as the *request* argument can be found in the :mod:`termios` module, under the same names as used in the relevant C header files. - The parameter *arg* can be one of an integer, absent (treated identically to the - integer ``0``), an object supporting the read-only buffer interface (most likely - a plain Python string) or an object supporting the read-write buffer interface. + The parameter *arg* can be one of an integer, an object supporting the + read-only buffer interface (like :class:`bytes`) or an object supporting + the read-write buffer interface (like :class:`bytearray`). In all but the last case, behaviour is as for the :func:`~fcntl.fcntl` function. @@ -72,7 +72,7 @@ The module defines the following functions: If it is false, the buffer's mutability is ignored and behaviour is as for a read-only buffer, except that the 1024 byte limit mentioned above is avoided -- - so long as the buffer you pass is as least as long as what the operating system + so long as the buffer you pass is at least as long as what the operating system wants to put there, things should work. If *mutate_flag* is true (the default), then the buffer is (in effect) passed @@ -97,25 +97,25 @@ The module defines the following functions: array('h', [13341]) -.. function:: flock(fd, op) +.. function:: flock(fd, operation) - Perform the lock operation *op* on file descriptor *fd* (file objects providing + Perform the lock operation *operation* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). See the Unix manual :manpage:`flock(2)` for details. (On some systems, this function is emulated using :c:func:`fcntl`.) -.. function:: lockf(fd, operation, [length, [start, [whence]]]) +.. function:: lockf(fd, cmd, len=0, start=0, whence=0) This is essentially a wrapper around the :func:`~fcntl.fcntl` locking calls. - *fd* is the file descriptor of the file to lock or unlock, and *operation* + *fd* is the file descriptor of the file to lock or unlock, and *cmd* is one of the following values: * :const:`LOCK_UN` -- unlock * :const:`LOCK_SH` -- acquire a shared lock * :const:`LOCK_EX` -- acquire an exclusive lock - When *operation* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be + When *cmd* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be bitwise ORed with :const:`LOCK_NB` to avoid blocking on lock acquisition. If :const:`LOCK_NB` is used and the lock cannot be acquired, an :exc:`OSError` will be raised and the exception will have an *errno* @@ -124,7 +124,7 @@ The module defines the following functions: systems, :const:`LOCK_EX` can only be used if the file descriptor refers to a file opened for writing. - *length* is the number of bytes to lock, *start* is the byte offset at + *len* is the number of bytes to lock, *start* is the byte offset at which the lock starts, relative to *whence*, and *whence* is as with :func:`io.IOBase.seek`, specifically: @@ -133,7 +133,7 @@ The module defines the following functions: * :const:`2` -- relative to the end of the file (:data:`os.SEEK_END`) The default for *start* is 0, which means to start at the beginning of the file. - The default for *length* is 0 which means to lock to the end of the file. The + The default for *len* is 0 which means to lock to the end of the file. The default for *whence* is also 0. Examples (all on a SVR4 compliant system):: @@ -147,9 +147,9 @@ Examples (all on a SVR4 compliant system):: rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata) Note that in the first example the return value variable *rv* will hold an -integer value; in the second example it will hold a string value. The structure -lay-out for the *lockdata* variable is system dependent --- therefore using the -:func:`flock` call may be better. +integer value; in the second example it will hold a :class:`bytes` object. The +structure lay-out for the *lockdata* variable is system dependent --- therefore +using the :func:`flock` call may be better. .. seealso:: diff --git a/Doc/library/formatter.rst b/Doc/library/formatter.rst index 1847a8094b..a515f74f64 100644 --- a/Doc/library/formatter.rst +++ b/Doc/library/formatter.rst @@ -5,7 +5,7 @@ :synopsis: Generic output formatter and device interface. :deprecated: -.. deprecated:: 3.4 +.. deprecated-removed:: 3.4 3.6 Due to lack of usage, the formatter module has been deprecated and is slated for removal in Python 3.6. diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index c2c74013f0..e0f0682489 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -172,6 +172,9 @@ another rational number, or from a string. sign as *b* if *b* is nonzero; otherwise it takes the sign of *a*. ``gcd(0, 0)`` returns ``0``. + .. deprecated:: 3.5 + Use :func:`math.gcd` instead. + .. seealso:: diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 5377335a6e..409f6c4fe1 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -156,11 +156,12 @@ are always available. They are listed here in alphabetical order. .. function:: chr(i) - Return the string representing a character whose Unicode code point is the integer - *i*. For example, ``chr(97)`` returns the string ``'a'``. This is the - inverse of :func:`ord`. The valid range for the argument is from 0 through - 1,114,111 (0x10FFFF in base 16). :exc:`ValueError` will be raised if *i* is - outside that range. + Return the string representing a character whose Unicode code point is the + integer *i*. For example, ``chr(97)`` returns the string ``'a'``, while + ``chr(957)`` returns the string ``'ν'``. This is the inverse of :func:`ord`. + + The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in + base 16). :exc:`ValueError` will be raised if *i* is outside that range. .. function:: classmethod(function) @@ -972,9 +973,11 @@ are always available. They are listed here in alphabetical order. Characters not supported by the encoding are replaced with the appropriate XML character reference ``&#nnn;``. - * ``'backslashreplace'`` (also only supported when writing) - replaces unsupported characters with Python's backslashed escape - sequences. + * ``'backslashreplace'`` replaces malformed data by Python's backslashed + escape sequences. + + * ``'namereplace'`` (also only supported when writing) + replaces unsupported characters with ``\N{...}`` escape sequences. .. index:: single: universal newlines; open() built-in function @@ -999,8 +1002,8 @@ are always available. They are listed here in alphabetical order. If *closefd* is ``False`` and a file descriptor rather than a filename was given, the underlying file descriptor will be kept open when the file is - closed. If a filename is given *closefd* has no effect and must be ``True`` - (the default). + closed. If a filename is given *closefd* must be ``True`` (the default) + otherwise an error will be raised. A custom opener can be used by passing a callable as *opener*. The underlying file descriptor for the file object is then obtained by calling *opener* with @@ -1062,14 +1065,18 @@ are always available. They are listed here in alphabetical order. The ``'U'`` mode. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + -.. XXX works for bytes too, but should it? .. function:: ord(c) Given a string representing one Unicode character, return an integer - representing the Unicode code - point of that character. For example, ``ord('a')`` returns the integer ``97`` - and ``ord('\u2020')`` returns ``8224``. This is the inverse of :func:`chr`. + representing the Unicode code point of that character. For example, + ``ord('a')`` returns the integer ``97`` and ``ord('ν')`` returns ``957``. + This is the inverse of :func:`chr`. .. function:: pow(x, y[, z]) @@ -1186,6 +1193,9 @@ are always available. They are listed here in alphabetical order. The returned property object also has the attributes ``fget``, ``fset``, and ``fdel`` corresponding to the constructor arguments. + .. versionchanged:: 3.5 + The docstrings of property objects are now writeable. + .. _func-range: .. function:: range(stop) @@ -1218,8 +1228,8 @@ are always available. They are listed here in alphabetical order. .. function:: round(number[, ndigits]) Return the floating point value *number* rounded to *ndigits* digits after - the decimal point. If *ndigits* is omitted, it defaults to zero. Delegates - to ``number.__round__(ndigits)``. + the decimal point. If *ndigits* is omitted, it returns the nearest integer + to its input. Delegates to ``number.__round__(ndigits)``. For the built-in types supporting :func:`round`, values are rounded to the closest multiple of 10 to the power minus *ndigits*; if two multiples are @@ -1485,7 +1495,9 @@ are always available. They are listed here in alphabetical order. The left-to-right evaluation order of the iterables is guaranteed. This makes possible an idiom for clustering a data series into n-length groups - using ``zip(*[iter(s)]*n)``. + using ``zip(*[iter(s)]*n)``. This repeats the *same* iterator ``n`` times + so that each output tuple has the result of ``n`` calls to the iterator. + This has the effect of dividing the input into n-length chunks. :func:`zip` should only be used with unequal length inputs when you don't care about trailing, unmatched values from the longer iterables. If those diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 813554206a..d11c2e1282 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -186,7 +186,7 @@ values but should not rebind them): added to this list rather than freed. .. versionchanged:: 3.2 - If this list is non-empty at interpreter shutdown, a + If this list is non-empty at :term:`interpreter shutdown`, a :exc:`ResourceWarning` is emitted, which is silent by default. If :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects are printed. @@ -252,8 +252,8 @@ The following constants are provided for use with :func:`set_debug`: to the ``garbage`` list. .. versionchanged:: 3.2 - Also print the contents of the :data:`garbage` list at interpreter - shutdown, if it isn't empty. + Also print the contents of the :data:`garbage` list at + :term:`interpreter shutdown`, if it isn't empty. .. data:: DEBUG_SAVEALL diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index ff23b59156..514cc5a989 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -344,9 +344,9 @@ will assume message ids as Unicode strings, not byte strings. The entire set of key/value pairs are placed into a dictionary and set as the "protected" :attr:`_info` instance variable. -If the :file:`.mo` file's magic number is invalid, or if other problems occur -while reading the file, instantiating a :class:`GNUTranslations` class can raise -:exc:`OSError`. +If the :file:`.mo` file's magic number is invalid, the major version number is +unexpected, or if other problems occur while reading the file, instantiating a +:class:`GNUTranslations` class can raise :exc:`OSError`. The following methods are overridden from the base class implementation: diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index abcbf380d9..50f38a4d36 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -29,7 +29,7 @@ For example, ``'[?]'`` matches the character ``'?'``. The :mod:`pathlib` module offers high-level path objects. -.. function:: glob(pathname) +.. function:: glob(pathname, *, recursive=False) Return a possibly-empty list of path names that match *pathname*, which must be a string containing a path specification. *pathname* can be either absolute @@ -37,8 +37,19 @@ For example, ``'[?]'`` matches the character ``'?'``. :file:`../../Tools/\*/\*.gif`), and can contain shell-style wildcards. Broken symlinks are included in the results (as in the shell). + If *recursive* is true, the pattern "``**``" will match any files and zero or + more directories and subdirectories. If the pattern is followed by a + ``os.sep``, only directories and subdirectories match. -.. function:: iglob(pathname) + .. note:: + Using the "``**``" pattern in large directory trees may consume + an inordinate amount of time. + + .. versionchanged:: 3.5 + Support for recursive globs using "``**``". + + +.. function:: iglob(pathname, recursive=False) Return an :term:`iterator` which yields the same values as :func:`glob` without actually storing them all simultaneously. @@ -55,8 +66,9 @@ For example, ``'[?]'`` matches the character ``'?'``. .. versionadded:: 3.4 -For example, consider a directory containing only the following files: -:file:`1.gif`, :file:`2.txt`, and :file:`card.gif`. :func:`glob` will produce +For example, consider a directory containing the following files: +:file:`1.gif`, :file:`2.txt`, :file:`card.gif` and a subdirectory :file:`sub` +which contains only the file :file:`3.txt`. :func:`glob` will produce the following results. Notice how any leading components of the path are preserved. :: @@ -67,6 +79,10 @@ preserved. :: ['1.gif', 'card.gif'] >>> glob.glob('?.gif') ['1.gif'] + >>> glob.glob('**/*.txt', recursive=True) + ['2.txt', 'sub/3.txt'] + >>> glob.glob('./**/', recursive=True) + ['./', './sub/'] If the directory contains files starting with ``.`` they won't be matched by default. For example, consider a directory containing :file:`card.gif` and diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index 78536fab51..04c41d585c 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -90,13 +90,9 @@ The module defines the following items: is no compression. The default is ``9``. The *mtime* argument is an optional numeric timestamp to be written to - the stream when compressing. All :program:`gzip` compressed streams are - required to contain a timestamp. If omitted or ``None``, the current - time is used. This module ignores the timestamp when decompressing; - however, some programs, such as :program:`gunzip`\ , make use of it. - The format of the timestamp is the same as that of the return value of - ``time.time()`` and of the ``st_mtime`` attribute of the object returned - by ``os.stat()``. + the last modification time field in the stream when compressing. It + should only be provided in compression mode. If omitted or ``None``, the + current time is used. See the :attr:`mtime` attribute for more details. Calling a :class:`GzipFile` object's :meth:`close` method does not close *fileobj*, since you might wish to append more material after the compressed @@ -108,9 +104,9 @@ The module defines the following items: including iteration and the :keyword:`with` statement. Only the :meth:`truncate` method isn't implemented. - :class:`GzipFile` also provides the following method: + :class:`GzipFile` also provides the following method and attribute: - .. method:: peek([n]) + .. method:: peek(n) Read *n* uncompressed bytes without advancing the file position. At most one single read on the compressed stream is done to satisfy @@ -124,9 +120,21 @@ The module defines the following items: .. versionadded:: 3.2 + .. attribute:: mtime + + When decompressing, the value of the last modification time field in + the most recently read header may be read from this attribute, as an + integer. The initial value before reading any headers is ``None``. + + All :program:`gzip` compressed streams are required to contain this + timestamp field. Some programs, such as :program:`gunzip`\ , make use + of the timestamp. The format is the same as the return value of + :func:`time.time` and the :attr:`~os.stat_result.st_mtime` attribute of + the object returned by :func:`os.stat`. + .. versionchanged:: 3.1 Support for the :keyword:`with` statement was added, along with the - *mtime* argument. + *mtime* constructor argument and :attr:`mtime` attribute. .. versionchanged:: 3.2 Support for zero-padded and unseekable files was added. @@ -137,6 +145,12 @@ The module defines the following items: .. versionchanged:: 3.4 Added support for the ``'x'`` and ``'xb'`` modes. + .. versionchanged:: 3.5 + Added support for writing arbitrary + :term:`bytes-like objects <bytes-like object>`. + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + .. function:: compress(data, compresslevel=9) @@ -175,9 +189,10 @@ Example of how to create a compressed GZIP file:: Example of how to GZIP compress an existing file:: import gzip + import shutil with open('/home/joe/file.txt', 'rb') as f_in: with gzip.open('/home/joe/file.txt.gz', 'wb') as f_out: - f_out.writelines(f_in) + shutil.copyfileobj(f_in, f_out) Example of how to GZIP compress a binary string:: diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst index f8970bed73..9fbbcc6904 100644 --- a/Doc/library/heapq.rst +++ b/Doc/library/heapq.rst @@ -82,7 +82,7 @@ The following functions are provided: The module also offers three general purpose functions based on heaps. -.. function:: merge(*iterables) +.. function:: merge(*iterables, key=None, reverse=False) Merge multiple sorted inputs into a single sorted output (for example, merge timestamped entries from multiple log files). Returns an :term:`iterator` @@ -92,6 +92,18 @@ The module also offers three general purpose functions based on heaps. not pull the data into memory all at once, and assumes that each of the input streams is already sorted (smallest to largest). + Has two optional arguments which must be specified as keyword arguments. + + *key* specifies a :term:`key function` of one argument that is used to + extract a comparison key from each input element. The default value is + ``None`` (compare the elements directly). + + *reverse* is a boolean value. If set to ``True``, then the input elements + are merged as if each comparison were reversed. + + .. versionchanged:: 3.5 + Added the optional *key* and *reverse* parameters. + .. function:: nlargest(n, iterable, key=None) diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index fef9c38411..824995eddc 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -16,21 +16,13 @@ This module defines a class :class:`HTMLParser` which serves as the basis for parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. -.. class:: HTMLParser(strict=False, *, convert_charrefs=False) +.. class:: HTMLParser(*, convert_charrefs=True) - Create a parser instance. + Create a parser instance able to parse invalid markup. - If *convert_charrefs* is ``True`` (default: ``False``), all character + If *convert_charrefs* is ``True`` (the default), all character references (except the ones in ``script``/``style`` elements) are automatically converted to the corresponding Unicode characters. - The use of ``convert_charrefs=True`` is encouraged and will become - the default in Python 3.5. - - If *strict* is ``False`` (the default), the parser will accept and parse - invalid markup. If *strict* is ``True`` the parser will raise an - :exc:`~html.parser.HTMLParseError` exception instead [#]_ when it's not - able to parse the markup. The use of ``strict=True`` is discouraged and - the *strict* argument is deprecated. An :class:`.HTMLParser` instance is fed HTML data and calls handler methods when start tags, end tags, text, comments, and other markup elements are @@ -40,31 +32,11 @@ parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. This parser does not check that end tags match start tags or call the end-tag handler for elements which are closed implicitly by closing an outer element. - .. versionchanged:: 3.2 - *strict* argument added. - - .. deprecated-removed:: 3.3 3.5 - The *strict* argument and the strict mode have been deprecated. - The parser is now able to accept and parse invalid markup too. - .. versionchanged:: 3.4 *convert_charrefs* keyword argument added. -An exception is defined as well: - - -.. exception:: HTMLParseError - - Exception raised by the :class:`HTMLParser` class when it encounters an error - while parsing and *strict* is ``True``. This exception provides three - attributes: :attr:`msg` is a brief message explaining the error, - :attr:`lineno` is the number of the line on which the broken construct was - detected, and :attr:`offset` is the number of characters into the line at - which the construct starts. - - .. deprecated-removed:: 3.3 3.5 - This exception has been deprecated because it's never raised by the parser - (when the default non-strict mode is used). + .. versionchanged:: 3.5 + The default value for argument *convert_charrefs* is now ``True``. Example HTML Parser Application @@ -246,8 +218,7 @@ implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): The *data* parameter will be the entire contents of the declaration inside the ``<![...]>`` markup. It is sometimes useful to be overridden by a - derived class. The base class implementation raises an :exc:`HTMLParseError` - when *strict* is ``True``. + derived class. The base class implementation does nothing. .. _htmlparser-examples: @@ -358,9 +329,3 @@ Parsing invalid HTML (e.g. unquoted attributes) also works:: Data : tag soup End tag : p End tag : a - -.. rubric:: Footnotes - -.. [#] For backward compatibility reasons *strict* mode does not raise - exceptions for all non-compliant HTML. That is, some invalid HTML - is tolerated even in *strict* mode. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 807f685248..d57649c4aa 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -180,6 +180,17 @@ The following exceptions are raised as appropriate: is received in the HTTP protocol from the server. +.. exception:: RemoteDisconnected + + A subclass of :exc:`ConnectionResetError` and :exc:`BadStatusLine`. Raised + by :meth:`HTTPConnection.getresponse` when the attempt to read the response + results in no data read from the connection, indicating that the remote end + has closed the connection. + + .. versionadded:: 3.5 + Previously, :exc:`BadStatusLine`\ ``('')`` was raised. + + The constants defined in this module are: .. data:: HTTP_PORT @@ -191,221 +202,15 @@ The constants defined in this module are: The default port for the HTTPS protocol (always ``443``). -and also the following constants for integer status codes: - -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| Constant | Value | Definition | -+==========================================+=========+=======================================================================+ -| :const:`CONTINUE` | ``100`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SWITCHING_PROTOCOLS` | ``101`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROCESSING` | ``102`` | WEBDAV, `RFC 2518, Section 10.1 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_102>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`OK` | ``200`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CREATED` | ``201`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`ACCEPTED` | ``202`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NON_AUTHORITATIVE_INFORMATION` | ``203`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NO_CONTENT` | ``204`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`RESET_CONTENT` | ``205`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PARTIAL_CONTENT` | ``206`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTI_STATUS` | ``207`` | WEBDAV `RFC 2518, Section 10.2 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_207>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`IM_USED` | ``226`` | Delta encoding in HTTP, | -| | | :rfc:`3229`, Section 10.4.1 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTIPLE_CHOICES` | ``300`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MOVED_PERMANENTLY` | ``301`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FOUND` | ``302`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SEE_OTHER` | ``303`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_MODIFIED` | ``304`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`USE_PROXY` | ``305`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TEMPORARY_REDIRECT` | ``307`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_REQUEST` | ``400`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNAUTHORIZED` | ``401`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PAYMENT_REQUIRED` | ``402`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FORBIDDEN` | ``403`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_FOUND` | ``404`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`METHOD_NOT_ALLOWED` | ``405`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_ACCEPTABLE` | ``406`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROXY_AUTHENTICATION_REQUIRED` | ``407`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_TIMEOUT` | ``408`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.9 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.9>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CONFLICT` | ``409`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.10 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GONE` | ``410`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.11 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LENGTH_REQUIRED` | ``411`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.12 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.12>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_FAILED` | ``412`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.13 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.13>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_ENTITY_TOO_LARGE` | ``413`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.14 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.14>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_URI_TOO_LONG` | ``414`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.15 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.15>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNSUPPORTED_MEDIA_TYPE` | ``415`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.16 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.16>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUESTED_RANGE_NOT_SATISFIABLE` | ``416`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.17 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.17>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`EXPECTATION_FAILED` | ``417`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.18 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.18>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNPROCESSABLE_ENTITY` | ``422`` | WEBDAV, `RFC 2518, Section 10.3 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_422>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LOCKED` | ``423`` | WEBDAV `RFC 2518, Section 10.4 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_423>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FAILED_DEPENDENCY` | ``424`` | WEBDAV, `RFC 2518, Section 10.5 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_424>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | -| | | :rfc:`2817`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 3 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 4 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 5 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_IMPLEMENTED` | ``501`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_GATEWAY` | ``502`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SERVICE_UNAVAILABLE` | ``503`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GATEWAY_TIMEOUT` | ``504`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`HTTP_VERSION_NOT_SUPPORTED` | ``505`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INSUFFICIENT_STORAGE` | ``507`` | WEBDAV, `RFC 2518, Section 10.6 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_507>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | -| | | :rfc:`2774`, Section 7 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ - -.. versionchanged:: 3.3 - Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. - - .. data:: responses This dictionary maps the HTTP 1.1 status codes to the W3C names. Example: ``http.client.responses[http.client.NOT_FOUND]`` is ``'Not Found'``. +See :ref:`http-status-codes` for a list of HTTP status codes that are +available in this module as constants. + .. _httpconnection-objects: @@ -458,6 +263,11 @@ HTTPConnection Objects Note that you must have read the whole response before you can send a new request to the server. + .. versionchanged:: 3.5 + If a :exc:`ConnectionError` or subclass is raised, the + :class:`HTTPConnection` object will be ready to reconnect when + a new request is sent. + .. method:: HTTPConnection.set_debuglevel(level) @@ -496,7 +306,9 @@ HTTPConnection Objects .. method:: HTTPConnection.connect() - Connect to the server specified when the object was created. + Connect to the server specified when the object was created. By default, + this is called automatically when making a request if the client does not + already have a connection. .. method:: HTTPConnection.close() diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 646f2e8860..7c85d0954d 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -143,26 +143,43 @@ Morsel Objects The keys are case-insensitive. + .. versionchanged:: 3.5 + :meth:`~Morsel.__eq__` now takes :attr:`~Morsel.key` and :attr:`~Morsel.value` + into account. + .. attribute:: Morsel.value The value of the cookie. + .. deprecated:: 3.5 + assigning to ``value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.coded_value The encoded value of the cookie --- this is what should be sent. + .. deprecated:: 3.5 + assigning to ``coded_value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.key The name of the cookie. + .. deprecated:: 3.5 + assigning to ``key``; use :meth:`~Morsel.set` instead. + .. method:: Morsel.set(key, value, coded_value) Set the *key*, *value* and *coded_value* attributes. + .. deprecated:: 3.5 + The undocumented *LegalChars* parameter is ignored and will be removed in + a future version. + .. method:: Morsel.isReservedKey(K) @@ -193,6 +210,30 @@ Morsel Objects The meaning for *attrs* is the same as in :meth:`output`. +.. method:: Morsel.update(values) + + Update the values in the Morsel dictionary with the values in the dictionary + *values*. Raise an error if any of the keys in the *values* dict is not a + valid :rfc:`2109` attribute. + + .. versionchanged:: 3.5 + an error is raised for invalid keys. + + +.. method:: Morsel.copy(value) + + Return a shallow copy of the Morsel object. + + .. versionchanged:: 3.5 + return a Morsel object instead of a dict. + + +.. method:: Morsel.setdefault(key, value=None) + + Raise an error if key is not a valid :rfc:`2109` attribute, otherwise + behave the same as :meth:`dict.setdefault`. + + .. _cookie-example: Example diff --git a/Doc/library/http.rst b/Doc/library/http.rst index a387a37ddd..b6f2c58218 100644 --- a/Doc/library/http.rst +++ b/Doc/library/http.rst @@ -1,7 +1,16 @@ :mod:`http` --- HTTP modules ============================ -``http`` is a package that collects several modules for working with the +.. module:: http + :synopsis: HTTP status codes and messages + +.. index:: + pair: HTTP; protocol + single: HTTP; http (standard module) + +**Source code:** :source:`Lib/http/__init__.py` + +:mod:`http` is a package that collects several modules for working with the HyperText Transfer Protocol: * :mod:`http.client` is a low-level HTTP protocol client; for high-level URL @@ -9,3 +18,105 @@ HyperText Transfer Protocol: * :mod:`http.server` contains basic HTTP server classes based on :mod:`socketserver` * :mod:`http.cookies` has utilities for implementing state management with cookies * :mod:`http.cookiejar` provides persistence of cookies + +:mod:`http` is also a module that defines a number of HTTP status codes and +associated messages through the :class:`http.HTTPStatus` enum: + +.. class:: HTTPStatus + + .. versionadded:: 3.5 + + A subclass of :class:`enum.IntEnum` that defines a set of HTTP status codes, + reason phrases and long descriptions written in English. + + Usage:: + + >>> from http import HTTPStatus + >>> HTTPStatus.OK + <HTTPStatus.OK: 200> + >>> HTTPStatus.OK == 200 + True + >>> http.HTTPStatus.OK.value + 200 + >>> HTTPStatus.OK.phrase + 'OK' + >>> HTTPStatus.OK.description + 'Request fulfilled, document follows' + >>> list(HTTPStatus) + [<HTTPStatus.CONTINUE: 100>, <HTTPStatus.SWITCHING_PROTOCOLS: 101>, ...] + +.. _http-status-codes: + +HTTP status codes +----------------- + +Supported, +`IANA-registered <http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml>`_ +status codes available in :class:`http.HTTPStatus` are: + +======= =================================== ================================================================== +Code Enum Name Details +======= =================================== ================================================================== +``100`` ``CONTINUE`` HTTP/1.1 :rfc:`7231`, Section 6.2.1 +``101`` ``SWITCHING_PROTOCOLS`` HTTP/1.1 :rfc:`7231`, Section 6.2.2 +``102`` ``PROCESSING`` WebDAV :rfc:`2518`, Section 10.1 +``200`` ``OK`` HTTP/1.1 :rfc:`7231`, Section 6.3.1 +``201`` ``CREATED`` HTTP/1.1 :rfc:`7231`, Section 6.3.2 +``202`` ``ACCEPTED`` HTTP/1.1 :rfc:`7231`, Section 6.3.3 +``203`` ``NON_AUTHORITATIVE_INFORMATION`` HTTP/1.1 :rfc:`7231`, Section 6.3.4 +``204`` ``NO_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.5 +``205`` ``RESET_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.6 +``206`` ``PARTIAL_CONTENT`` HTTP/1.1 :rfc:`7233`, Section 4.1 +``207`` ``MULTI_STATUS`` WebDAV :rfc:`4918`, Section 11.1 +``208`` ``ALREADY_REPORTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.1 (Experimental) +``226`` ``IM_USED`` Delta Encoding in HTTP :rfc:`3229`, Section 10.4.1 +``300`` ``MULTIPLE_CHOICES`` HTTP/1.1 :rfc:`7231`, Section 6.4.1 +``301`` ``MOVED_PERMANENTLY`` HTTP/1.1 :rfc:`7231`, Section 6.4.2 +``302`` ``FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.4.3 +``303`` ``SEE_OTHER`` HTTP/1.1 :rfc:`7231`, Section 6.4.4 +``304`` ``NOT_MODIFIED`` HTTP/1.1 :rfc:`7232`, Section 4.1 +``305`` ``USE_PROXY`` HTTP/1.1 :rfc:`7231`, Section 6.4.5 +``307`` ``TEMPORARY_REDIRECT`` HTTP/1.1 :rfc:`7231`, Section 6.4.7 +``308`` ``PERMANENT_REDIRECT`` Permanent Redirect :rfc:`7238`, Section 3 (Experimental) +``400`` ``BAD_REQUEST`` HTTP/1.1 :rfc:`7231`, Section 6.5.1 +``401`` ``UNAUTHORIZED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.1 +``402`` ``PAYMENT_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.2 +``403`` ``FORBIDDEN`` HTTP/1.1 :rfc:`7231`, Section 6.5.3 +``404`` ``NOT_FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.5.4 +``405`` ``METHOD_NOT_ALLOWED`` HTTP/1.1 :rfc:`7231`, Section 6.5.5 +``406`` ``NOT_ACCEPTABLE`` HTTP/1.1 :rfc:`7231`, Section 6.5.6 +``407`` ``PROXY_AUTHENTICATION_REQUIRED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.2 +``408`` ``REQUEST_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.5.7 +``409`` ``CONFLICT`` HTTP/1.1 :rfc:`7231`, Section 6.5.8 +``410`` ``GONE`` HTTP/1.1 :rfc:`7231`, Section 6.5.9 +``411`` ``LENGTH_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.10 +``412`` ``PRECONDITION_FAILED`` HTTP/1.1 :rfc:`7232`, Section 4.2 +``413`` ``REQUEST_ENTITY_TOO_LARGE`` HTTP/1.1 :rfc:`7231`, Section 6.5.11 +``414`` ``REQUEST_URI_TOO_LONG`` HTTP/1.1 :rfc:`7231`, Section 6.5.12 +``415`` ``UNSUPPORTED_MEDIA_TYPE`` HTTP/1.1 :rfc:`7231`, Section 6.5.13 +``416`` ``REQUEST_RANGE_NOT_SATISFIABLE`` HTTP/1.1 Range Requests :rfc:`7233`, Section 4.4 +``417`` ``EXPECTATION_FAILED`` HTTP/1.1 :rfc:`7231`, Section 6.5.14 +``422`` ``UNPROCESSABLE_ENTITY`` WebDAV :rfc:`4918`, Section 11.2 +``423`` ``LOCKED`` WebDAV :rfc:`4918`, Section 11.3 +``424`` ``FAILED_DEPENDENCY`` WebDAV :rfc:`4918`, Section 11.4 +``426`` ``UPGRADE_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.15 +``428`` ``PRECONDITION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585` +``429`` ``TOO_MANY_REQUESTS`` Additional HTTP Status Codes :rfc:`6585` +``431`` ``REQUEST_HEADER_FIELDS_TOO_LARGE`` Additional HTTP Status Codes :rfc:`6585` +``500`` ``INTERNAL_SERVER_ERROR`` HTTP/1.1 :rfc:`7231`, Section 6.6.1 +``501`` ``NOT_IMPLEMENTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.2 +``502`` ``BAD_GATEWAY`` HTTP/1.1 :rfc:`7231`, Section 6.6.3 +``503`` ``SERVICE_UNAVAILABLE`` HTTP/1.1 :rfc:`7231`, Section 6.6.4 +``504`` ``GATEWAY_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.6.5 +``505`` ``HTTP_VERSION_NOT_SUPPORTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.6 +``506`` ``VARIANT_ALSO_NEGOTIATES`` Transparent Content Negotiation in HTTP :rfc:`2295`, Section 8.1 (Experimental) +``507`` ``INSUFFICIENT_STORAGE`` WebDAV :rfc:`4918`, Section 11.5 +``508`` ``LOOP_DETECTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.2 (Experimental) +``510`` ``NOT_EXTENDED`` An HTTP Extension Framework :rfc:`2774`, Section 7 (Experimental) +``511`` ``NETWORK_AUTHENTICATION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585`, Section 6 +======= =================================== ================================================================== + +In order to preserve backwards compatibility, enum values are also present +in the :mod:`http.client` module in the form of constants. The enum name is +equal to the constant name (i.e. ``http.HTTPStatus.OK`` is also available as +``http.client.OK``). diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index fa736fe3af..15b0932973 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -37,6 +37,19 @@ base class: initialized. If *host* is not specified, ``''`` (the local host) is used. If *port* is omitted, the standard IMAP4 port (143) is used. + The :class:`IMAP4` class supports the :keyword:`with` statement. When used + like this, the IMAP4 ``LOGOUT`` command is issued automatically when the + :keyword:`with` statement exits. E.g.:: + + >>> from imaplib import IMAP4 + >>> with IMAP4("domain.org") as M: + ... M.noop() + ... + ('OK', [b'Nothing Accomplished. d25if65hy903weo.87']) + + .. versionchanged:: 3.5 + Support for the :keyword:`with` statement was added. + Three exceptions are defined as attributes of the :class:`IMAP4` class: @@ -64,7 +77,8 @@ Three exceptions are defined as attributes of the :class:`IMAP4` class: There's also a subclass for secure connections: -.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, certfile=None, ssl_context=None) +.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, \ + certfile=None, ssl_context=None) This is a subclass derived from :class:`IMAP4` that connects over an SSL encrypted socket (to use this class you need a socket module that was compiled @@ -198,6 +212,10 @@ An :class:`IMAP4` instance has the following methods: that will be base64 encoded and sent to the server. It should return ``None`` if the client abort response ``*`` should be sent instead. + .. versionchanged:: 3.5 + string usernames and passwords are now encoded to ``utf-8`` instead of + being limited to ASCII. + .. method:: IMAP4.check() @@ -230,6 +248,16 @@ An :class:`IMAP4` instance has the following methods: Delete the ACLs (remove any rights) set for who on mailbox. +.. method:: IMAP4.enable(capability) + + Enable *capability* (see :rfc:`5161`). Most capabilities do not need to be + enabled. Currently only the ``UTF8=ACCEPT`` capability is supported + (see :RFC:`6855`). + + .. versionadded:: 3.5 + The :meth:`enable` method itself, and :RFC:`6855` support. + + .. method:: IMAP4.expunge() Permanently remove deleted items from selected mailbox. Generates an ``EXPUNGE`` @@ -367,7 +395,9 @@ An :class:`IMAP4` instance has the following methods: Search mailbox for matching messages. *charset* may be ``None``, in which case no ``CHARSET`` will be specified in the request to the server. The IMAP protocol requires that at least one criterion be specified; an exception will be - raised when the server returns an error. + raised when the server returns an error. *charset* must be ``None`` if + the ``UTF8=ACCEPT`` capability was enabled using the :meth:`enable` + command. Example:: @@ -529,6 +559,15 @@ The following attributes are defined on instances of :class:`IMAP4`: the module variable ``Debug``. Values greater than three trace each command. +.. attribute:: IMAP4.utf8_enabled + + Boolean value that is normally ``False``, but is set to ``True`` if an + :meth:`enable` command is successfully issued for the ``UTF8=ACCEPT`` + capability. + + .. versionadded:: 3.5 + + .. _imap4-example: IMAP4 Example diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index 9e8952339c..f11f6dcf8e 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -48,6 +48,14 @@ from :func:`what`: +------------+-----------------------------------+ | ``'png'`` | Portable Network Graphics | +------------+-----------------------------------+ +| ``'webp'`` | WebP files | ++------------+-----------------------------------+ +| ``'exr'`` | OpenEXR Files | ++------------+-----------------------------------+ + +.. versionadded:: 3.5 + The *exr* and *webp* formats were added. + You can extend the list of file types :mod:`imghdr` can recognize by appending to this variable: diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 83a52e4e14..68a6b681ef 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -197,11 +197,9 @@ file paths. value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. + :exc:`NotImplementedError` will be raised). By passing in ``True`` or + ``False`` for *debug_override* you can override the system's value for + ``__debug__``, leading to optimized bytecode. *path* need not exist. @@ -212,6 +210,9 @@ file paths. .. deprecated:: 3.4 Use :func:`importlib.util.cache_from_source` instead. + .. versionchanged:: 3.5 + The *debug_override* parameter no longer creates a ``.pyo`` file. + .. function:: source_from_cache(path) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index c84d2dfbb4..632df75d57 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -55,6 +55,12 @@ generically as an :term:`importer`) to participate in the import process. :pep:`451` A ModuleSpec Type for the Import System + :pep:`488` + Elimination of PYO files + + :pep:`489` + Multi-phase extension module initialization + :pep:`3120` Using UTF-8 as the Default Source Encoding @@ -69,6 +75,10 @@ Functions An implementation of the built-in :func:`__import__` function. + .. note:: + Programmatic importing of modules should use :func:`import_module` + instead of this function. + .. function:: import_module(name, package=None) Import a module. The *name* argument specifies what module to @@ -81,12 +91,15 @@ Functions The :func:`import_module` function acts as a simplifying wrapper around :func:`importlib.__import__`. This means all semantics of the function are - derived from :func:`importlib.__import__`, including requiring the package - from which an import is occurring to have been previously imported - (i.e., *package* must already be imported). The most important difference - is that :func:`import_module` returns the specified package or module - (e.g. ``pkg.mod``), while :func:`__import__` returns the - top-level package or module (e.g. ``pkg``). + derived from :func:`importlib.__import__`. The most important difference + between these two functions is that :func:`import_module` returns the + specified package or module (e.g. ``pkg.mod``), while :func:`__import__` + returns the top-level package or module (e.g. ``pkg``). + + If you are dynamically importing a module that was created since the + interpreter began execution (e.g., created a Python source file), you may + need to call :func:`invalidate_caches` in order for the new module to be + noticed by the import system. .. versionchanged:: 3.3 Parent packages are automatically imported. @@ -341,13 +354,16 @@ ABC hierarchy:: .. method:: create_module(spec) - An optional method that returns the module object to use when - importing a module. create_module() may also return ``None``, - indicating that the default module creation should take place - instead. + A method that returns the module object to use when + importing a module. This method may return ``None``, + indicating that default module creation semantics should take place. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Starting in Python 3.6, this method will not be optional when + :meth:`exec_module` is defined. + .. method:: exec_module(module) An abstract method that executes the module in its own namespace @@ -411,7 +427,7 @@ ABC hierarchy:: .. deprecated:: 3.4 The recommended API for loading a module is :meth:`exec_module` - (and optionally :meth:`create_module`). Loaders should implement + (and :meth:`create_module`). Loaders should implement it instead of load_module(). The import machinery takes care of all the other responsibilities of load_module() when exec_module() is implemented. @@ -493,7 +509,7 @@ ABC hierarchy:: .. versionchanged:: 3.4 Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. - .. method:: source_to_code(data, path='<string>') + .. staticmethod:: source_to_code(data, path='<string>') Create a code object from Python source. @@ -502,8 +518,14 @@ ABC hierarchy:: the "path" to where the source code originated from, which can be an abstract concept (e.g. location in a zip file). + With the subsequent code object one can execute it in a module by + running ``exec(code, module.__dict__)``. + .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Made the method static. + .. method:: exec_module(module) Implementation of :meth:`Loader.exec_module`. @@ -689,6 +711,9 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: OPTIMIZED_BYTECODE_SUFFIXES A list of strings representing the file suffixes for optimized bytecode @@ -696,14 +721,19 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: BYTECODE_SUFFIXES A list of strings representing the recognized file suffixes for bytecode - modules. Set to either :attr:`DEBUG_BYTECODE_SUFFIXES` or - :attr:`OPTIMIZED_BYTECODE_SUFFIXES` based on whether ``__debug__`` is true. + modules (including the leading dot). .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The value is no longer dependent on ``__debug__``. + .. attribute:: EXTENSION_SUFFIXES A list of strings representing the recognized file suffixes for @@ -732,9 +762,9 @@ find and load modules. Only class methods are defined by this class to alleviate the need for instantiation. - .. note:: - Due to limitations in the extension module C-API, for now - BuiltinImporter does not implement :meth:`Loader.exec_module`. + .. versionchanged:: 3.5 + As part of :pep:`489`, the builtin importer now implements + :meth:`Loader.create_module` and :meth:`Loader.exec_module` .. class:: FrozenImporter @@ -782,6 +812,11 @@ find and load modules. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + If the current working directory -- represented by an empty string -- + is no longer valid then ``None`` is returned but no value is cached + in :data:`sys.path_importer_cache`. + .. classmethod:: find_module(fullname, path=None) A legacy wrapper around :meth:`find_spec`. @@ -944,14 +979,18 @@ find and load modules. Path to the extension module. - .. method:: load_module(name=None) + .. method:: create_module(spec) + + Creates the module object from the given specification in accordance + with :pep:`489`. + + .. versionadded:: 3.5 - Loads the extension module if and only if *fullname* is the same as - :attr:`name` or is ``None``. + .. method:: exec_module(module) - .. note:: - Due to limitations in the extension module C-API, for now - ExtensionFileLoader does not implement :meth:`Loader.exec_module`. + Initializes the given module object in accordance with :pep:`489`. + + .. versionadded:: 3.5 .. method:: is_package(fullname) @@ -1047,23 +1086,37 @@ an :term:`importer`. .. versionadded:: 3.4 -.. function:: cache_from_source(path, debug_override=None) +.. function:: cache_from_source(path, debug_override=None, *, optimization=None) - Return the :pep:`3147` path to the byte-compiled file associated with the - source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return + Return the :pep:`3147`/:pep:`488` path to the byte-compiled file associated + with the source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. - - *path* need not exist. + :exc:`NotImplementedError` will be raised). + + The *optimization* parameter is used to specify the optimization level of the + bytecode file. An empty string represents no optimization, so + ``/foo/bar/baz.py`` with an *optimization* of ``''`` will result in a + bytecode path of ``/foo/bar/__pycache__/baz.cpython-32.pyc``. ``None`` causes + the interpter's optimization level to be used. Any other value's string + representation being used, so ``/foo/bar/baz.py`` with an *optimization* of + ``2`` will lead to the bytecode path of + ``/foo/bar/__pycache__/baz.cpython-32.opt-2.pyc``. The string representation + of *optimization* can only be alphanumeric, else :exc:`ValueError` is raised. + + The *debug_override* parameter is deprecated and can be used to override + the system's value for ``__debug__``. A ``True`` value is the equivalent of + setting *optimization* to the empty string. A ``False`` value is the same as + setting *optimization* to ``1``. If both *debug_override* an *optimization* + are not ``None`` then :exc:`TypeError` is raised. .. versionadded:: 3.4 + .. versionchanged ::3.5 + The *optimization* parameter was added and the *debug_override* parameter + was deprecated. + .. function:: source_from_cache(path) @@ -1071,7 +1124,7 @@ an :term:`importer`. file path. For example, if *path* is ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be ``/foo/bar/baz.py``. *path* need not exist, however if it does not conform - to :pep:`3147` format, a ``ValueError`` is raised. If + to :pep:`3147` or :pep:`488` format, a ``ValueError`` is raised. If :attr:`sys.implementation.cache_tag` is not defined, :exc:`NotImplementedError` is raised. @@ -1117,6 +1170,21 @@ an :term:`importer`. .. versionadded:: 3.4 +.. function:: module_from_spec(spec) + + Create a new module based on **spec** and ``spec.loader.create_module()``. + + If ``spec.loader.create_module()`` does not return ``None``, then any + pre-existing attributes will not be reset. Also, no :exc:`AttributeError` + will be raised if triggered while accessing **spec** or setting an attribute + on the module. + + This function is preferred over using :class:`types.ModuleType` to create a + new module as **spec** is used to set as many import-controlled attributes on + the module as possible. + + .. versionadded:: 3.5 + .. decorator:: module_for_loader A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` @@ -1195,3 +1263,39 @@ an :term:`importer`. module will be file-based. .. versionadded:: 3.4 + +.. class:: LazyLoader(loader) + + A class which postpones the execution of the loader of a module until the + module has an attribute accessed. + + This class **only** works with loaders that define + :meth:`~importlib.abc.Loader.exec_module` as control over what module type + is used for the module is required. For those same reasons, the loader's + :meth:`~importlib.abc.Loader.create_module` method will be ignored (i.e., the + loader's method should only return ``None``). Finally, + modules which substitute the object placed into :attr:`sys.modules` will + not work as there is no way to properly replace the module references + throughout the interpreter safely; :exc:`ValueError` is raised if such a + substitution is detected. + + .. note:: + For projects where startup time is critical, this class allows for + potentially minimizing the cost of loading a module if it is never used. + For projects where startup time is not essential then use of this class is + **heavily** discouraged due to error messages created during loading being + postponed and thus occurring out of context. + + .. versionadded:: 3.5 + + .. classmethod:: factory(loader) + + A static method which returns a callable that creates a lazy loader. This + is meant to be used in situations where the loader is passed by class + instead of by instance. + :: + + suffixes = importlib.machinery.SOURCE_SUFFIXES + loader = importlib.machinery.SourceFileLoader + lazy_loader = importlib.util.LazyLoader.factory(loader) + finder = importlib.machinery.FileFinder(path, [(lazy_loader, suffixes)]) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 57eb4fff82..26a2913d79 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -28,7 +28,7 @@ Types and members ----------------- The :func:`getmembers` function retrieves the members of an object such as a -class or module. The sixteen functions whose names begin with "is" are mainly +class or module. The functions whose names begin with "is" are mainly provided as convenient choices for the second argument to :func:`getmembers`. They also help you determine when you can expect to find the following special attributes: @@ -168,6 +168,33 @@ attributes: | | | arguments and local | | | | variables | +-----------+-----------------+---------------------------+ +| generator | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | gi_frame | frame | ++-----------+-----------------+---------------------------+ +| | gi_running | is the generator running? | ++-----------+-----------------+---------------------------+ +| | gi_code | code | ++-----------+-----------------+---------------------------+ +| | gi_yieldfrom | object being iterated by | +| | | ``yield from``, or | +| | | ``None`` | ++-----------+-----------------+---------------------------+ +| coroutine | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | cr_await | object being awaited on, | +| | | or ``None`` | ++-----------+-----------------+---------------------------+ +| | cr_frame | frame | ++-----------+-----------------+---------------------------+ +| | cr_running | is the coroutine running? | ++-----------+-----------------+---------------------------+ +| | cr_code | code | ++-----------+-----------------+---------------------------+ | builtin | __doc__ | documentation string | +-----------+-----------------+---------------------------+ | | __name__ | original name of this | @@ -180,6 +207,13 @@ attributes: | | | ``None`` | +-----------+-----------------+---------------------------+ +.. versionchanged:: 3.5 + + Add ``__qualname__`` and ``gi_yieldfrom`` attributes to generators. + + The ``__name__`` attribute of generators is now set from the function + name, instead of the code name, and it can now be modified. + .. function:: getmembers(object[, predicate]) @@ -261,6 +295,41 @@ attributes: Return true if the object is a generator. +.. function:: iscoroutinefunction(object) + + Return true if the object is a :term:`coroutine function` + (a function defined with an :keyword:`async def` syntax). + + .. versionadded:: 3.5 + + +.. function:: iscoroutine(object) + + Return true if the object is a :term:`coroutine` created by an + :keyword:`async def` function. + + .. versionadded:: 3.5 + + +.. function:: isawaitable(object) + + Return true if the object can be used in :keyword:`await` expression. + + Can also be used to distinguish generator-based coroutines from regular + generators:: + + def gen(): + yield + @types.coroutine + def gen_coro(): + yield + + assert not isawaitable(gen()) + assert isawaitable(gen_coro()) + + .. versionadded:: 3.5 + + .. function:: istraceback(object) Return true if the object is a traceback. @@ -351,6 +420,12 @@ Retrieving source code .. function:: getdoc(object) Get the documentation string for an object, cleaned up with :func:`cleandoc`. + If the documentation string for an object is not provided and the object is + a class, a method, a property or a descriptor, retrieve the documentation + string from the inheritance hierarchy. + + .. versionchanged:: 3.5 + Documentation strings are now inherited if not overridden. .. function:: getcomments(object) @@ -423,7 +498,7 @@ The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the :func:`signature` function. -.. function:: signature(callable) +.. function:: signature(callable, \*, follow_wrapped=True) Return a :class:`Signature` object for the given ``callable``:: @@ -448,6 +523,11 @@ function. Raises :exc:`ValueError` if no signature can be provided, and :exc:`TypeError` if that type of object is not supported. + .. versionadded:: 3.5 + ``follow_wrapped`` parameter. Pass ``False`` to get a signature of + ``callable`` specifically (``callable.__wrapped__`` will not be used to + unwrap decorated callables.) + .. note:: Some callables may not be introspectable in certain implementations of @@ -473,6 +553,9 @@ function. Signature objects are *immutable*. Use :meth:`Signature.replace` to make a modified copy. + .. versionchanged:: 3.5 + Signature objects are picklable and hashable. + .. attribute:: Signature.empty A special class-level marker to specify absence of a return annotation. @@ -517,12 +600,30 @@ function. >>> str(new_sig) "(a, b) -> 'new return anno'" + .. classmethod:: Signature.from_callable(obj, \*, follow_wrapped=True) + + Return a :class:`Signature` (or its subclass) object for a given callable + ``obj``. Pass ``follow_wrapped=False`` to get a signature of ``obj`` + without unwrapping its ``__wrapped__`` chain. + + This method simplifies subclassing of :class:`Signature`:: + + class MySignature(Signature): + pass + sig = MySignature.from_callable(min) + assert isinstance(sig, MySignature) + + .. versionadded:: 3.5 + .. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) Parameter objects are *immutable*. Instead of modifying a Parameter object, you can use :meth:`Parameter.replace` to create a modified copy. + .. versionchanged:: 3.5 + Parameter objects are picklable and hashable. + .. attribute:: Parameter.empty A special class-level marker to specify absence of default values and @@ -639,27 +740,8 @@ function. Arguments for which :meth:`Signature.bind` or :meth:`Signature.bind_partial` relied on a default value are skipped. - However, if needed, it is easy to include them. - - :: - - >>> def foo(a, b=10): - ... pass - - >>> sig = signature(foo) - >>> ba = sig.bind(5) - - >>> ba.args, ba.kwargs - ((5,), {}) - - >>> for param in sig.parameters.values(): - ... if (param.name not in ba.arguments - ... and param.default is not param.empty): - ... ba.arguments[param.name] = param.default - - >>> ba.args, ba.kwargs - ((5, 10), {}) - + However, if needed, use :meth:`BoundArguments.apply_defaults` to add + them. .. attribute:: BoundArguments.args @@ -675,6 +757,26 @@ function. A reference to the parent :class:`Signature` object. + .. method:: BoundArguments.apply_defaults() + + Set default values for missing arguments. + + For variable-positional arguments (``*args``) the default is an + empty tuple. + + For variable-keyword arguments (``**kwargs``) the default is an + empty dict. + + :: + + >>> def foo(a, b='ham', *args): pass + >>> ba = inspect.signature(foo).bind('spam') + >>> ba.apply_defaults() + >>> ba.arguments + OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())]) + + .. versionadded:: 3.5 + The :attr:`args` and :attr:`kwargs` properties can be used to invoke functions:: @@ -719,8 +821,10 @@ Classes and functions *n* elements listed in *args*. .. deprecated:: 3.0 - Use :func:`getfullargspec` instead, which provides information about - keyword-only arguments and annotations. + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. This function will be removed + in Python 3.6. .. function:: getfullargspec(func) @@ -741,15 +845,16 @@ Classes and functions The first four items in the tuple correspond to :func:`getargspec`. - .. note:: - Consider using the new :ref:`Signature Object <inspect-signature-object>` - interface, which provides a better way of introspecting functions. - .. versionchanged:: 3.4 This function is now based on :func:`signature`, but still ignores ``__wrapped__`` attributes and includes the already bound first parameter in the signature output for bound methods. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: getargvalues(frame) @@ -759,6 +864,11 @@ Classes and functions are the names of the ``*`` and ``**`` arguments or ``None``. *locals* is the locals dictionary of the given frame. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: formatargspec(args[, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations[, formatarg, formatvarargs, formatvarkw, formatvalue, formatreturns, formatannotations]]) @@ -781,6 +891,11 @@ Classes and functions >>> formatargspec(*getfullargspec(f)) '(a: int, b: float)' + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: formatargvalues(args[, varargs, varkw, locals, formatarg, formatvarargs, formatvarkw, formatvalue]) @@ -788,6 +903,11 @@ Classes and functions :func:`getargvalues`. The format\* arguments are the corresponding optional formatting functions that are called to turn names and values into strings. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: getmro(cls) @@ -822,8 +942,8 @@ Classes and functions .. versionadded:: 3.2 - .. note:: - Consider using the new :meth:`Signature.bind` instead. + .. deprecated:: 3.5 + Use :meth:`Signature.bind` and :meth:`Signature.bind_partial` instead. .. function:: getclosurevars(func) @@ -864,11 +984,17 @@ Classes and functions The interpreter stack --------------------- -When the following functions return "frame records," each record is a tuple of -six items: the frame object, the filename, the line number of the current line, +When the following functions return "frame records," each record is a +:term:`named tuple` +``FrameInfo(frame, filename, lineno, function, code_context, index)``. +The tuple contains the frame object, the filename, the line number of the +current line, the function name, a list of lines of context from the source code, and the index of the current line within that list. +.. versionchanged:: 3.5 + Return a named tuple instead of a tuple. + .. note:: Keeping references to frame objects, as found in the first element of the frame @@ -913,6 +1039,11 @@ line. returned list represents *frame*; the last entry represents the outermost call on *frame*'s stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: getinnerframes(traceback, context=1) @@ -921,6 +1052,11 @@ line. list represents *traceback*; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: currentframe() @@ -940,6 +1076,11 @@ line. returned list represents the caller; the last entry represents the outermost call on the stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: trace(context=1) @@ -948,6 +1089,11 @@ line. entry in the list represents the caller; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + Fetching attributes statically ------------------------------ @@ -1007,8 +1153,8 @@ code execution:: pass -Current State of a Generator ----------------------------- +Current State of Generators and Coroutines +------------------------------------------ When implementing coroutine schedulers and for other advanced uses of generators, it is useful to determine whether a generator is currently @@ -1028,6 +1174,21 @@ generator to be determined easily. .. versionadded:: 3.2 +.. function:: getcoroutinestate(coroutine) + + Get current state of a coroutine object. The function is intended to be + used with coroutine objects created by :keyword:`async def` functions, but + will accept any coroutine-like object that has ``cr_running`` and + ``cr_frame`` attributes. + + Possible states are: + * CORO_CREATED: Waiting to start execution. + * CORO_RUNNING: Currently being executed by the interpreter. + * CORO_SUSPENDED: Currently suspended at an await expression. + * CORO_CLOSED: Execution has completed. + + .. versionadded:: 3.5 + The current internal state of the generator can also be queried. This is mostly useful for testing purposes, to ensure that internal state is being updated as expected: @@ -1052,6 +1213,13 @@ updated as expected: .. versionadded:: 3.3 +.. function:: getcoroutinelocals(coroutine) + + This function is analogous to :func:`~inspect.getgeneratorlocals`, but + works for coroutine objects created by :keyword:`async def` functions. + + .. versionadded:: 3.5 + .. _inspect-module-cli: diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 592bc484ad..cb3e9edf11 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -339,8 +339,11 @@ I/O Base Classes if *size* is not specified). The current stream position isn't changed. This resizing can extend or reduce the current file size. In case of extension, the contents of the new file area depend on the platform - (on most systems, additional bytes are zero-filled, on Windows they're - undetermined). The new file size is returned. + (on most systems, additional bytes are zero-filled). The new file size + is returned. + + .. versionchanged:: 3.5 + Windows will now zero-fill files when extending. .. method:: writable() @@ -391,8 +394,8 @@ I/O Base Classes .. method:: readinto(b) Read up to ``len(b)`` bytes into :class:`bytearray` *b* and return the - number of bytes read. If the object is in non-blocking mode and no - bytes are available, ``None`` is returned. + number of bytes read. If the object is in non-blocking mode and no bytes + are available, ``None`` is returned. .. method:: write(b) @@ -465,9 +468,10 @@ I/O Base Classes .. method:: read1(size=-1) - Read and return up to *size* bytes, with at most one call to the underlying - raw stream's :meth:`~RawIOBase.read` method. This can be useful if you - are implementing your own buffering on top of a :class:`BufferedIOBase` + Read and return up to *size* bytes, with at most one call to the + underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. This can be useful if you are + implementing your own buffering on top of a :class:`BufferedIOBase` object. .. method:: readinto(b) @@ -478,8 +482,19 @@ I/O Base Classes Like :meth:`read`, multiple reads may be issued to the underlying raw stream, unless the latter is interactive. - A :exc:`BlockingIOError` is raised if the underlying raw stream is in - non blocking-mode, and has no data available at the moment. + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. method:: readinto1(b) + + Read up to ``len(b)`` bytes into bytearray *b*, using at most one call to + the underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. Return the number of bytes read. + + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. versionadded:: 3.5 .. method:: write(b) @@ -507,9 +522,12 @@ Raw File I/O The *name* can be one of two things: * a character string or :class:`bytes` object representing the path to the - file which will be opened; + file which will be opened. In this case closefd must be True (the default) + otherwise an error will be raised. * an integer representing the number of an existing OS-level file descriptor - to which the resulting :class:`FileIO` object will give access. + to which the resulting :class:`FileIO` object will give access. When the + FileIO object is closed this fd will be closed as well, unless *closefd* + is set to ``False``. The *mode* can be ``'r'``, ``'w'``, ``'x'`` or ``'a'`` for reading (default), writing, exclusive creation or appending. The file will be @@ -598,6 +616,11 @@ than raw I/O does. In :class:`BytesIO`, this is the same as :meth:`read`. + .. method:: readinto1() + + In :class:`BytesIO`, this is the same as :meth:`readinto`. + + .. versionadded:: 3.5 .. class:: BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE) @@ -808,11 +831,13 @@ Text I/O exception if there is an encoding error (the default of ``None`` has the same effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding errors can lead to data loss.) ``'replace'`` causes a replacement marker - (such as ``'?'``) to be inserted where there is malformed data. When - writing, ``'xmlcharrefreplace'`` (replace with the appropriate XML character - reference) or ``'backslashreplace'`` (replace with backslashed escape - sequences) can be used. Any other error handling name that has been - registered with :func:`codecs.register_error` is also valid. + (such as ``'?'``) to be inserted where there is malformed data. + ``'backslashreplace'`` causes malformed data to be replaced by a + backslashed escape sequence. When writing, ``'xmlcharrefreplace'`` + (replace with the appropriate XML character reference) or ``'namereplace'`` + (replace with ``\N{...}`` escape sequences) can be used. Any other error + handling name that has been registered with + :func:`codecs.register_error` is also valid. .. index:: single: universal newlines; io.TextIOWrapper class diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 301048ef6a..90fcc748e7 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -146,6 +146,20 @@ write code that handles both IP versions correctly. the appropriate length (most significant octet first). This is 4 bytes for IPv4 and 16 bytes for IPv6. + .. attribute:: reverse_pointer + + The name of the reverse DNS PTR record for the IP address, e.g.:: + + >>> ipaddress.ip_address("127.0.0.1").reverse_pointer + '1.0.0.127.in-addr.arpa' + >>> ipaddress.ip_address("2001:db8::1").reverse_pointer + '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa' + + This is the name that could be used for performing a PTR lookup, not the + resolved hostname itself. + + .. versionadded:: 3.5 + .. attribute:: is_multicast ``True`` if the address is reserved for multicast use. See @@ -226,6 +240,7 @@ write code that handles both IP versions correctly. :class:`IPv4Address` class: .. attribute:: packed + .. attribute:: reverse_pointer .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast @@ -377,6 +392,12 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 4, big-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 32-bits integer, a 4-bytes packed + integer, or an existing IPv4Address object; and the netmask is either + an integer representing the prefix length (e.g. ``24``) or a string + representing the prefix mask (e.g. ``255.255.255.0``). + An :exc:`AddressValueError` is raised if *address* is not a valid IPv4 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv4 address. @@ -389,6 +410,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. objects will raise :exc:`TypeError` if the argument's IP version is incompatible to ``self`` + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen @@ -553,6 +578,11 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 16, big-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 128-bits integer, a 16-bytes packed + integer, or an existing IPv4Address object; and the netmask is an + integer representing the prefix length. + An :exc:`AddressValueError` is raised if *address* is not a valid IPv6 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv6 address. @@ -561,6 +591,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. then :exc:`ValueError` is raised. Otherwise, the host bits are masked out to determine the appropriate network address. + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index f489535c53..8c7592d17e 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -87,10 +87,15 @@ loops that truncate the stream. .. function:: accumulate(iterable[, func]) - Make an iterator that returns accumulated sums. Elements may be any addable - type including :class:`~decimal.Decimal` or :class:`~fractions.Fraction`. - If the optional *func* argument is supplied, it should be a function of two - arguments and it will be used instead of addition. + Make an iterator that returns accumulated sums, or accumulated + results of other binary functions (specified via the optional + *func* argument). If *func* is supplied, it should be a function + of two arguments. Elements of the input *iterable* may be any type + that can be accepted as arguments to *func*. (For example, with + the default operation of addition, elements may be any addable + type including :class:`~decimal.Decimal` or + :class:`~fractions.Fraction`.) If the input iterable is empty, the + output iterable will also be empty. Equivalent to:: @@ -99,7 +104,10 @@ loops that truncate the stream. # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) - total = next(it) + try: + total = next(it) + except StopIteration: + return yield total for element in it: total = func(total, element) @@ -400,7 +408,10 @@ loops that truncate the stream. def _grouper(self, tgtkey): while self.currkey == tgtkey: yield self.currvalue - self.currvalue = next(self.it) # Exit on StopIteration + try: + self.currvalue = next(self.it) + except StopIteration: + return self.currkey = self.keyfunc(self.currvalue) @@ -424,7 +435,10 @@ loops that truncate the stream. # islice('ABCDEFG', 0, None, 2) --> A C E G s = slice(*args) it = iter(range(s.start or 0, s.stop or sys.maxsize, s.step or 1)) - nexti = next(it) + try: + nexti = next(it) + except StopIteration: + return for i, element in enumerate(iterable): if i == nexti: yield element @@ -582,7 +596,10 @@ loops that truncate the stream. def gen(mydeque): while True: if not mydeque: # when the local deque is empty - newval = next(it) # fetch a new value and + try: + newval = next(it) # fetch a new value and + except StopIteration: + return for d in deques: # load it to all the deques d.append(newval) yield mydeque.popleft() @@ -657,6 +674,11 @@ which incur interpreter overhead. "Return function(0), function(1), ..." return map(function, count(start)) + def tail(n, iterable): + "Return an iterator over the last n items" + # tail(3, 'ABCDEFG') --> E F G + return iter(collections.deque(iterable, maxlen=n)) + def consume(iterator, n): "Advance the iterator n-steps ahead. If n is none, consume entirely." # Use functions that consume iterators at C speed. diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 6f5f8b15a8..49bb0905e0 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -106,6 +106,8 @@ Using json.tool from the shell to validate and pretty-print:: $ echo '{1.2:3.4}' | python -m json.tool Expecting property name enclosed in double quotes: line 1 column 2 (char 1) +See :ref:`json-commandline` for detailed documentation. + .. highlight:: python3 .. note:: @@ -248,7 +250,7 @@ Basic Usage will be passed to the constructor of the class. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. function:: loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) @@ -259,7 +261,7 @@ Basic Usage *encoding* which is ignored and deprecated. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. Encoders and Decoders --------------------- @@ -332,13 +334,16 @@ Encoders and Decoders ``'\n'``, ``'\r'`` and ``'\0'``. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. method:: decode(s) Return the Python representation of *s* (a :class:`str` instance containing a JSON document) + :exc:`JSONDecodeError` will be raised if the given JSON document is not + valid. + .. method:: raw_decode(s) Decode a JSON document from *s* (a :class:`str` beginning with a @@ -467,6 +472,36 @@ Encoders and Decoders mysocket.write(chunk) +Exceptions +---------- + +.. exception:: JSONDecodeError(msg, doc, pos, end=None) + + Subclass of :exc:`ValueError` with the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: doc + + The JSON document being parsed. + + .. attribute:: pos + + The start index of *doc* where parsing failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + + .. versionadded:: 3.5 + + Standard Compliance and Interoperability ---------------------------------------- @@ -588,6 +623,68 @@ when serializing Python :class:`int` values of extremely large magnitude, or when serializing instances of "exotic" numerical types such as :class:`decimal.Decimal`. +.. highlight:: bash +.. module:: json.tool + +.. _json-commandline: + +Command Line Interface +---------------------- + +The :mod:`json.tool` module provides a simple command line interface to validate +and pretty-print JSON objects. + +If the optional ``infile`` and ``outfile`` arguments are not +specified, :attr:`sys.stdin` and :attr:`sys.stdout` will be used respectively:: + + $ echo '{"json": "obj"}' | python -m json.tool + { + "json": "obj" + } + $ echo '{1.2:3.4}' | python -m json.tool + Expecting property name enclosed in double quotes: line 1 column 2 (char 1) + +.. versionchanged:: 3.5 + The output is now in the same order as the input. Use the + :option:`--sort-keys` option to sort the output of dictionaries + alphabetically by key. + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +.. cmdoption:: infile + + The JSON file to be validated or pretty-printed:: + + $ python -m json.tool mp_films.json + [ + { + "title": "And Now for Something Completely Different", + "year": 1971 + }, + { + "title": "Monty Python and the Holy Grail", + "year": 1975 + } + ] + + If *infile* is not specified, read from :attr:`sys.stdin`. + +.. cmdoption:: outfile + + Write the output of the *infile* to the given *outfile*. Otherwise, write it + to :attr:`sys.stdout`. + +.. cmdoption:: --sort-keys + + Sort the output of dictionaries alphabetically by key. + + .. versionadded:: 3.5 + +.. cmdoption:: -h, --help + + Show the help message. + .. rubric:: Footnotes diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst index f18b1cdac0..6c92cc58d2 100644 --- a/Doc/library/linecache.rst +++ b/Doc/library/linecache.rst @@ -47,6 +47,14 @@ The :mod:`linecache` module defines the following functions: changed on disk, and you require the updated version. If *filename* is omitted, it will check all the entries in the cache. +.. function:: lazycache(filename, module_globals) + + Capture enough detail about a non-file based module to permit getting its + lines later via :func:`getline` even if *module_globals* is None in the later + call. This avoids doing I/O until a line is actually needed, without having + to carry the module globals around indefinitely. + + .. versionadded:: 3.5 Example:: diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 9600193547..bc7f5f932b 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -387,6 +387,14 @@ The :mod:`locale` module defines the following exception and functions: ``str(float)``, but takes the decimal point into account. +.. function:: delocalize(string) + + Converts a string into a normalized number string, following the + :const:`LC_NUMERIC` settings. + + .. versionadded:: 3.5 + + .. function:: atof(string) Converts a string to a floating point number, following the :const:`LC_NUMERIC` diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index d946892ecc..67403a9bc6 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -853,7 +853,7 @@ supports sending logging messages to a Web server, using either ``GET`` or credentials, you should also specify secure=True so that your userid and password are not passed in cleartext across the wire. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 The *context* parameter was added. .. method:: mapLogRecord(record) @@ -953,13 +953,20 @@ applications where threads servicing clients need to respond as quickly as possible, while any potentially slow operations (such as sending an email via :class:`SMTPHandler`) are done on a separate thread. -.. class:: QueueListener(queue, *handlers) +.. class:: QueueListener(queue, *handlers, respect_handler_level=False) Returns a new instance of the :class:`QueueListener` class. The instance is initialized with the queue to send messages to and a list of handlers which will handle entries placed on the queue. The queue can be any queue- like object; it's passed as-is to the :meth:`dequeue` method, which needs - to know how to get messages from it. + to know how to get messages from it. If ``respect_handler_level`` is ``True``, + a handler's level is respected (compared with the level for the message) when + deciding whether to pass messages to that handler; otherwise, the behaviour + is as in previous Python versions - to always pass each message to each + handler. + + .. versionchanged:: 3.5 + The ``respect_handler_levels`` argument was added. .. method:: dequeue(block) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 90fb9b06d1..bf821abac1 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -159,11 +159,13 @@ is the module's name in the Python package namespace. *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) - There are three keyword arguments in *kwargs* which are inspected: *exc_info* - which, if it does not evaluate as false, causes exception information to be + There are three keyword arguments in *kwargs* which are inspected: + *exc_info*, *stack_info*, and *extra*. + + If *exc_info* does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by - :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` - is called to get the exception information. + :func:`sys.exc_info`) or an exception instance is provided, it is used; + otherwise, :func:`sys.exc_info` is called to get the exception information. The second optional keyword argument is *stack_info*, which defaults to ``False``. If true, stack information is added to the logging @@ -220,6 +222,9 @@ is the module's name in the Python package namespace. .. versionadded:: 3.2 The *stack_info* parameter was added. + .. versionchanged:: 3.5 + The *exc_info* parameter can now accept exception instances. + .. method:: Logger.info(msg, *args, **kwargs) diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst index b71051da85..054600530b 100644 --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -110,6 +110,10 @@ Reading and writing compressed files .. versionchanged:: 3.4 Added support for the ``"x"`` and ``"xb"`` modes. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Compressing and decompressing data in memory -------------------------------------------- @@ -221,13 +225,32 @@ Compressing and decompressing data in memory decompress a multi-stream input with :class:`LZMADecompressor`, you must create a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. + + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. - Decompress *data* (a :class:`bytes` object), returning a :class:`bytes` - object containing the decompressed data for at least part of the input. - Some of *data* may be buffered internally, for use in later calls to - :meth:`decompress`. The returned data should be concatenated with the - output of any previous calls to :meth:`decompress`. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: check @@ -245,6 +268,12 @@ Compressing and decompressing data in memory Before the end of the stream is reached, this will be ``b""``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 .. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None) diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 0fc7c7c9b9..244663eda9 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -100,6 +100,48 @@ Number-theoretic and representation functions <http://code.activestate.com/recipes/393090/>`_\. +.. function:: gcd(a, b) + + Return the greatest common divisor of the integers *a* and *b*. If either + *a* or *b* is nonzero, then the value of ``gcd(a, b)`` is the largest + positive integer that divides both *a* and *b*. ``gcd(0, 0)`` returns + ``0``. + + .. versionadded:: 3.5 + + +.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) + + Return ``True`` if the values *a* and *b* are close to each other and + ``False`` otherwise. + + Whether or not two values are considered close is determined according to + given absolute and relative tolerances. + + *rel_tol* is the relative tolerance -- it is the maximum allowed difference + between *a* and *b*, relative to the larger absolute value of *a* or *b*. + For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default + tolerance is ``1e-09``, which assures that the two values are the same + within about 9 decimal digits. *rel_tol* must be greater than zero. + + *abs_tol* is the minimum absolute tolerance -- useful for comparisons near + zero. *abs_tol* must be at least zero. + + If no errors occur, the result will be: + ``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``. + + The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be + handled according to IEEE rules. Specifically, ``NaN`` is not considered + close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only + considered close to themselves. + + .. versionadded:: 3.5 + + .. seealso:: + + :pep:`485` -- A function for testing approximate equality + + .. function:: isfinite(x) Return ``True`` if *x* is neither an infinity nor a NaN, and @@ -383,6 +425,22 @@ Constants The mathematical constant e = 2.718281..., to available precision. +.. data:: inf + + A floating-point positive infinity. (For negative infinity, use + ``-math.inf``.) Equivalent to the output of ``float('inf')``. + + .. versionadded:: 3.5 + + +.. data:: nan + + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. + + .. versionadded:: 3.5 + + .. impl-detail:: The :mod:`math` module consists mostly of thin wrappers around the platform C diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 18e05e31f7..b74a8231ea 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -174,6 +174,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: flush([offset[, size]]) @@ -234,6 +237,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: seek(pos[, whence]) @@ -261,6 +267,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length were written. If the mmap was created with :const:`ACCESS_READ`, then writing to it will raise a :exc:`TypeError` exception. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: write_byte(byte) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 82de19b5c8..a4d42abb34 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -1458,6 +1458,9 @@ processes. Note that accessing the ctypes object through the wrapper can be a lot slower than accessing the raw ctypes object. + .. versionchanged:: 3.5 + Synchronized objects support the :term:`context manager` protocol. + The table below compares the syntax for creating shared ctypes objects from shared memory with the normal ctypes syntax. (In the table ``MyStruct`` is some @@ -1945,9 +1948,9 @@ itself. This means, for example, that one shared object can contain a second: >>> l = manager.list(range(10)) >>> l._callmethod('__len__') 10 - >>> l._callmethod('__getitem__', (slice(2, 7),)) # equiv to `l[2:7]` + >>> l._callmethod('__getitem__', (slice(2, 7),)) # equivalent to l[2:7] [2, 3, 4, 5, 6] - >>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]` + >>> l._callmethod('__getitem__', (20,)) # equivalent to l[20] Traceback (most recent call last): ... IndexError: list index out of range diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index f9e2a3d003..c01e63b77a 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -138,6 +138,14 @@ The mathematical and bitwise operations are the most numerous: Return ``a * b``, for *a* and *b* numbers. +.. function:: matmul(a, b) + __matmul__(a, b) + + Return ``a @ b``. + + .. versionadded:: 3.5 + + .. function:: neg(obj) __neg__(obj) @@ -391,6 +399,8 @@ Python syntax and the functions in the :mod:`operator` module. +-----------------------+-------------------------+---------------------------------------+ | Multiplication | ``a * b`` | ``mul(a, b)`` | +-----------------------+-------------------------+---------------------------------------+ +| Matrix Multiplication | ``a @ b`` | ``matmul(a, b)`` | ++-----------------------+-------------------------+---------------------------------------+ | Negation (Arithmetic) | ``- a`` | ``neg(a)`` | +-----------------------+-------------------------+---------------------------------------+ | Negation (Logical) | ``not a`` | ``not_(a)`` | @@ -499,6 +509,14 @@ will perform the update, so no subsequent assignment is necessary: ``a = imul(a, b)`` is equivalent to ``a *= b``. +.. function:: imatmul(a, b) + __imatmul__(a, b) + + ``a = imatmul(a, b)`` is equivalent to ``a @= b``. + + .. versionadded:: 3.5 + + .. function:: ior(a, b) __ior__(a, b) diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 92631b2e41..a3fe73c152 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -66,11 +66,37 @@ the :mod:`glob` module.) empty string (``''``). +.. function:: commonpath(paths) + + Return the longest common sub-path of each pathname in the sequence + *paths*. Raise ValueError if *paths* contains both absolute and relative + pathnames, or if *paths* is empty. Unlike :func:`commonprefix`, this + returns a valid path. + + Availability: Unix, Windows + + .. versionadded:: 3.5 + + .. function:: commonprefix(list) - Return the longest path prefix (taken character-by-character) that is a prefix - of all paths in *list*. If *list* is empty, return the empty string (``''``). - Note that this may return invalid paths because it works a character at a time. + Return the longest path prefix (taken character-by-character) that is a + prefix of all paths in *list*. If *list* is empty, return the empty string + (``''``). + + .. note:: + + This function may return invalid paths because it works a + character at a time. To obtain a valid path, see + :func:`commonpath`. + + :: + + >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) + '/usr/l' + + >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) + '/usr' .. function:: dirname(path) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 916f5e56cf..30f1605c10 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -78,9 +78,10 @@ uses the file system encoding to perform this conversion (see .. versionchanged:: 3.1 On some systems, conversion using the file system encoding may fail. In this - case, Python uses the ``surrogateescape`` encoding error handler, which means - that undecodable bytes are replaced by a Unicode character U+DCxx on - decoding, and these are again translated to the original byte on encoding. + case, Python uses the :ref:`surrogateescape encoding error handler + <surrogateescape>`, which means that undecodable bytes are replaced by a + Unicode character U+DCxx on decoding, and these are again translated to the + original byte on encoding. The file system encoding must guarantee to successfully decode all bytes @@ -311,8 +312,6 @@ process and user. Return the current process id. - Availability: Unix, Windows. - .. function:: getppid() @@ -549,8 +548,6 @@ process and user. On platforms where :c:func:`strerror` returns ``NULL`` when given an unknown error number, :exc:`ValueError` is raised. - Availability: Unix, Windows. - .. data:: supports_bytes_environ @@ -564,8 +561,6 @@ process and user. Set the current numeric umask and return the previous umask. - Availability: Unix, Windows. - .. function:: uname() @@ -656,8 +651,6 @@ as internal buffering of data. Close file descriptor *fd*. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -677,8 +670,6 @@ as internal buffering of data. except OSError: pass - Availability: Unix, Windows. - .. function:: device_encoding(fd) @@ -695,8 +686,6 @@ as internal buffering of data. 2: stderr), the new file descriptor is :ref:`inheritable <fd_inheritance>`. - Availability: Unix, Windows. - .. versionchanged:: 3.4 The new file descriptor is now non-inheritable. @@ -707,8 +696,6 @@ as internal buffering of data. The file descriptor *fd2* is :ref:`inheritable <fd_inheritance>` by default, or non-inheritable if *inheritable* is ``False``. - Availability: Unix, Windows. - .. versionchanged:: 3.4 Add the optional *inheritable* parameter. @@ -774,8 +761,6 @@ as internal buffering of data. The :func:`.stat` function. - Availability: Unix, Windows. - .. function:: fstatvfs(fd) @@ -804,8 +789,21 @@ as internal buffering of data. most *length* bytes in size. As of Python 3.3, this is equivalent to ``os.truncate(fd, length)``. + Availability: Unix, Windows. + + .. versionchanged:: 3.5 + Added support for Windows + +.. function:: get_blocking(fd) + + Get the blocking mode of the file descriptor: ``False`` if the + :data:`O_NONBLOCK` flag is set, ``True`` if the flag is cleared. + + See also :func:`set_blocking` and :meth:`socket.socket.setblocking`. + Availability: Unix. + .. versionadded:: 3.5 .. function:: isatty(fd) @@ -846,8 +844,6 @@ as internal buffering of data. current position; :const:`SEEK_END` or ``2`` to set it relative to the end of the file. Return the new cursor position in bytes, starting from the beginning. - Availability: Unix, Windows. - .. data:: SEEK_SET SEEK_CUR @@ -856,8 +852,6 @@ as internal buffering of data. Parameters to the :func:`lseek` function. Their values are 0, 1, and 2, respectively. - Availability: Unix, Windows. - .. versionadded:: 3.3 Some operating systems could support additional values, like :data:`os.SEEK_HOLE` or :data:`os.SEEK_DATA`. @@ -878,8 +872,6 @@ as internal buffering of data. This function can support :ref:`paths relative to directory descriptors <dir_fd>` with the *dir_fd* parameter. - Availability: Unix, Windows. - .. versionchanged:: 3.4 The new file descriptor is now non-inheritable. @@ -893,6 +885,11 @@ as internal buffering of data. .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + The following constants are options for the *flags* parameter to the :func:`~os.open` function. They can be combined using the bitwise OR operator ``|``. Some of them are not available on all platforms. For descriptions of @@ -1060,8 +1057,6 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window bytes read. If the end of the file referred to by *fd* has been reached, an empty bytes object is returned. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -1070,6 +1065,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`popen` or :func:`fdopen`, or :data:`sys.stdin`, use its :meth:`~file.read` or :meth:`~file.readline` methods. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: sendfile(out, in, offset, count) sendfile(out, in, offset, count, [headers], [trailers], flags=0) @@ -1099,9 +1099,26 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window Availability: Unix. + .. note:: + + For a higher-level wrapper of :func:`sendfile`, see + :meth:`socket.socket.sendfile`. + .. versionadded:: 3.3 +.. function:: set_blocking(fd, blocking) + + Set the blocking mode of the specified file descriptor. Set the + :data:`O_NONBLOCK` flag if blocking is ``False``, clear the flag otherwise. + + See also :func:`get_blocking` and :meth:`socket.socket.setblocking`. + + Availability: Unix. + + .. versionadded:: 3.5 + + .. data:: SF_NODISKIO SF_MNOWAIT SF_SYNC @@ -1158,8 +1175,6 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window Write the bytestring in *str* to file descriptor *fd*. Return the number of bytes actually written. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -1168,6 +1183,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`fdopen`, or :data:`sys.stdout` or :data:`sys.stderr`, use its :meth:`~file.write` method. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: writev(fd, buffers) @@ -1330,8 +1350,6 @@ features: or not it is available using :data:`os.supports_effective_ids`. If it is unavailable, using it will raise a :exc:`NotImplementedError`. - Availability: Unix, Windows. - .. note:: Using :func:`access` to check if a user is authorized to e.g. open a file @@ -1384,8 +1402,6 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. The descriptor must refer to an opened directory, not an open file. - Availability: Unix, Windows. - .. versionadded:: 3.3 Added support for specifying *path* as a file descriptor on some platforms. @@ -1447,8 +1463,6 @@ features: :ref:`paths relative to directory descriptors <dir_fd>` and :ref:`not following symlinks <follow_symlinks>`. - Availability: Unix, Windows. - .. note:: Although Windows supports :func:`chmod`, you can only set the file's @@ -1499,15 +1513,11 @@ features: Return a string representing the current working directory. - Availability: Unix, Windows. - .. function:: getcwdb() Return a bytestring representing the current working directory. - Availability: Unix, Windows. - .. function:: lchflags(path, flags) @@ -1570,7 +1580,11 @@ features: .. note:: To encode ``str`` filenames to ``bytes``, use :func:`~os.fsencode`. - Availability: Unix, Windows. + .. seealso:: + + The :func:`scandir` function returns directory entries along with + file attribute information, giving better performance for many + common use cases. .. versionchanged:: 3.2 The *path* parameter became optional. @@ -1619,8 +1633,6 @@ features: It is also possible to create temporary directories; see the :mod:`tempfile` module's :func:`tempfile.mkdtemp` function. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* argument. @@ -1779,8 +1791,6 @@ features: This function is identical to :func:`unlink`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* argument. @@ -1814,8 +1824,6 @@ features: If you want cross-platform overwriting of the destination, use :func:`replace`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *src_dir_fd* and *dst_dir_fd* arguments. @@ -1844,8 +1852,6 @@ features: This function can support specifying *src_dir_fd* and/or *dst_dir_fd* to supply :ref:`paths relative to directory descriptors <dir_fd>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 @@ -1858,12 +1864,180 @@ features: This function can support :ref:`paths relative to directory descriptors <dir_fd>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* parameter. +.. function:: scandir(path='.') + + Return an iterator of :class:`DirEntry` objects corresponding to the entries + in the directory given by *path*. The entries are yielded in arbitrary + order, and the special entries ``'.'`` and ``'..'`` are not included. + + Using :func:`scandir` instead of :func:`listdir` can significantly + increase the performance of code that also needs file type or file + attribute information, because :class:`DirEntry` objects expose this + information if the operating system provides it when scanning a directory. + All :class:`DirEntry` methods may perform a system call, but + :func:`~DirEntry.is_dir` and :func:`~DirEntry.is_file` usually only + require a system call for symbolic links; :func:`DirEntry.stat` + always requires a system call on Unix but only requires one for + symbolic links on Windows. + + On Unix, *path* can be of type :class:`str` or :class:`bytes` (use + :func:`~os.fsencode` and :func:`~os.fsdecode` to encode and decode + :class:`bytes` paths). On Windows, *path* must be of type :class:`str`. + On both sytems, the type of the :attr:`~DirEntry.name` and + :attr:`~DirEntry.path` attributes of each :class:`DirEntry` will be of + the same type as *path*. + + The following example shows a simple use of :func:`scandir` to display all + the files (excluding directories) in the given *path* that don't start with + ``'.'``. The ``entry.is_file()`` call will generally not make an additional + system call:: + + for entry in os.scandir(path): + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) + + .. note:: + + On Unix-based systems, :func:`scandir` uses the system's + `opendir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/opendir.html>`_ + and + `readdir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir_r.html>`_ + functions. On Windows, it uses the Win32 + `FindFirstFileW <http://msdn.microsoft.com/en-us/library/windows/desktop/aa364418(v=vs.85).aspx>`_ + and + `FindNextFileW <http://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx>`_ + functions. + + .. versionadded:: 3.5 + + +.. class:: DirEntry + + Object yielded by :func:`scandir` to expose the file path and other file + attributes of a directory entry. + + :func:`scandir` will provide as much of this information as possible without + making additional system calls. When a ``stat()`` or ``lstat()`` system call + is made, the ``DirEntry`` object will cache the result. + + ``DirEntry`` instances are not intended to be stored in long-lived data + structures; if you know the file metadata has changed or if a long time has + elapsed since calling :func:`scandir`, call ``os.stat(entry.path)`` to fetch + up-to-date information. + + Because the ``DirEntry`` methods can make operating system calls, they may + also raise :exc:`OSError`. If you need very fine-grained + control over errors, you can catch :exc:`OSError` when calling one of the + ``DirEntry`` methods and handle as appropriate. + + Attributes and methods on a ``DirEntry`` instance are as follows: + + .. attribute:: name + + The entry's base filename, relative to the :func:`scandir` *path* + argument. + + The :attr:`name` attribute will be of the same type (``str`` or + ``bytes``) as the :func:`scandir` *path* argument. Use + :func:`~os.fsdecode` to decode byte filenames. + + .. attribute:: path + + The entry's full path name: equivalent to ``os.path.join(scandir_path, + entry.name)`` where *scandir_path* is the :func:`scandir` *path* + argument. The path is only absolute if the :func:`scandir` *path* + argument was absolute. + + The :attr:`path` attribute will be of the same type (``str`` or + ``bytes``) as the :func:`scandir` *path* argument. Use + :func:`~os.fsdecode` to decode byte filenames. + + .. method:: inode() + + Return the inode number of the entry. + + The result is cached on the ``DirEntry`` object, use ``os.stat(entry.path, + follow_symlinks=False).st_ino`` to fetch up-to-date information. + + On Unix, no system call is required. + + .. method:: is_dir(\*, follow_symlinks=True) + + If *follow_symlinks* is ``True`` (the default), return ``True`` if the + entry is a directory or a symbolic link pointing to a directory; + return ``False`` if it is or points to any other kind of file, or if it + doesn't exist anymore. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a directory; return ``False`` if it is any other kind of file + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + along with :func:`stat.S_ISDIR` to fetch up-to-date information. + + This method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: is_file(\*, follow_symlinks=True) + + If *follow_symlinks* is ``True`` (the default), return ``True`` if the + entry is a file or a symbolic link pointing to a file; return ``False`` + if it is or points to a directory or other non-file entry, or if it + doesn't exist anymore. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a file; return ``False`` if it is a directory or other non-file entry, + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + along with :func:`stat.S_ISREG` to fetch up-to-date information. + + This method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: is_symlink() + + Return ``True`` if this entry is a symbolic link (even if broken); + return ``False`` if it points to a directory or any kind of file, + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call + :func:`os.path.islink` to fetch up-to-date information. + + The method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: stat(\*, follow_symlinks=True) + + Return a :class:`stat_result` object for this entry. This method + follows symbolic links by default; to stat a symbolic link add the + ``follow_symlinks=False`` argument. + + On Unix, this method always requires a system call. On Windows, + ``DirEntry.stat()`` requires a system call only if the + entry is a symbolic link, and ``DirEntry.stat(follow_symlinks=False)`` + never requires a system call. + + On Windows, the ``st_ino``, ``st_dev`` and ``st_nlink`` attributes of the + :class:`stat_result` are always set to zero. Call :func:`os.stat` to + get these attributes. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + to fetch up-to-date information. + + .. versionadded:: 3.5 + + .. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) Get the status of a file or a file descriptor. Perform the equivalent of a @@ -1890,8 +2064,6 @@ features: >>> statinfo.st_size 264 - Availability: Unix, Windows. - .. seealso:: :func:`fstat` and :func:`lstat` functions. @@ -2039,6 +2211,15 @@ features: File type. + On Windows systems, the following attribute is also available: + + .. attribute:: st_file_attributes + + Windows file attributes: ``dwFileAttributes`` member of the + ``BY_HANDLE_FILE_INFORMATION`` structure returned by + :c:func:`GetFileInformationByHandle`. See the ``FILE_ATTRIBUTE_*`` + constants in the :mod:`stat` module. + The standard module :mod:`stat` defines functions and constants that are useful for extracting information from a :c:type:`stat` structure. (On Windows, some items are filled with dummy values.) @@ -2056,6 +2237,9 @@ features: Added the :attr:`st_atime_ns`, :attr:`st_mtime_ns`, and :attr:`st_ctime_ns` members. + .. versionadded:: 3.5 + Added the :attr:`st_file_attributes` member on Windows. + .. function:: stat_float_times([newvalue]) @@ -2259,10 +2443,12 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. - Availability: Unix. + Availability: Unix, Windows. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + Added support for Windows .. function:: unlink(path, *, dir_fd=None) @@ -2271,8 +2457,6 @@ features: name. Please see the documentation for :func:`remove` for further information. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* parameter. @@ -2309,8 +2493,6 @@ features: :ref:`paths relative to directory descriptors <dir_fd>` and :ref:`not following symlinks <follow_symlinks>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 Added support for specifying an open file descriptor for *path*, and the *dir_fd*, *follow_symlinks*, and *ns* parameters. @@ -2401,6 +2583,10 @@ features: for name in dirs: os.rmdir(os.path.join(root, name)) + .. versionchanged:: 3.5 + This function now calls :func:`os.scandir` instead of :func:`os.listdir`, + making it faster by reducing the number of calls to :func:`os.stat`. + .. function:: fwalk(top='.', topdown=True, onerror=None, *, follow_symlinks=False, dir_fd=None) @@ -2557,8 +2743,6 @@ to be ignored. Python signal handler registered for :const:`SIGABRT` with :func:`signal.signal`. - Availability: Unix, Windows. - .. function:: execl(path, arg0, arg1, ...) execle(path, arg0, arg1, ..., env) @@ -2622,8 +2806,6 @@ to be ignored. Exit the process with status *n*, without calling cleanup handlers, flushing stdio buffers, etc. - Availability: Unix, Windows. - .. note:: The standard way to exit is ``sys.exit(n)``. :func:`_exit` should @@ -2986,6 +3168,10 @@ written in Python, such as a mail server's external command delivery program. doesn't work if it is. Use the :func:`os.path.normpath` function to ensure that the path is properly encoded for Win32. + To reduce interpreter startup overhead, the Win32 :c:func:`ShellExecute` + function is not resolved until this function is first called. If the function + cannot be resolved, :exc:`NotImplementedError` will be raised. + Availability: Windows. @@ -3133,6 +3319,11 @@ written in Python, such as a mail server's external command delivery program. id is known, not necessarily a child process. The :func:`spawn\* <spawnl>` functions called with :const:`P_NOWAIT` return suitable process handles. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: wait3(options) @@ -3540,3 +3731,9 @@ Miscellaneous Functions For an easy-to-use interface to the random number generator provided by your platform, please see :class:`random.SystemRandom`. + + .. versionchanged:: 3.5 + On Linux 3.17 and newer, the ``getrandom()`` syscall is now used + when available. On OpenBSD 5.6 and newer, the C ``getentropy()`` + function is now used. These functions avoid the usage of an internal file + descriptor. diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst index bb5081afde..c60d596506 100644 --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -148,21 +148,30 @@ and (read-only) attributes: .. method:: oss_audio_device.write(data) - Write the Python string *data* to the audio device and return the number of - bytes written. If the audio device is in blocking mode (the default), the - entire string is always written (again, this is different from usual Unix device - semantics). If the device is in non-blocking mode, some data may not be written + Write a :term:`bytes-like object` *data* to the audio device and return the + number of bytes written. If the audio device is in blocking mode (the + default), the entire data is always written (again, this is different from + usual Unix device semantics). If the device is in non-blocking mode, some + data may not be written ---see :meth:`writeall`. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: oss_audio_device.writeall(data) - Write the entire Python string *data* to the audio device: waits until the audio - device is able to accept data, writes as much data as it will accept, and - repeats until *data* has been completely written. If the device is in blocking - mode (the default), this has the same effect as :meth:`write`; :meth:`writeall` - is only useful in non-blocking mode. Has no return value, since the amount of - data written is always equal to the amount of data supplied. + Write a :term:`bytes-like object` *data* to the audio device: waits until + the audio device is able to accept data, writes as much data as it will + accept, and repeats until *data* has been completely written. If the device + is in blocking mode (the default), this has the same effect as + :meth:`write`; :meth:`writeall` is only useful in non-blocking mode. Has + no return value, since the amount of data written is always equal to the + amount of data supplied. + + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.2 Audio device objects also support the context management protocol, i.e. they can diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 24e2a308df..2f0654440d 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -628,6 +628,17 @@ call fails (for example because the path doesn't exist): PosixPath('/home/antoine/pathlib') +.. classmethod:: Path.home() + + Return a new path object representing the user's home directory (as + returned by :func:`os.path.expanduser` with ``~`` construct):: + + >>> Path.home() + PosixPath('/home/antoine') + + .. versionadded:: 3.5 + + .. method:: Path.stat() Return information about this path (similarly to :func:`os.stat`). @@ -670,6 +681,18 @@ call fails (for example because the path doesn't exist): symlink *points to* an existing file or directory. +.. method:: Path.expanduser() + + Return a new path with expanded ``~`` and ``~user`` constructs, + as returned by :meth:`os.path.expanduser`:: + + >>> p = PosixPath('~/films/Monty Python') + >>> p.expanduser() + PosixPath('/home/eric/films/Monty Python') + + .. versionadded:: 3.5 + + .. method:: Path.glob(pattern) Glob the given *pattern* in the directory represented by this path, @@ -791,7 +814,7 @@ call fails (for example because the path doesn't exist): the symbolic link's information rather than its target's. -.. method:: Path.mkdir(mode=0o777, parents=False) +.. method:: Path.mkdir(mode=0o777, parents=False, exist_ok=False) Create a new directory at this given path. If *mode* is given, it is combined with the process' ``umask`` value to determine the file mode @@ -805,6 +828,16 @@ call fails (for example because the path doesn't exist): If *parents* is false (the default), a missing parent raises :exc:`FileNotFoundError`. + If *exist_ok* is false (the default), an :exc:`FileExistsError` is + raised if the target directory already exists. + + If *exist_ok* is true, :exc:`FileExistsError` exceptions will be + ignored (same behavior as the POSIX ``mkdir -p`` command), but only if the + last path component is not an existing non-directory file. + + .. versionchanged:: 3.5 + The *exist_ok* parameter was added. + .. method:: Path.open(mode='r', buffering=-1, encoding=None, errors=None, newline=None) @@ -824,6 +857,34 @@ call fails (for example because the path doesn't exist): if the file's uid isn't found in the system database. +.. method:: Path.read_bytes() + + Return the binary contents of the pointed-to file as a bytes object:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + .. versionadded:: 3.5 + + +.. method:: Path.read_text(encoding=None, errors=None) + + Return the decoded contents of the pointed-to file as a string:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + The optional parameters have the same meaning as in :func:`open`. + + .. versionadded:: 3.5 + + .. method:: Path.rename(target) Rename this file or directory to the given *target*. *target* can be @@ -884,6 +945,25 @@ call fails (for example because the path doesn't exist): Remove this directory. The directory must be empty. +.. method:: Path.samefile(other_path) + + Return whether this path points to the same file as *other_path*, which + can be either a Path object, or a string. The semantics are similar + to :func:`os.path.samefile` and :func:`os.path.samestat`. + + An :exc:`OSError` can be raised if either file cannot be accessed for some + reason. + + >>> p = Path('spam') + >>> q = Path('eggs') + >>> p.samefile(q) + False + >>> p.samefile('spam') + True + + .. versionadded:: 3.5 + + .. method:: Path.symlink_to(target, target_is_directory=False) Make this path a symbolic link to *target*. Under Windows, @@ -917,3 +997,33 @@ call fails (for example because the path doesn't exist): Remove this file or symbolic link. If the path points to a directory, use :func:`Path.rmdir` instead. + + +.. method:: Path.write_bytes(data) + + Open the file pointed to in bytes mode, write *data* to it, and close the + file:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + An existing file of the same name is overwritten. + + .. versionadded:: 3.5 + + +.. method:: Path.write_text(data, encoding=None, errors=None) + + Open the file pointed to in text mode, write *data* to it, and close the + file:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + .. versionadded:: 3.5 diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 0f48cb188f..a92526f5a3 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -425,7 +425,7 @@ The following types can be pickled: Attempts to pickle unpicklable objects will raise the :exc:`PicklingError` exception; when this happens, an unspecified number of bytes may have already been written to the underlying file. Trying to pickle a highly recursive data -structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be +structure may exceed the maximum recursion depth, a :exc:`RecursionError` will be raised in this case. You can carefully raise this limit with :func:`sys.setrecursionlimit`. @@ -859,7 +859,7 @@ For the simplest code, use the :func:`dump` and :func:`load` functions. :: data = { 'a': [1, 2.0, 3, 4+6j], 'b': ("character string", b"byte string"), - 'c': set([None, True, False]) + 'c': {None, True, False} } with open('data.pickle', 'wb') as f: diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 13ea7b9b33..5d3295db7e 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -58,7 +58,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. class:: ImpLoader(fullname, file, filename, etc) @@ -67,7 +67,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. function:: find_loader(fullname) diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index e27f2ad8b0..679cc6b23c 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -247,6 +247,8 @@ Unix Platforms This is another name for :func:`linux_distribution`. + .. deprecated-removed:: 3.5 3.7 + .. function:: linux_distribution(distname='', version='', id='', supported_dists=('SuSE','debian','redhat','mandrake',...), full_distribution_name=1) Tries to determine the name of the Linux OS distribution name. @@ -263,6 +265,8 @@ Unix Platforms parameters. ``id`` is the item in parentheses after the version number. It is usually the version codename. + .. deprecated-removed:: 3.5 3.7 + .. function:: libc_ver(executable=sys.executable, lib='', version='', chunksize=2048) Tries to determine the libc version against which the file executable (defaults diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst index 45baad96af..8468f4c498 100644 --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -194,6 +194,15 @@ An :class:`POP3` instance has the following methods: the unique id for that message in the form ``'response mesgnum uid``, otherwise result is list ``(response, ['mesgnum uid', ...], octets)``. + +.. method:: POP3.utf8() + + Try to switch to UTF-8 mode. Returns the server response if successful, + raises :class:`error_proto` if not. Specified in :RFC:`6856`. + + .. versionadded:: 3.5 + + .. method:: POP3.stls(context=None) Start a TLS session on the active connection as specified in :rfc:`2595`. diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index c0589a31a9..0b44dc865e 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -235,10 +235,10 @@ In its basic form, :func:`pprint` shows the whole object:: 'classifiers': ['Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2 :: Only'], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -288,10 +288,10 @@ contents):: 'cheesecake_documentation_id': None, 'cheesecake_installability_id': None, 'classifiers': [...], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -323,13 +323,12 @@ cannot be split, the specified width will be exceeded:: 'cheesecake_installability_id': None, 'classifiers': [...], 'description': 'An extensible ' - 'framework for ' - 'Python programming, ' - 'with special ' - 'focus\r\n' - 'on event-based ' - 'network programming ' - 'and multiprotocol ' + 'framework for Python ' + 'programming, with ' + 'special focus\r\n' + 'on event-based network ' + 'programming and ' + 'multiprotocol ' 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', @@ -344,8 +343,8 @@ cannot be split, the specified width will be exceeded:: 'release_url': 'http://pypi.python.org/pypi/Twisted/12.3.0', 'requires_python': None, 'stable_version': None, - 'summary': 'An asynchronous ' - 'networking framework ' - 'written in Python', + 'summary': 'An asynchronous networking ' + 'framework written in ' + 'Python', 'version': '12.3.0'}, 'urls': [{...}, {...}]} diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 2928821d55..959d9b98a8 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -33,7 +33,7 @@ profiling interface: 2. :mod:`profile`, a pure Python module whose interface is imitated by :mod:`cProfile`, but which adds significant overhead to profiled programs. If you're trying to extend the profiler in some way, the task might be easier - with this module. + with this module. Originally designed and written by Jim Roskind. .. note:: diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index bae8450b7c..97f2b20d2a 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -29,9 +29,9 @@ byte-code cache files in the directory containing the source code. .. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1) Compile a source file to byte-code and write out the byte-code cache file. - The source code is loaded from the file name *file*. The byte-code is - written to *cfile*, which defaults to the :PEP:`3147` path, ending in - ``.pyc`` (``.pyo`` if optimization is enabled in the current interpreter). + The source code is loaded from the file name *file*. The byte-code is + written to *cfile*, which defaults to the :pep:`3147`/:pep:`488` path, ending + in ``.pyc``. For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the name of the source file in error messages when @@ -68,7 +68,7 @@ byte-code cache files in the directory containing the source code. .. function:: main(args=None) Compile several source files. The files named in *args* (or on the command - line, if *args* is ``None``) are compiled and the resulting bytecode is + line, if *args* is ``None``) are compiled and the resulting byte-code is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. If ``'-'`` is the only parameter in args, the list of files is taken from @@ -86,4 +86,3 @@ could not be compiled. Module :mod:`compileall` Utilities to compile all Python source files in a directory tree. - diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 680d690052..1cb0935377 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -158,22 +158,32 @@ fully processed by daemon consumer threads. Example of how to wait for enqueued tasks to be completed:: - def worker(): - while True: - item = q.get() - do_work(item) - q.task_done() - - q = Queue() - for i in range(num_worker_threads): - t = Thread(target=worker) - t.daemon = True + def worker(): + while True: + item = q.get() + if item is None: + break + do_work(item) + q.task_done() + + q = queue.Queue() + threads = [] + for i in range(num_worker_threads): + t = threading.Thread(target=worker) t.start() + threads.append(t) - for item in source(): - q.put(item) + for item in source(): + q.put(item) - q.join() # block until all tasks are done + # block until all tasks are done + q.join() + + # stop workers + for i in range(num_worker_threads): + q.put(None) + for t in threads: + t.join() .. seealso:: diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 11dd367f8a..f8b772749f 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -46,8 +46,7 @@ from sources provided by the operating system. .. warning:: The pseudo-random generators of this module should not be used for - security purposes. Use :func:`os.urandom` or :class:`SystemRandom` if - you require a cryptographically secure pseudo-random number generator. + security purposes. Bookkeeping functions: diff --git a/Doc/library/re.rst b/Doc/library/re.rst index c3c8b65d8d..888458449a 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -281,9 +281,7 @@ The special characters are: assertion`. ``(?<=abc)def`` will find a match in ``abcdef``, since the lookbehind will back up 3 characters and check if the contained pattern matches. The contained pattern must only match strings of some fixed length, meaning that - ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Group - references are not supported even if they match strings of some fixed length. - Note that + ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Note that patterns which start with positive lookbehind assertions will not match at the beginning of the string being searched; you will most likely want to use the :func:`search` function rather than the :func:`match` function: @@ -299,12 +297,14 @@ The special characters are: >>> m.group(0) 'egg' + .. versionchanged: 3.5 + Added support for group references of fixed length. + ``(?<!...)`` Matches if the current position in the string is not preceded by a match for ``...``. This is called a :dfn:`negative lookbehind assertion`. Similar to positive lookbehind assertions, the contained pattern must only match strings of - some fixed length and shouldn't contain group references. - Patterns which start with negative lookbehind assertions may + some fixed length. Patterns which start with negative lookbehind assertions may match at the beginning of the string being searched. ``(?(id/name)yes-pattern|no-pattern)`` @@ -438,6 +438,10 @@ three digits in length. .. versionchanged:: 3.3 The ``'\u'`` and ``'\U'`` escape sequences have been added. +.. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. seealso:: @@ -524,7 +528,11 @@ form. current locale. The use of this flag is discouraged as the locale mechanism is very unreliable, and it only handles one "culture" at a time anyway; you should use Unicode matching instead, which is the default in Python 3 - for Unicode (str) patterns. + for Unicode (str) patterns. This flag makes sense only with bytes patterns. + + .. deprecated-removed:: 3.5 3.6 + Deprecated the use of :const:`re.LOCALE` with string patterns or + :const:`re.ASCII`. .. data:: M @@ -625,17 +633,37 @@ form. That way, separator components are always found at the same relative indices within the result list. - Note that *split* will never split a string on an empty pattern match. - For example: + .. note:: + + :func:`split` doesn't currently split a string on an empty pattern match. + For example: - >>> re.split('x*', 'foo') - ['foo'] - >>> re.split("(?m)^$", "foo\n\nbar\n") - ['foo\n\nbar\n'] + >>> re.split('x*', 'axbc') + ['a', 'bc'] + + Even though ``'x*'`` also matches 0 'x' before 'a', between 'b' and 'c', + and after 'c', currently these matches are ignored. The correct behavior + (i.e. splitting on empty matches too and returning ``['', 'a', 'b', 'c', + '']``) will be implemented in future versions of Python, but since this + is a backward incompatible change, a :exc:`FutureWarning` will be raised + in the meanwhile. + + Patterns that can only match empty strings currently never split the + string. Since this doesn't match the expected behavior, a + :exc:`ValueError` will be raised starting from Python 3.5:: + + >>> re.split("^$", "foo\n\nbar\n", flags=re.M) + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ... + ValueError: split() requires a non-empty pattern match. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Splitting on a pattern that could match an empty string now raises + a warning. Patterns that can only match empty strings are now rejected. .. function:: findall(pattern, string, flags=0) @@ -663,7 +691,7 @@ form. *string* is returned unchanged. *repl* can be a string or a function; if it is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a carriage return, and - so forth. Unknown escapes such as ``\j`` are left alone. Backreferences, such + so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by group 6 in the pattern. For example: @@ -705,6 +733,13 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + + .. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. function:: subn(pattern, repl, string, count=0, flags=0) @@ -714,6 +749,9 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + .. function:: escape(string) @@ -730,13 +768,36 @@ form. Clear the regular expression cache. -.. exception:: error +.. exception:: error(msg, pattern=None, pos=None) Exception raised when a string passed to one of the functions here is not a valid regular expression (for example, it might contain unmatched parentheses) or when some other error occurs during compilation or matching. It is never an - error if a string contains no match for a pattern. + error if a string contains no match for a pattern. The error instance has + the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: pattern + + The regular expression pattern. + + .. attribute:: pos + + The index of *pattern* where compilation failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + .. versionchanged:: 3.5 + Added additional attributes. .. _re-objects: @@ -889,6 +950,8 @@ Match objects support the following methods and attributes: (``\g<1>``, ``\g<name>``) are replaced by the contents of the corresponding group. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. .. method:: match.group([group1, ...]) diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index 692310b941..3864f0d225 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -59,6 +59,14 @@ The :mod:`readline` module defines the following functions: Save a readline history file. The default filename is :file:`~/.history`. +.. function:: append_history_file(nelements[, filename]) + + Append the last *nelements* of history to a file. The default filename is + :file:`~/.history`. The file must already exist. + + .. versionadded:: 3.5 + + .. function:: clear_history() Clear the current history. (Note: this function is not available if the @@ -209,6 +217,26 @@ from the user's :envvar:`PYTHONSTARTUP` file. :: This code is actually automatically run when Python is run in :ref:`interactive mode <tut-interactive>` (see :ref:`rlcompleter-config`). +The following example achieves the same goal but supports concurrent interactive +sessions, by only appending the new history. :: + + import atexit + import os + import realine + histfile = os.path.join(os.path.expanduser("~"), ".python_history") + + try: + readline.read_history_file(histfile) + h_len = readline.get_history_length() + except FileNotFoundError: + open(histfile, 'wb').close() + h_len = 0 + + def save(prev_h_len, histfile): + new_h_len = readline.get_history_length() + readline.append_history_file(new_h_len - prev_h_len, histfile) + atexit.register(save, h_len, histfile) + The following example extends the :class:`code.InteractiveConsole` class to support history save/restore. :: @@ -234,4 +262,3 @@ support history save/restore. :: def save_history(self, histfile): readline.write_history_file(histfile) - diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 5334af8ea4..a62dc844e4 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -145,6 +145,13 @@ The module defines the following: library, and does not handle file descriptors that don't originate from WinSock. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + + .. attribute:: PIPE_BUF The minimum number of bytes which can be written without blocking to a pipe @@ -242,6 +249,12 @@ object. returning. If *timeout* is omitted, -1, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _epoll-objects: @@ -322,6 +335,12 @@ Edge and Level Trigger Polling (epoll) Objects Wait for events. timeout in seconds (float) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _poll-objects: @@ -401,6 +420,12 @@ linearly scanned again. :c:func:`select` is O(highest file descriptor), while returning. If *timeout* is omitted, negative, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kqueue-objects: @@ -435,6 +460,12 @@ Kqueue Objects - max_events must be 0 or a positive integer - timeout in seconds (floats possible) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kevent-objects: diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst index 98377c890f..56cfc6bdb7 100644 --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -45,12 +45,13 @@ Classes hierarchy:: +-- SelectSelector +-- PollSelector +-- EpollSelector + +-- DevpollSelector +-- KqueueSelector In the following, *events* is a bitwise mask indicating which I/O events should -be waited for on a given file object. It can be a combination of the constants -below: +be waited for on a given file object. It can be a combination of the modules +constants below: +-----------------------+-----------------------------------------------+ | Constant | Meaning | @@ -158,6 +159,12 @@ below: timeout has elapsed if the current process receives a signal: in this case, an empty list will be returned. + .. versionchanged:: 3.5 + The selector is now retried with a recomputed timeout when interrupted + by a signal if the signal handler did not raise an exception (see + :pep:`475` for the rationale), instead of returning an empty list + of events before the timeout. + .. method:: close() Close the selector. @@ -207,6 +214,16 @@ below: This returns the file descriptor used by the underlying :func:`select.epoll` object. +.. class:: DevpollSelector() + + :func:`select.devpoll`-based selector. + + .. method:: fileno() + + This returns the file descriptor used by the underlying + :func:`select.devpoll` object. + + .. versionadded:: 3.5 .. class:: KqueueSelector() diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 04afe928c8..3b467e01c2 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -191,7 +191,8 @@ Directory and files operations match one of the glob-style *patterns* provided. See the example below. -.. function:: copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False) +.. function:: copytree(src, dst, symlinks=False, ignore=None, \ + copy_function=copy2, ignore_dangling_symlinks=False) Recursively copy an entire directory tree rooted at *src*, returning the destination directory. The destination @@ -282,7 +283,7 @@ Directory and files operations .. versionadded:: 3.3 -.. function:: move(src, dst) +.. function:: move(src, dst, copy_function=copy2) Recursively move a file or directory (*src*) to another location (*dst*) and return the destination. @@ -292,15 +293,26 @@ Directory and files operations be overwritten depending on :func:`os.rename` semantics. If the destination is on the current filesystem, then :func:`os.rename` is - used. Otherwise, *src* is copied (using :func:`shutil.copy2`) to *dst* and - then removed. In case of symlinks, a new symlink pointing to the target of - *src* will be created in or as *dst* and *src* will be removed. + used. Otherwise, *src* is copied to *dst* using *copy_function* and then + removed. In case of symlinks, a new symlink pointing to the target of *src* + will be created in or as *dst* and *src* will be removed. + + If *copy_function* is given, it must be a callable that takes two arguments + *src* and *dst*, and will be used to copy *src* to *dest* if + :func:`os.rename` cannot be used. If the source is a directory, + :func:`copytree` is called, passing it the :func:`copy_function`. The + default *copy_function* is :func:`copy2`. Using :func:`copy` as the + *copy_function* allows the move to succeed when it is not possible to also + copy the metadata, at the expense of not copying any of the metadata. .. versionchanged:: 3.3 Added explicit symlink handling for foreign filesystems, thus adapting it to the behavior of GNU's :program:`mv`. Now returns *dst*. + .. versionchanged:: 3.5 + Added the *copy_function* keyword argument. + .. function:: disk_usage(path) Return disk usage statistics about the given path as a :term:`named tuple` @@ -418,6 +430,26 @@ Another example that uses the *ignore* argument to add a logging call:: copytree(source, destination, ignore=_logpath) +.. _shutil-rmtree-example: + +rmtree example +~~~~~~~~~~~~~~ + +This example shows how to remove a directory tree on Windows where some +of the files have their read-only bit set. It uses the onerror callback +to clear the readonly bit and reattempt the remove. Any subsequent failure +will propagate. :: + + import os, stat + import shutil + + def remove_readonly(func, path, _): + "Clear the readonly bit and reattempt the removal" + os.chmod(path, stat.S_IWRITE) + func(path) + + shutil.rmtree(directory, onerror=remove_readonly) + .. _archiving-operations: Archiving operations @@ -434,7 +466,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. *base_name* is the name of the file to create, including the path, minus any format-specific extension. *format* is the archive format: one of - "zip", "tar", "bztar" (if the :mod:`bz2` module is available) or "gztar". + "zip", "tar", "bztar" (if the :mod:`bz2` module is available), "xztar" + (if the :mod:`lzma` module is available) or "gztar". *root_dir* is a directory that will be the root directory of the archive; for example, we typically chdir into *root_dir* before creating the @@ -457,6 +490,9 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. The *verbose* argument is unused and deprecated. + .. versionchanged:: 3.5 + Added support for the *xztar* format. + .. function:: get_archive_formats() @@ -467,6 +503,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file @@ -542,6 +579,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 8b90994846..8f814df8bc 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -65,6 +65,16 @@ Besides, only the main thread is allowed to set a new signal handler. Module contents --------------- +.. versionchanged:: 3.5 + signal (SIG*), handler (:const:`SIG_DFL`, :const:`SIG_IGN`) and sigmask + (:const:`SIG_BLOCK`, :const:`SIG_UNBLOCK`, :const:`SIG_SETMASK`) + related constants listed below were turned into + :class:`enums <enum.IntEnum>`. + :func:`getsignal`, :func:`pthread_sigmask`, :func:`sigpending` and + :func:`sigwait` functions return human-readable + :class:`enums <enum.IntEnum>`. + + The variables defined in the :mod:`signal` module are: @@ -209,21 +219,21 @@ The :mod:`signal` module defines the following functions: :func:`sigpending`. -.. function:: pthread_kill(thread_id, signum) +.. function:: pthread_kill(thread_id, signalnum) - Send the signal *signum* to the thread *thread_id*, another thread in the + Send the signal *signalnum* to the thread *thread_id*, another thread in the same process as the caller. The target thread can be executing any code (Python or not). However, if the target thread is executing the Python interpreter, the Python signal handlers will be :ref:`executed by the main - thread <signals-and-threads>`. Therefore, the only point of sending a signal to a particular - Python thread would be to force a running system call to fail with - :exc:`InterruptedError`. + thread <signals-and-threads>`. Therefore, the only point of sending a + signal to a particular Python thread would be to force a running system call + to fail with :exc:`InterruptedError`. Use :func:`threading.get_ident()` or the :attr:`~threading.Thread.ident` attribute of :class:`threading.Thread` objects to get a suitable value for *thread_id*. - If *signum* is 0, then no signal is sent, but error checking is still + If *signalnum* is 0, then no signal is sent, but error checking is still performed; this can be used to check if the target thread is still running. Availability: Unix (see the man page :manpage:`pthread_kill(3)` for further @@ -308,6 +318,9 @@ The :mod:`signal` module defines the following functions: attempting to call it from other threads will cause a :exc:`ValueError` exception to be raised. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + .. function:: siginterrupt(signalnum, flag) @@ -395,6 +408,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried if interrupted by a signal not in *sigset* + and the signal handler does not raise an exception (see :pep:`475` for + the rationale). + .. function:: sigtimedwait(sigset, timeout) @@ -409,6 +427,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried with the recomputed *timeout* if interrupted + by a signal not in *sigset* and the signal handler does not raise an + exception (see :pep:`475` for the rationale). + .. _signal-example: diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 51e5da81d1..43daf790b7 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -26,24 +26,23 @@ additions, call the :func:`site.main` function. :option:`-S`. .. index:: - pair: site-python; directory pair: site-packages; directory It starts by constructing up to four directories from a head and a tail part. For the head part, it uses ``sys.prefix`` and ``sys.exec_prefix``; empty heads are skipped. For the tail part, it uses the empty string and then :file:`lib/site-packages` (on Windows) or -:file:`lib/python{X.Y}/site-packages` and then :file:`lib/site-python` (on -Unix and Macintosh). For each of the distinct head-tail combinations, it sees -if it refers to an existing directory, and if so, adds it to ``sys.path`` and -also inspects the newly added path for configuration files. +:file:`lib/python{X.Y}/site-packages` (on Unix and Macintosh). For each +of the distinct head-tail combinations, it sees if it refers to an existing +directory, and if so, adds it to ``sys.path`` and also inspects the newly +added path for configuration files. -.. deprecated:: 3.4 - Support for the "site-python" directory will be removed in 3.5. +.. versionchanged:: 3.5 + Support for the "site-python" directory has been removed. If a file named "pyvenv.cfg" exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and -it is also checked for site-packages and site-python (sys.base_prefix and +it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the "real" prefixes of the Python installation). If "pyvenv.cfg" (a bootstrap configuration file) contains the key "include-system-site-packages" set to anything other than "false" @@ -195,8 +194,7 @@ Module contents .. function:: getsitepackages() - Return a list containing all global site-packages directories (and possibly - site-python). + Return a list containing all global site-packages directories. .. versionadded:: 3.2 diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index 3ebed06260..977f9a8748 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -20,7 +20,8 @@ specific mail-sending strategies. Additionally the SMTPChannel may be extended to implement very specific interaction behaviour with SMTP clients. -The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension. +The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE and :rfc:`6531` +SMTPUTF8 extensions. SMTPServer Objects @@ -28,7 +29,7 @@ SMTPServer Objects .. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432,\ - map=None) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. It @@ -39,25 +40,77 @@ SMTPServer Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. - A dictionary can be specified in *map* to avoid using a global socket map. - - .. method:: process_message(peer, mailfrom, rcpttos, data) - - Raise :exc:`NotImplementedError` exception. Override this in subclasses to + *map* is the socket map to use for connections (an initially empty + dictionary is a suitable value). If not specified the :mod:`asyncore` + global socket map is used. + + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. If set to + ``True``, *decode_data* must be ``False`` (otherwise an error is raised). + When ``True``, ``SMTPUTF8`` is accepted as a parameter to the ``MAIL`` + command and when present is passed to :meth:`process_message` in the + ``kwargs['mail_options']`` list. + + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6; specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. When + *decode_data* is set to ``False`` the server advertises the ``8BITMIME`` + extension (:rfc:`6152`), accepts the ``BODY=8BITMIME`` parameter to + the ``MAIL`` command, and when present passes it to :meth:`process_message` + in the ``kwargs['mail_options']`` list. + + .. method:: process_message(peer, mailfrom, rcpttos, data, **kwargs) + + Raise a :exc:`NotImplementedError` exception. Override this in subclasses to do something useful with this message. Whatever was passed in the constructor as *remoteaddr* will be available as the :attr:`_remoteaddr` attribute. *peer* is the remote host's address, *mailfrom* is the envelope originator, *rcpttos* are the envelope recipients and *data* is a string - containing the contents of the e-mail (which should be in :rfc:`2822` + containing the contents of the e-mail (which should be in :rfc:`5321` format). + If the *decode_data* constructor keyword is set to ``True``, the *data* + argument will be a unicode string. If it is set to ``False``, it + will be a bytes object. + + *kwargs* is a dictionary containing additional information. It is empty + unless at least one of ``decode_data=False`` or ``enable_SMTPUTF8=True`` + was given as an init parameter, in which case it contains the following + keys: + + *mail_options*: + a list of all received parameters to the ``MAIL`` + command (the elements are uppercase strings; example: + ``['BODY=8BITMIME', 'SMTPUTF8']``). + + *rcpt_options*: + same as *mail_options* but for the ``RCPT`` command. + Currently no ``RCPT TO`` options are supported, so for now + this will always be an empty list. + + Implementations of ``process_message`` should use the ``**kwargs`` + signature to accept arbitrary keyword arguments, since future feature + enhancements may add keys to the kwargs dictionary. + + Return ``None`` to request a normal ``250 Ok`` response; otherwise + return the desired response string in :RFC:`5321` format. + .. attribute:: channel_class Override this in subclasses to use a custom :class:`SMTPChannel` for managing SMTP clients. - .. versionchanged:: 3.4 - The *map* argument was added. + .. versionadded:: 3.4 + The *map* constructor argument. + + .. versionchanged:: 3.5 + *localaddr* and *remoteaddr* may now contain IPv6 addresses. + + .. versionadded:: 3.5 + the *decode_data* and *enable_SMTPUTF8* constructor arguments, and the + *kwargs* argument to :meth:`process_message` when one or more of these is + specified. DebuggingServer Objects @@ -97,7 +150,7 @@ SMTPChannel Objects ------------------- .. class:: SMTPChannel(server, conn, addr, data_size_limit=33554432,\ - map=None)) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPChannel` object which manages the communication between the server and a single SMTP client. @@ -108,11 +161,24 @@ SMTPChannel Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. A + :exc:`ValueError` is raised if both *enable_SMTPUTF8* and *decode_data* are + set to ``True`` at the same time. + A dictionary can be specified in *map* to avoid using a global socket map. + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6. Specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. + To use a custom SMTPChannel implementation you need to override the :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. + .. versionchanged:: 3.5 + the *decode_data* and *enable_SMTPUTF8* arguments were added. + The :class:`SMTPChannel` has the following instance variables: .. attribute:: smtp_server diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 8e1bfb57f9..a71ee585a3 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -61,6 +61,10 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). .. versionchanged:: 3.3 source_address argument was added. + .. versionadded:: 3.5 + The SMTPUTF8 extension (:rfc:`6531`) is now supported. + + .. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, \ certfile=None [, timeout], context=None, \ source_address=None) @@ -161,6 +165,13 @@ A nice selection of exceptions is defined as well: The server refused our ``HELO`` message. +.. exception:: SMTPNotSupportedError + + The command or option attempted is not supported by the server. + + .. versionadded:: 3.5 + + .. exception:: SMTPAuthenticationError SMTP authentication went wrong. Most probably the server didn't accept the @@ -189,8 +200,12 @@ An :class:`SMTP` instance has the following methods: .. method:: SMTP.set_debuglevel(level) - Set the debug output level. A true value for *level* results in debug messages - for connection and for all messages sent to and received from the server. + Set the debug output level. A value of 1 or ``True`` for *level* results in + debug messages for connection and for all messages sent to and received from + the server. A value of 2 for *level* results in these messages being + timestamped. + + .. versionchanged:: 3.5 Added debuglevel 2. .. method:: SMTP.docmd(cmd, args='') @@ -240,8 +255,7 @@ An :class:`SMTP` instance has the following methods: the server is stored as the :attr:`ehlo_resp` attribute, :attr:`does_esmtp` is set to true or false depending on whether the server supports ESMTP, and :attr:`esmtp_features` will be a dictionary containing the names of the - SMTP service extensions this server supports, and their - parameters (if any). + SMTP service extensions this server supports, and their parameters (if any). Unless you wish to use :meth:`has_extn` before sending mail, it should not be necessary to call this method explicitly. It will be implicitly called by @@ -274,7 +288,7 @@ An :class:`SMTP` instance has the following methods: Many sites disable SMTP ``VRFY`` in order to foil spammers. -.. method:: SMTP.login(user, password) +.. method:: SMTP.login(user, password, *, initial_response_ok=True) Log in on an SMTP server that requires authentication. The arguments are the username and the password to authenticate with. If there has been no previous @@ -288,9 +302,68 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPAuthenticationError` The server didn't accept the username/password combination. + :exc:`SMTPNotSupportedError` + The ``AUTH`` command is not supported by the server. + :exc:`SMTPException` No suitable authentication method was found. + Each of the authentication methods supported by :mod:`smtplib` are tried in + turn if they are advertised as supported by the server. See :meth:`auth` + for a list of supported authentication methods. *initial_response_ok* is + passed through to :meth:`auth`. + + Optional keyword argument *initial_response_ok* specifies whether, for + authentication methods that support it, an "initial response" as specified + in :rfc:`4954` can be sent along with the ``AUTH`` command, rather than + requiring a challenge/response. + + .. versionchanged:: 3.5 + :exc:`SMTPNotSupportedError` may be raised, and the + *initial_response_ok* parameter was added. + + +.. method:: SMTP.auth(mechanism, authobject, *, initial_response_ok=True) + + Issue an ``SMTP`` ``AUTH`` command for the specified authentication + *mechanism*, and handle the challenge response via *authobject*. + + *mechanism* specifies which authentication mechanism is to + be used as argument to the ``AUTH`` command; the valid values are + those listed in the ``auth`` element of :attr:`esmtp_features`. + + *authobject* must be a callable object taking an optional single argument: + + data = authobject(challenge=None) + + If optional keyword argument *initial_response_ok* is true, + ``authobject()`` will be called first with no argument. It can return the + :rfc:`4954` "initial response" bytes which will be encoded and sent with + the ``AUTH`` command as below. If the ``authobject()`` does not support an + initial response (e.g. because it requires a challenge), it should return + None when called with ``challenge=None``. If *initial_response_ok* is + false, then ``authobject()`` will not be called first with None. + + If the initial response check returns None, or if *initial_response_ok* is + false, ``authobject()`` will be called to process the server's challenge + response; the *challenge* argument it is passed will be a ``bytes``. It + should return ``bytes`` *data* that will be base64 encoded and sent to the + server. + + The ``SMTP`` class provides ``authobjects`` for the ``CRAM-MD5``, ``PLAIN``, + and ``LOGIN`` mechanisms; they are named ``SMTP.auth_cram_md5``, + ``SMTP.auth_plain``, and ``SMTP.auth_login`` respectively. They all require + that the ``user`` and ``password`` properties of the ``SMTP`` instance are + set to appropriate values. + + User code does not normally need to call ``auth`` directly, but can instead + call the :meth:`login` method, which will try each of the above mechanisms + in turn, in the order listed. ``auth`` is exposed to facilitate the + implementation of authentication methods not (or not yet) supported + directly by :mod:`smtplib`. + + .. versionadded:: 3.5 + .. method:: SMTP.starttls(keyfile=None, certfile=None, context=None) @@ -310,7 +383,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPHeloError` The server didn't reply properly to the ``HELO`` greeting. - :exc:`SMTPException` + :exc:`SMTPNotSupportedError` The server does not support the STARTTLS extension. :exc:`RuntimeError` @@ -324,6 +397,11 @@ An :class:`SMTP` instance has the following methods: :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see :data:`~ssl.HAS_SNI`). + .. versionchanged:: 3.5 + The error raised for lack of STARTTLS support is now the + :exc:`SMTPNotSupportedError` subclass instead of the base + :exc:`SMTPException`. + .. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) @@ -360,6 +438,9 @@ An :class:`SMTP` instance has the following methods: recipient that was refused. Each entry contains a tuple of the SMTP error code and the accompanying error message sent by the server. + If ``SMTPUTF8`` is included in *mail_options*, and the server supports it, + *from_addr* and *to_addr* may contain non-ASCII characters. + This method may raise the following exceptions: :exc:`SMTPRecipientsRefused` @@ -378,12 +459,20 @@ An :class:`SMTP` instance has the following methods: The server replied with an unexpected error code (other than a refusal of a recipient). + :exc:`SMTPNotSupportedError` + ``SMTPUTF8`` was given in the *mail_options* but is not supported by the + server. + Unless otherwise noted, the connection will be open even after an exception is raised. .. versionchanged:: 3.2 *msg* may be a byte string. + .. versionchanged:: 3.5 + ``SMTPUTF8`` support added, and :exc:`SMTPNotSupportedError` may be + raised if ``SMTPUTF8`` is specified but the server does not support it. + .. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, \ mail_options=[], rcpt_options=[]) @@ -395,7 +484,7 @@ An :class:`SMTP` instance has the following methods: If *from_addr* is ``None`` or *to_addrs* is ``None``, ``send_message`` fills those arguments with addresses extracted from the headers of *msg* as - specified in :rfc:`2822`\: *from_addr* is set to the :mailheader:`Sender` + specified in :rfc:`5322`\: *from_addr* is set to the :mailheader:`Sender` field if it is present, and otherwise to the :mailheader:`From` field. *to_adresses* combines the values (if any) of the :mailheader:`To`, :mailheader:`Cc`, and :mailheader:`Bcc` fields from *msg*. If exactly one @@ -410,10 +499,18 @@ An :class:`SMTP` instance has the following methods: calls :meth:`sendmail` to transmit the resulting message. Regardless of the values of *from_addr* and *to_addrs*, ``send_message`` does not transmit any :mailheader:`Bcc` or :mailheader:`Resent-Bcc` headers that may appear - in *msg*. + in *msg*. If any of the addresses in *from_addr* and *to_addrs* contain + non-ASCII characters and the server does not advertise ``SMTPUTF8`` support, + an :exc:`SMTPNotSupported` error is raised. Otherwise the ``Message`` is + serialized with a clone of its :mod:`~email.policy` with the + :attr:`~email.policy.EmailPolicy.utf8` attribute set to ``True``, and + ``SMTPUTF8`` and ``BODY=8BITMIME`` are added to *mail_options*. .. versionadded:: 3.2 + .. versionadded:: 3.5 + Support for internationalized addresses (``SMTPUTF8``). + .. method:: SMTP.quit() diff --git a/Doc/library/sndhdr.rst b/Doc/library/sndhdr.rst index f36df68703..f8b5d8b246 100644 --- a/Doc/library/sndhdr.rst +++ b/Doc/library/sndhdr.rst @@ -16,8 +16,9 @@ The :mod:`sndhdr` provides utility functions which attempt to determine the type of sound data which is in a file. When these functions are able to determine -what type of sound data is stored in a file, they return a tuple ``(type, -sampling_rate, channels, frames, bits_per_sample)``. The value for *type* +what type of sound data is stored in a file, they return a +:func:`~collections.namedtuple`, containing five attributes: (``filetype``, +``framerate``, ``nchannels``, ``nframes``, ``sampwidth``). The value for *type* indicates the data type and will be one of the strings ``'aifc'``, ``'aiff'``, ``'au'``, ``'hcom'``, ``'sndr'``, ``'sndt'``, ``'voc'``, ``'wav'``, ``'8svx'``, ``'sb'``, ``'ub'``, or ``'ul'``. The *sampling_rate* will be either the actual @@ -31,13 +32,19 @@ be the sample size in bits or ``'A'`` for A-LAW or ``'U'`` for u-LAW. .. function:: what(filename) Determines the type of sound data stored in the file *filename* using - :func:`whathdr`. If it succeeds, returns a tuple as described above, otherwise + :func:`whathdr`. If it succeeds, returns a namedtuple as described above, otherwise ``None`` is returned. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + .. function:: whathdr(filename) Determines the type of sound data stored in a file based on the file header. - The name of the file is given by *filename*. This function returns a tuple as + The name of the file is given by *filename*. This function returns a namedtuple as described above on success, or ``None``. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 638a77e854..f7b7d3d3b7 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -46,17 +46,20 @@ created. Socket addresses are represented as follows: - The address of an :const:`AF_UNIX` socket bound to a file system node is represented as a string, using the file system encoding and the ``'surrogateescape'`` error handler (see :pep:`383`). An address in - Linux's abstract namespace is returned as a :class:`bytes` object with + Linux's abstract namespace is returned as a :term:`bytes-like object` with an initial null byte; note that sockets in this namespace can communicate with normal file system sockets, so programs intended to run on Linux may need to deal with both types of address. A string or - :class:`bytes` object can be used for either type of address when + bytes-like object can be used for either type of address when passing it as an argument. .. versionchanged:: 3.3 Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 encoding. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + - A pair ``(host, port)`` is used for the :const:`AF_INET` address family, where *host* is a string representing either a hostname in Internet domain notation like ``'daring.cwi.nl'`` or an IPv4 address like ``'100.50.200.5'``, @@ -298,6 +301,18 @@ Constants .. versionadded:: 3.4 +.. data:: CAN_RAW_FD_FRAMES + + Enables CAN FD support in a CAN_RAW socket. This is disabled by default. + This allows your application to send both CAN and CAN FD frames; however, + you one must accept both CAN and CAN FD frames when reading from the socket. + + This constant is documented in the Linux documentation. + + Availability: Linux >= 3.6. + + .. versionadded:: 3.5 + .. data:: AF_RDS PF_RDS SOL_RDS @@ -394,7 +409,6 @@ The following functions all create :ref:`socket objects <socket-objects>`. type, and protocol number. Address family, socket type, and protocol number are as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` if defined on the platform; otherwise, the default is :const:`AF_INET`. - Availability: Unix. The newly created sockets are :ref:`non-inheritable <fd_inheritance>`. @@ -405,6 +419,9 @@ The following functions all create :ref:`socket objects <socket-objects>`. .. versionchanged:: 3.4 The returned sockets are now non-inheritable. + .. versionchanged:: 3.5 + Windows support added. + .. function:: create_connection(address[, timeout[, source_address]]) @@ -651,8 +668,8 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntoa(packed_ip) - Convert a 32-bit packed IPv4 address (a bytes object four characters in - length) to its standard dotted-quad string representation (for example, + Convert a 32-bit packed IPv4 address (a :term:`bytes-like object` four + bytes in length) to its standard dotted-quad string representation (for example, '123.45.67.89'). This is useful when conversing with a program that uses the standard C library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary data this function takes as an @@ -663,6 +680,9 @@ The :mod:`socket` module also offers various network-related services: support IPv6, and :func:`inet_ntop` should be used instead for IPv4/v6 dual stack support. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. function:: inet_pton(address_family, ip_string) @@ -685,22 +705,26 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntop(address_family, packed_ip) - Convert a packed IP address (a bytes object of some number of characters) to its - standard, family-specific string representation (for example, ``'7.10.0.5'`` or - ``'5aef:2b::8'``). :func:`inet_ntop` is useful when a library or network protocol - returns an object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) - or :c:type:`struct in6_addr`. + Convert a packed IP address (a :term:`bytes-like object` of some number of + bytes) to its standard, family-specific string representation (for + example, ``'7.10.0.5'`` or ``'5aef:2b::8'``). + :func:`inet_ntop` is useful when a library or network protocol returns an + object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) or + :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and - :const:`AF_INET6`. If the string *packed_ip* is not the correct length for the - specified address family, :exc:`ValueError` will be raised. A - :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. + :const:`AF_INET6`. If the bytes object *packed_ip* is not the correct + length for the specified address family, :exc:`ValueError` will be raised. + A :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. Availability: Unix (maybe not all platforms), Windows. .. versionchanged:: 3.4 Windows support added + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any @@ -825,6 +849,11 @@ to sockets. .. versionchanged:: 3.4 The socket is now non-inheritable. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.bind(address) @@ -857,6 +886,19 @@ to sockets. Connect to a remote socket at *address*. (The format of *address* depends on the address family --- see above.) + If the connection is interrupted by a signal, the method waits until the + connection completes, or raise a :exc:`socket.timeout` on timeout, if the + signal handler doesn't raise an exception and the socket is blocking or has + a timeout. For non-blocking sockets, the method raises an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal (or the exception raised by the signal handler). + + .. versionchanged:: 3.5 + The method now waits until the connection completes instead of raising an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal, the signal handler doesn't raise an exception and the socket is + blocking or has a timeout (see the :pep:`475` for the rationale). + .. method:: socket.connect_ex(address) @@ -952,12 +994,15 @@ to sockets. On other platforms, the generic :func:`fcntl.fcntl` and :func:`fcntl.ioctl` functions may be used; they accept a socket object as their first argument. -.. method:: socket.listen(backlog) +.. method:: socket.listen([backlog]) - Listen for connections made to the socket. The *backlog* argument specifies the - maximum number of queued connections and should be at least 0; the maximum value - is system-dependent (usually 5), the minimum value is forced to 0. + Enable a server to accept connections. If *backlog* is specified, it must + be at least 0 (if it is lower, it is set to 0); it specifies the number of + unaccepted connections that the system will allow before refusing new + connections. If not specified, a default reasonable value is chosen. + .. versionchanged:: 3.5 + The *backlog* parameter is now optional. .. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, \ errors=None, newline=None) @@ -995,6 +1040,11 @@ to sockets. For best match with hardware and network realities, the value of *bufsize* should be a relatively small power of 2, for example, 4096. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvfrom(bufsize[, flags]) @@ -1004,6 +1054,11 @@ to sockets. :manpage:`recv(2)` for the meaning of the optional argument *flags*; it defaults to zero. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]]) @@ -1070,6 +1125,11 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg_into(buffers[, ancbufsize[, flags]]) @@ -1136,6 +1196,11 @@ to sockets. application needs to attempt delivery of the remaining data. For further information on this topic, consult the :ref:`socket-howto`. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendall(bytes[, flags]) @@ -1146,6 +1211,15 @@ to sockets. success. On error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time data is sent successfuly. + The socket timeout is now the maximum total duration to send all data. + + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendto(bytes, address) socket.sendto(bytes, flags, address) @@ -1156,6 +1230,11 @@ to sockets. bytes sent. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendmsg(buffers[, ancdata[, flags[, address]]]) @@ -1192,6 +1271,26 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + +.. method:: socket.sendfile(file, offset=0, count=None) + + Send a file until EOF is reached by using high-performance + :mod:`os.sendfile` and return the total number of bytes which were sent. + *file* must be a regular file object opened in binary mode. If + :mod:`os.sendfile` is not available (e.g. Windows) or *file* is not a + regular file :meth:`send` will be used instead. *offset* tells from where to + start reading the file. If specified, *count* is the total number of bytes + to transmit as opposed to sending the file until EOF is reached. File + position is updated on return or also in case of error in which case + :meth:`file.tell() <io.IOBase.tell>` can be used to figure out the number of + bytes which were sent. The socket must be of :const:`SOCK_STREAM` type. Non- + blocking sockets are not supported. + + .. versionadded:: 3.5 .. method:: socket.set_inheritable(inheritable) @@ -1231,11 +1330,15 @@ to sockets. Set the value of the given socket option (see the Unix manual page :manpage:`setsockopt(2)`). The needed symbolic constants are defined in the - :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or a - bytes object representing a buffer. In the latter case it is up to the caller to + :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or + a :term:`bytes-like object` representing a buffer. In the latter case it is + up to the caller to ensure that the bytestring contains the proper bits (see the optional built-in module :mod:`struct` for a way to encode C structures as bytestrings). + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: socket.shutdown(how) diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 3e49af6bba..18be936b08 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -114,7 +114,7 @@ the request handler class :meth:`handle` method. Another approach to handling multiple simultaneous requests in an environment that supports neither threads nor :func:`~os.fork` (or where these are too expensive or inappropriate for the service) is to maintain an explicit table of -partially finished requests and to use :func:`~select.select` to decide which +partially finished requests and to use :mod:`selectors` to decide which request to work on next (or whether to handle a new incoming request). This is particularly important for stream services where each client can potentially be connected for a long time (if threads or subprocesses cannot be used). See @@ -137,7 +137,7 @@ Server Objects .. method:: BaseServer.fileno() Return an integer file descriptor for the socket on which the server is - listening. This function is most commonly passed to :func:`select.select`, to + listening. This function is most commonly passed to :mod:`selectors`, to allow monitoring multiple servers in the same process. diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 6097e7a30d..fc69a804d5 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -649,6 +649,9 @@ Row Objects This method returns a list of column names. Immediately after a query, it is the first member of each tuple in :attr:`Cursor.description`. + .. versionchanged:: 3.5 + Added support of slicing. + Let's assume we initialize a table as in the example given above:: conn = sqlite3.connect(":memory:") diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 233de8dd91..40751347ae 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -315,6 +315,8 @@ Random generation For almost all applications :func:`os.urandom` is preferable. + For almost all applications :func:`os.urandom` is preferable. + .. versionadded:: 3.3 .. function:: RAND_status() @@ -335,6 +337,8 @@ Random generation See http://egd.sourceforge.net/ or http://prngd.sourceforge.net/ for sources of entropy-gathering daemons. + Availability: not available with LibreSSL. + .. function:: RAND_add(bytes, entropy) Mix the given *bytes* into the SSL pseudo-random number generator. The @@ -342,6 +346,9 @@ Random generation string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + Certificate handling ^^^^^^^^^^^^^^^^^^^^ @@ -350,10 +357,9 @@ Certificate handling Verify that *cert* (in decoded format as returned by :meth:`SSLSocket.getpeercert`) matches the given *hostname*. The rules applied are those for checking the identity of HTTPS servers as outlined - in :rfc:`2818` and :rfc:`6125`, except that IP addresses are not currently - supported. In addition to HTTPS, this function should be suitable for - checking the identity of servers in various SSL-based protocols such as - FTPS, IMAPS, POPS and others. + in :rfc:`2818` and :rfc:`6125`. In addition to HTTPS, this function + should be suitable for checking the identity of servers in various + SSL-based protocols such as FTPS, IMAPS, POPS and others. :exc:`CertificateError` is raised on failure. On success, the function returns nothing:: @@ -375,22 +381,38 @@ Certificate handling IDN A-labels such as ``www*.xn--pthon-kva.org`` are still supported, but ``x*.python.org`` no longer matches ``xn--tda.python.org``. -.. function:: cert_time_to_seconds(timestring) + .. versionchanged:: 3.5 + Matching of IP addresses, when present in the subjectAltName field + of the certificate, is now supported. + +.. function:: cert_time_to_seconds(cert_time) - Returns a floating-point value containing a normal seconds-after-the-epoch - time value, given the time-string representing the "notBefore" or "notAfter" - date from a certificate. + Return the time in seconds since the Epoch, given the ``cert_time`` + string representing the "notBefore" or "notAfter" date from a + certificate in ``"%b %d %H:%M:%S %Y %Z"`` strptime format (C + locale). - Here's an example:: + Here's an example: - >>> import ssl - >>> ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT") - 1178694000.0 - >>> import time - >>> time.ctime(ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT")) - 'Wed May 9 00:00:00 2007' + .. doctest:: newcontext -.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv3, ca_certs=None) + >>> import ssl + >>> timestamp = ssl.cert_time_to_seconds("Jan 5 09:34:43 2018 GMT") + >>> timestamp + 1515144883 + >>> from datetime import datetime + >>> print(datetime.utcfromtimestamp(timestamp)) + 2018-01-05 09:34:43 + + "notBefore" or "notAfter" dates must use GMT (:rfc:`5280`). + + .. versionchanged:: 3.5 + Interpret the input time as a time in UTC as specified by 'GMT' + timezone in the input string. Local timezone was used + previously. Return an integer (no fractions of a second in the + input format) + +.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv23, ca_certs=None) Given the address ``addr`` of an SSL-protected server, as a (*hostname*, *port-number*) pair, fetches the server's certificate, and returns it as a @@ -404,6 +426,10 @@ Certificate handling .. versionchanged:: 3.3 This function is now IPv6-compatible. + .. versionchanged:: 3.5 + The default *ssl_version* is changed from :data:`PROTOCOL_SSLv3` to + :data:`PROTOCOL_SSLv23` for maximum compatibility with modern servers. + .. function:: DER_cert_to_PEM_cert(DER_cert_bytes) Given a certificate as a DER-encoded blob of bytes, returns a PEM-encoded @@ -671,6 +697,13 @@ Constants .. versionadded:: 3.3 +.. data:: HAS_ALPN + + Whether the OpenSSL library has built-in support for the *Application-Layer + Protocol Negotiation* TLS extension as described in :rfc:`7301`. + + .. versionadded:: 3.5 + .. data:: HAS_ECDH Whether the OpenSSL library has built-in support for Elliptic Curve-based @@ -788,6 +821,8 @@ SSL Sockets (but passing a non-zero ``flags`` argument is not allowed) - :meth:`~socket.socket.send()`, :meth:`~socket.socket.sendall()` (with the same limitation) + - :meth:`~socket.socket.sendfile()` (but :mod:`os.sendfile` will be used + for plain-text sockets only, else :meth:`~socket.socket.send()` will be used) - :meth:`~socket.socket.shutdown()` However, since the SSL (and TLS) protocol has its own framing atop @@ -798,6 +833,15 @@ SSL Sockets Usually, :class:`SSLSocket` are not created directly, but using the :func:`wrap_socket` function or the :meth:`SSLContext.wrap_socket` method. + .. versionchanged:: 3.5 + The :meth:`sendfile` method was added. + + .. versionchanged:: 3.5 + The :meth:`shutdown` does not reset the socket timeout each time bytes + are received or sent. The socket timeout is now to maximum total duration + of the shutdown. + + SSL sockets also have the following additional methods and attributes: .. method:: SSLSocket.read(len=0, buffer=None) @@ -812,6 +856,11 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`read` can also cause write operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to read up to *len* + bytes. + .. method:: SSLSocket.write(buf) Write *buf* to the SSL socket and return the number of bytes written. The @@ -823,6 +872,10 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`write` can also cause read operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to write *buf*. + .. note:: The :meth:`~SSLSocket.read` and :meth:`~SSLSocket.write` methods are the @@ -844,6 +897,10 @@ SSL sockets also have the following additional methods and attributes: :attr:`~SSLContext.check_hostname` attribute of the socket's :attr:`~SSLSocket.context` is true. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration of the handshake. + .. method:: SSLSocket.getpeercert(binary_form=False) If there is no certificate for the peer on the other end of the connection, @@ -917,6 +974,17 @@ SSL sockets also have the following additional methods and attributes: version of the SSL protocol that defines its use, and the number of secret bits being used. If no connection has been established, returns ``None``. +.. method:: SSLSocket.shared_ciphers() + + Return the list of ciphers shared by the client during the handshake. Each + entry of the returned list is a three-value tuple containing the name of the + cipher, the version of the SSL protocol that defines its use, and the number + of secret bits the cipher uses. :meth:`~SSLSocket.shared_ciphers` returns + ``None`` if no connection has been established or the socket is a client + socket. + + .. versionadded:: 3.5 + .. method:: SSLSocket.compression() Return the compression algorithm being used as a string, or ``None`` @@ -940,12 +1008,22 @@ SSL sockets also have the following additional methods and attributes: .. versionadded:: 3.3 +.. method:: SSLSocket.selected_alpn_protocol() + + Return the protocol that was selected during the TLS handshake. If + :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does + not support ALPN, if this socket does not support any of the client's + proposed protocols, or if the handshake has not happened yet, ``None`` is + returned. + + .. versionadded:: 3.5 + .. method:: SSLSocket.selected_npn_protocol() - Returns the protocol that was selected during the TLS/SSL handshake. If - :meth:`SSLContext.set_npn_protocols` was not called, or if the other party - does not support NPN, or if the handshake has not yet happened, this will - return ``None``. + Return the higher-level protocol that was selected during the TLS/SSL + handshake. If :meth:`SSLContext.set_npn_protocols` was not called, or + if the other party does not support NPN, or if the handshake has not yet + happened, this will return ``None``. .. versionadded:: 3.3 @@ -957,6 +1035,16 @@ SSL sockets also have the following additional methods and attributes: returned socket should always be used for further communication with the other side of the connection, rather than the original socket. +.. method:: SSLSocket.version() + + Return the actual SSL protocol version negotiated by the connection + as a string, or ``None`` is no secure connection is established. + As of this writing, possible return values include ``"SSLv2"``, + ``"SSLv3"``, ``"TLSv1"``, ``"TLSv1.1"`` and ``"TLSv1.2"``. + Recent OpenSSL versions may define more return values. + + .. versionadded:: 3.5 + .. method:: SSLSocket.pending() Returns the number of already decrypted bytes available for read, pending on @@ -1135,6 +1223,20 @@ to speed up repeated connections from the same clients. when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_alpn_protocols(protocols) + + Specify which protocols the socket should advertise during the SSL/TLS + handshake. It should be a list of ASCII strings, like ``['http/1.1', + 'spdy/2']``, ordered by preference. The selection of a protocol will happen + during the handshake, and will play out according to :rfc:`7301`. After a + successful handshake, the :meth:`SSLSocket.selected_alpn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is + False. + + .. versionadded:: 3.5 + .. method:: SSLContext.set_npn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS @@ -1175,7 +1277,7 @@ to speed up repeated connections from the same clients. Due to the early negotiation phase of the TLS connection, only limited methods and attributes are usable like - :meth:`SSLSocket.selected_npn_protocol` and :attr:`SSLSocket.context`. + :meth:`SSLSocket.selected_alpn_protocol` and :attr:`SSLSocket.context`. :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.cipher` and :meth:`SSLSocket.compress` methods require that the TLS connection has progressed beyond the TLS Client Hello and therefore @@ -1251,10 +1353,20 @@ to speed up repeated connections from the same clients. quite similarly to HTTP virtual hosts. Specifying *server_hostname* will raise a :exc:`ValueError` if *server_side* is true. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Always allow a server_hostname to be passed, even if OpenSSL does not have SNI. +.. method:: SSLContext.wrap_bio(incoming, outgoing, server_side=False, \ + server_hostname=None) + + Create a new :class:`SSLObject` instance by wrapping the BIO objects + *incoming* and *outgoing*. The SSL routines will read input data from the + incoming BIO and write data to the outgoing BIO. + + The *server_side* and *server_hostname* parameters have the same meaning as + in :meth:`SSLContext.wrap_socket`. + .. method:: SSLContext.session_stats() Get statistics about the SSL sessions created or managed by this context. @@ -1627,7 +1739,7 @@ are finished with the client (or the client is finished with you):: And go back to listening for new client connections (of course, a real server would probably handle each client connection in a separate thread, or put -the sockets in non-blocking mode and use an event loop). +the sockets in :ref:`non-blocking mode <ssl-nonblocking>` and use an event loop). .. _ssl-nonblocking: @@ -1649,6 +1761,12 @@ thus several things you need to be aware of: socket first, and attempts to *read* from the SSL socket may require a prior *write* to the underlying socket. + .. versionchanged:: 3.5 + + In earlier Python versions, the :meth:`!SSLSocket.send` method + returned zero instead of raising :exc:`SSLWantWriteError` or + :exc:`SSLWantReadError`. + - Calling :func:`~select.select` tells you that the OS-level socket can be read from (or written to), but it does not imply that there is sufficient data at the upper SSL layer. For example, only part of an SSL frame might @@ -1681,13 +1799,143 @@ thus several things you need to be aware of: .. seealso:: - The :mod:`asyncio` module supports non-blocking SSL sockets and provides a + The :mod:`asyncio` module supports :ref:`non-blocking SSL sockets + <ssl-nonblocking>` and provides a higher level API. It polls for events using the :mod:`selectors` module and handles :exc:`SSLWantWriteError`, :exc:`SSLWantReadError` and :exc:`BlockingIOError` exceptions. It runs the SSL handshake asynchronously as well. +Memory BIO Support +------------------ + +.. versionadded:: 3.5 + +Ever since the SSL module was introduced in Python 2.6, the :class:`SSLSocket` +class has provided two related but distinct areas of functionality: + +- SSL protocol handling +- Network IO + +The network IO API is identical to that provided by :class:`socket.socket`, +from which :class:`SSLSocket` also inherits. This allows an SSL socket to be +used as a drop-in replacement for a regular socket, making it very easy to add +SSL support to an existing application. + +Combining SSL protocol handling and network IO usually works well, but there +are some cases where it doesn't. An example is async IO frameworks that want to +use a different IO multiplexing model than the "select/poll on a file +descriptor" (readiness based) model that is assumed by :class:`socket.socket` +and by the internal OpenSSL socket IO routines. This is mostly relevant for +platforms like Windows where this model is not efficient. For this purpose, a +reduced scope variant of :class:`SSLSocket` called :class:`SSLObject` is +provided. + +.. class:: SSLObject + + A reduced-scope variant of :class:`SSLSocket` representing an SSL protocol + instance that does not contain any network IO methods. This class is + typically used by framework authors that want to implement asynchronous IO + for SSL through memory buffers. + + This class implements an interface on top of a low-level SSL object as + implemented by OpenSSL. This object captures the state of an SSL connection + but does not provide any network IO itself. IO needs to be performed through + separate "BIO" objects which are OpenSSL's IO abstraction layer. + + An :class:`SSLObject` instance can be created using the + :meth:`~SSLContext.wrap_bio` method. This method will create the + :class:`SSLObject` instance and bind it to a pair of BIOs. The *incoming* + BIO is used to pass data from Python to the SSL protocol instance, while the + *outgoing* BIO is used to pass data the other way around. + + The following methods are available: + + - :attr:`~SSLSocket.context` + - :attr:`~SSLSocket.server_side` + - :attr:`~SSLSocket.server_hostname` + - :meth:`~SSLSocket.read` + - :meth:`~SSLSocket.write` + - :meth:`~SSLSocket.getpeercert` + - :meth:`~SSLSocket.selected_npn_protocol` + - :meth:`~SSLSocket.cipher` + - :meth:`~SSLSocket.shared_ciphers` + - :meth:`~SSLSocket.compression` + - :meth:`~SSLSocket.pending` + - :meth:`~SSLSocket.do_handshake` + - :meth:`~SSLSocket.unwrap` + - :meth:`~SSLSocket.get_channel_binding` + + When compared to :class:`SSLSocket`, this object lacks the following + features: + + - Any form of network IO incluging methods such as ``recv()`` and + ``send()``. + + - There is no *do_handshake_on_connect* machinery. You must always manually + call :meth:`~SSLSocket.do_handshake` to start the handshake. + + - There is no handling of *suppress_ragged_eofs*. All end-of-file conditions + that are in violation of the protocol are reported via the + :exc:`SSLEOFError` exception. + + - The method :meth:`~SSLSocket.unwrap` call does not return anything, + unlike for an SSL socket where it returns the underlying socket. + + - The *server_name_callback* callback passed to + :meth:`SSLContext.set_servername_callback` will get an :class:`SSLObject` + instance instead of a :class:`SSLSocket` instance as its first parameter. + + Some notes related to the use of :class:`SSLObject`: + + - All IO on an :class:`SSLObject` is :ref:`non-blocking <ssl-nonblocking>`. + This means that for example :meth:`~SSLSocket.read` will raise an + :exc:`SSLWantReadError` if it needs more data than the incoming BIO has + available. + + - There is no module-level ``wrap_bio()`` call like there is for + :meth:`~SSLContext.wrap_socket`. An :class:`SSLObject` is always created + via an :class:`SSLContext`. + +An SSLObject communicates with the outside world using memory buffers. The +class :class:`MemoryBIO` provides a memory buffer that can be used for this +purpose. It wraps an OpenSSL memory BIO (Basic IO) object: + +.. class:: MemoryBIO + + A memory buffer that can be used to pass data between Python and an SSL + protocol instance. + + .. attribute:: MemoryBIO.pending + + Return the number of bytes currently in the memory buffer. + + .. attribute:: MemoryBIO.eof + + A boolean indicating whether the memory BIO is current at the end-of-file + position. + + .. method:: MemoryBIO.read(n=-1) + + Read up to *n* bytes from the memory buffer. If *n* is not specified or + negative, all bytes are returned. + + .. method:: MemoryBIO.write(buf) + + Write the bytes from *buf* to the memory BIO. The *buf* argument must be an + object supporting the buffer protocol. + + The return value is the number of bytes written, which is always equal to + the length of *buf*. + + .. method:: MemoryBIO.write_eof() + + Write an EOF marker to the memory BIO. After this method has been called, it + is illegal to call :meth:`~MemoryBIO.write`. The attribute :attr:`eof` will + become true after all data currently in the buffer has been read. + + .. _ssl-security: Security considerations diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 24769f6893..845b2ef7da 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -126,7 +126,7 @@ Example:: if __name__ == '__main__': walktree(sys.argv[1], visitfile) -An additional utility function is provided to covert a file's mode in a human +An additional utility function is provided to convert a file's mode in a human readable string: .. function:: filemode(mode) @@ -399,3 +399,29 @@ The following flags can be used in the *flags* argument of :func:`os.chflags`: The file is a snapshot file. See the \*BSD or Mac OS systems man page :manpage:`chflags(2)` for more information. + +On Windows, the following file attribute constants are available for use when +testing bits in the ``st_file_attributes`` member returned by :func:`os.stat`. +See the `Windows API documentation +<http://msdn.microsoft.com/en-us/library/windows/desktop/gg258117.aspx>`_ +for more detail on the meaning of these constants. + +.. data:: FILE_ATTRIBUTE_ARCHIVE + FILE_ATTRIBUTE_COMPRESSED + FILE_ATTRIBUTE_DEVICE + FILE_ATTRIBUTE_DIRECTORY + FILE_ATTRIBUTE_ENCRYPTED + FILE_ATTRIBUTE_HIDDEN + FILE_ATTRIBUTE_INTEGRITY_STREAM + FILE_ATTRIBUTE_NORMAL + FILE_ATTRIBUTE_NOT_CONTENT_INDEXED + FILE_ATTRIBUTE_NO_SCRUB_DATA + FILE_ATTRIBUTE_OFFLINE + FILE_ATTRIBUTE_READONLY + FILE_ATTRIBUTE_REPARSE_POINT + FILE_ATTRIBUTE_SPARSE_FILE + FILE_ATTRIBUTE_SYSTEM + FILE_ATTRIBUTE_TEMPORARY + FILE_ATTRIBUTE_VIRTUAL + + .. versionadded:: 3.5 diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index ce3c5320d8..f274edb880 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -354,7 +354,7 @@ Notes: The numeric literals accepted include the digits ``0`` to ``9`` or any Unicode equivalent (code points with the ``Nd`` property). - See http://www.unicode.org/Public/6.3.0/ucd/extracted/DerivedNumericType.txt + See http://www.unicode.org/Public/8.0.0/ucd/extracted/DerivedNumericType.txt for a complete list of code points with the ``Nd`` property. @@ -1950,6 +1950,16 @@ expression support in the :mod:`re` module). >>> 'www.example.com'.strip('cmowz.') 'example' + The outermost leading and trailing *chars* argument values are stripped + from the string. Characters are removed from the leading end until + reaching a string character that is not contained in the set of + characters in *chars*. A similar action takes place on the trailing end. + For example:: + + >>> comment_string = '#....... Section 3.2.1 Issue #32 .......' + >>> comment_string.strip('.#! ') + 'Section 3.2.1 Issue #32' + .. method:: str.swapcase() @@ -2303,6 +2313,19 @@ the bytes type has an additional class method to read data in that format: >>> bytes.fromhex('2Ef0 F1f2 ') b'.\xf0\xf1\xf2' +A reverse conversion function exists to transform a bytes object into its +hexadecimal representation. + +.. method:: bytes.hex() + + Return a string object containing two hexadecimal digits for each + byte in the instance. + + >>> b'\xf0\xf1\xf2'.hex() + 'f0f1f2' + + .. versionadded:: 3.5 + Since bytes objects are sequences of integers (akin to a tuple), for a bytes object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytes object of length 1. (This contrasts with text strings, where both indexing @@ -2358,6 +2381,19 @@ the bytearray type has an additional class method to read data in that format: >>> bytearray.fromhex('2Ef0 F1f2 ') bytearray(b'.\xf0\xf1\xf2') +A reverse conversion function exists to transform a bytearray object into its +hexadecimal representation. + +.. method:: bytearray.hex() + + Return a string object containing two hexadecimal digits for each + byte in the instance. + + >>> bytearray(b'\xf0\xf1\xf2').hex() + 'f0f1f2' + + .. versionadded:: 3.5 + Since bytearray objects are sequences of integers (akin to a list), for a bytearray object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytearray object of length 1. (This contrasts with text strings, where @@ -3103,6 +3139,203 @@ place, and instead produce new objects. always produces a new object, even if no changes were made. +.. _bytes-formatting: + +``printf``-style Bytes Formatting +---------------------------------- + +.. index:: + single: formatting, bytes (%) + single: formatting, bytearray (%) + single: interpolation, bytes (%) + single: interpolation, bytearray (%) + single: bytes; formatting + single: bytearray; formatting + single: bytes; interpolation + single: bytearray; interpolation + single: printf-style formatting + single: sprintf-style formatting + single: % formatting + single: % interpolation + +.. note:: + + The formatting operations described here exhibit a variety of quirks that + lead to a number of common errors (such as failing to display tuples and + dictionaries correctly). If the value being printed may be a tuple or + dictionary, wrap it in a tuple. + +Bytes objects (``bytes``/``bytearray``) have one unique built-in operation: +the ``%`` operator (modulo). +This is also known as the bytes *formatting* or *interpolation* operator. +Given ``format % values`` (where *format* is a bytes object), ``%`` conversion +specifications in *format* are replaced with zero or more elements of *values*. +The effect is similar to using the :c:func:`sprintf` in the C language. + +If *format* requires a single argument, *values* may be a single non-tuple +object. [5]_ Otherwise, *values* must be a tuple with exactly the number of +items specified by the format bytes object, or a single mapping object (for +example, a dictionary). + +A conversion specifier contains two or more characters and has the following +components, which must occur in this order: + +#. The ``'%'`` character, which marks the start of the specifier. + +#. Mapping key (optional), consisting of a parenthesised sequence of characters + (for example, ``(somename)``). + +#. Conversion flags (optional), which affect the result of some conversion + types. + +#. Minimum field width (optional). If specified as an ``'*'`` (asterisk), the + actual width is read from the next element of the tuple in *values*, and the + object to convert comes after the minimum field width and optional precision. + +#. Precision (optional), given as a ``'.'`` (dot) followed by the precision. If + specified as ``'*'`` (an asterisk), the actual precision is read from the next + element of the tuple in *values*, and the value to convert comes after the + precision. + +#. Length modifier (optional). + +#. Conversion type. + +When the right argument is a dictionary (or other mapping type), then the +formats in the bytes object *must* include a parenthesised mapping key into that +dictionary inserted immediately after the ``'%'`` character. The mapping key +selects the value to be formatted from the mapping. For example: + + >>> print(b'%(language)s has %(number)03d quote types.' % + ... {b'language': b"Python", b"number": 2}) + b'Python has 002 quote types.' + +In this case no ``*`` specifiers may occur in a format (since they require a +sequential parameter list). + +The conversion flag characters are: + ++---------+---------------------------------------------------------------------+ +| Flag | Meaning | ++=========+=====================================================================+ +| ``'#'`` | The value conversion will use the "alternate form" (where defined | +| | below). | ++---------+---------------------------------------------------------------------+ +| ``'0'`` | The conversion will be zero padded for numeric values. | ++---------+---------------------------------------------------------------------+ +| ``'-'`` | The converted value is left adjusted (overrides the ``'0'`` | +| | conversion if both are given). | ++---------+---------------------------------------------------------------------+ +| ``' '`` | (a space) A blank should be left before a positive number (or empty | +| | string) produced by a signed conversion. | ++---------+---------------------------------------------------------------------+ +| ``'+'`` | A sign character (``'+'`` or ``'-'``) will precede the conversion | +| | (overrides a "space" flag). | ++---------+---------------------------------------------------------------------+ + +A length modifier (``h``, ``l``, or ``L``) may be present, but is ignored as it +is not necessary for Python -- so e.g. ``%ld`` is identical to ``%d``. + +The conversion types are: + ++------------+-----------------------------------------------------+-------+ +| Conversion | Meaning | Notes | ++============+=====================================================+=======+ +| ``'d'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'i'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'o'`` | Signed octal value. | \(1) | ++------------+-----------------------------------------------------+-------+ +| ``'u'`` | Obsolete type -- it is identical to ``'d'``. | \(8) | ++------------+-----------------------------------------------------+-------+ +| ``'x'`` | Signed hexadecimal (lowercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'X'`` | Signed hexadecimal (uppercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'e'`` | Floating point exponential format (lowercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'E'`` | Floating point exponential format (uppercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'f'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'F'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'g'`` | Floating point format. Uses lowercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'G'`` | Floating point format. Uses uppercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'c'`` | Single byte (accepts integer or single | | +| | byte objects). | | ++------------+-----------------------------------------------------+-------+ +| ``'b'`` | Bytes (any object that follows the | \(5) | +| | :ref:`buffer protocol <bufferobjects>` or has | | +| | :meth:`__bytes__`). | | ++------------+-----------------------------------------------------+-------+ +| ``'s'`` | ``'s'`` is an alias for ``'b'`` and should only | \(6) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'a'`` | Bytes (converts any Python object using | \(5) | +| | ``repr(obj).encode('ascii','backslashreplace)``). | | ++------------+-----------------------------------------------------+-------+ +| ``'r'`` | ``'r'`` is an alias for ``'a'`` and should only | \(7) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'%'`` | No argument is converted, results in a ``'%'`` | | +| | character in the result. | | ++------------+-----------------------------------------------------+-------+ + +Notes: + +(1) + The alternate form causes a leading zero (``'0'``) to be inserted between + left-hand padding and the formatting of the number if the leading character + of the result is not already a zero. + +(2) + The alternate form causes a leading ``'0x'`` or ``'0X'`` (depending on whether + the ``'x'`` or ``'X'`` format was used) to be inserted between left-hand padding + and the formatting of the number if the leading character of the result is not + already a zero. + +(3) + The alternate form causes the result to always contain a decimal point, even if + no digits follow it. + + The precision determines the number of digits after the decimal point and + defaults to 6. + +(4) + The alternate form causes the result to always contain a decimal point, and + trailing zeroes are not removed as they would otherwise be. + + The precision determines the number of significant digits before and after the + decimal point and defaults to 6. + +(5) + If precision is ``N``, the output is truncated to ``N`` characters. + +(6) + ``b'%s'`` is deprecated, but will not be removed during the 3.x series. + +(7) + ``b'%r'`` is deprecated, but will not be removed during the 3.x series. + +(8) + See :pep:`237`. + +.. note:: + + The bytearray version of this method does *not* operate in place - it + always produces a new object, even if no changes were made. + +.. seealso:: :pep:`461`. +.. versionadded:: 3.5 + .. _typememoryview: Memory Views @@ -3131,10 +3364,8 @@ copying. the view. The :class:`~memoryview.itemsize` attribute will give you the number of bytes in a single element. - A :class:`memoryview` supports slicing to expose its data. If - :class:`~memoryview.format` is one of the native format specifiers - from the :mod:`struct` module, indexing will return a single element - with the correct type. Full slicing will result in a subview:: + A :class:`memoryview` supports slicing and indexing to expose its data. + One-dimensional slicing will result in a subview:: >>> v = memoryview(b'abcefg') >>> v[1] @@ -3146,25 +3377,29 @@ copying. >>> bytes(v[1:4]) b'bce' - Other native formats:: + If :class:`~memoryview.format` is one of the native format specifiers + from the :mod:`struct` module, indexing with an integer or a tuple of + integers is also supported and returns a single *element* with + the correct type. One-dimensional memoryviews can be indexed + with an integer or a one-integer tuple. Multi-dimensional memoryviews + can be indexed with tuples of exactly *ndim* integers where *ndim* is + the number of dimensions. Zero-dimensional memoryviews can be indexed + with the empty tuple. + + Here is an example with a non-byte format:: >>> import array >>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444]) - >>> a[0] + >>> m = memoryview(a) + >>> m[0] -11111111 - >>> a[-1] + >>> m[-1] 44444444 - >>> a[2:3].tolist() - [-33333333] - >>> a[::2].tolist() + >>> m[::2].tolist() [-11111111, -33333333] - >>> a[::-1].tolist() - [44444444, -33333333, 22222222, -11111111] - - .. versionadded:: 3.3 - If the underlying object is writable, the memoryview supports slice - assignment. Resizing is not allowed:: + If the underlying object is writable, the memoryview supports + one-dimensional slice assignment. Resizing is not allowed:: >>> data = bytearray(b'abcefg') >>> v = memoryview(data) @@ -3197,12 +3432,16 @@ copying. True .. versionchanged:: 3.3 + One-dimensional memoryviews can now be sliced. One-dimensional memoryviews with formats 'B', 'b' or 'c' are now hashable. .. versionchanged:: 3.4 memoryview is now registered automatically with :class:`collections.abc.Sequence` + .. versionchanged:: 3.5 + memoryviews can now be indexed with tuple of integers. + :class:`memoryview` has several methods: .. method:: __eq__(exporter) @@ -3269,6 +3508,17 @@ copying. supports all format strings, including those that are not in :mod:`struct` module syntax. + .. method:: hex() + + Return a string object containing two hexadecimal digits for each + byte in the buffer. :: + + >>> m = memoryview(b"abc") + >>> m.hex() + '616263' + + .. versionadded:: 3.5 + .. method:: tolist() Return the data in the buffer as a list of elements. :: @@ -3324,10 +3574,10 @@ copying. Cast a memoryview to a new format or shape. *shape* defaults to ``[byte_length//new_itemsize]``, which means that the result view will be one-dimensional. The return value is a new memoryview, but - the buffer itself is not copied. Supported casts are 1D -> C-contiguous + the buffer itself is not copied. Supported casts are 1D -> C-:term:`contiguous` and C-contiguous -> 1D. - Both formats are restricted to single element native formats in + The destination format is restricted to a single element native format in :mod:`struct` syntax. One of the formats must be a byte format ('B', 'b' or 'c'). The byte length of the result must be the same as the original length. @@ -3408,6 +3658,9 @@ copying. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The source format is no longer restricted when casting to a byte view. + There are also several readonly attributes available: .. attribute:: obj @@ -3512,19 +3765,19 @@ copying. .. attribute:: c_contiguous - A bool indicating whether the memory is C-contiguous. + A bool indicating whether the memory is C-:term:`contiguous`. .. versionadded:: 3.3 .. attribute:: f_contiguous - A bool indicating whether the memory is Fortran contiguous. + A bool indicating whether the memory is Fortran :term:`contiguous`. .. versionadded:: 3.3 .. attribute:: contiguous - A bool indicating whether the memory is contiguous. + A bool indicating whether the memory is :term:`contiguous`. .. versionadded:: 3.3 diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 19bdb21b3e..2bd8dfdc96 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -95,6 +95,10 @@ implementation as the built-in :meth:`format` method. an arbitrary set of positional and keyword arguments. :meth:`format` is just a wrapper that calls :meth:`vformat`. + .. deprecated:: 3.5 + Passing a format string as keyword argument *format_string* has been + deprecated. + .. method:: vformat(format_string, args, kwargs) This function does the actual work of formatting. It is exposed as a diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 9ac8882d32..adf99ec362 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -25,160 +25,99 @@ modules and functions can be found in the following sections. Using the :mod:`subprocess` Module ---------------------------------- -The recommended approach to invoking subprocesses is to use the following -convenience functions for all use cases they can handle. For more advanced -use cases, the underlying :class:`Popen` interface can be used directly. +The recommended approach to invoking subprocesses is to use the :func:`run` +function for all use cases it can handle. For more advanced use cases, the +underlying :class:`Popen` interface can be used directly. +The :func:`run` function was added in Python 3.5; if you need to retain +compatibility with older versions, see the :ref:`call-function-trio` section. -.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + +.. function:: run(args, *, stdin=None, input=None, stdout=None, stderr=None,\ + shell=False, timeout=None, check=False) Run the command described by *args*. Wait for command to complete, then - return the :attr:`returncode` attribute. + return a :class:`CompletedProcess` instance. The arguments shown above are merely the most common ones, described below in :ref:`frequently-used-arguments` (hence the use of keyword-only notation in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this function passes all - supplied arguments other than *timeout* directly through to that interface. + same as that of the :class:`Popen` constructor - apart from *timeout*, + *input* and *check*, all the arguments to this function are passed through to + that interface. + + This does not capture stdout or stderr by default. To do so, pass + :data:`PIPE` for the *stdout* and/or *stderr* arguments. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The + The *timeout* argument is passed to :meth:`Popen.communicate`. If the timeout + expires, the child process will be killed and waited for. The :exc:`TimeoutExpired` exception will be re-raised after the child process has terminated. - Examples:: - - >>> subprocess.call(["ls", "-l"]) - 0 - - >>> subprocess.call("exit 1", shell=True) - 1 - - .. note:: - - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. - - .. versionchanged:: 3.3 - *timeout* was added. - - -.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) - - Run command with arguments. Wait for command to complete. If the return - code was zero then return, otherwise raise :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`~CalledProcessError.returncode` attribute. - - The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the use of keyword-only notation - in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this function passes all - supplied arguments other than *timeout* directly through to that interface. + The *input* argument is passed to :meth:`Popen.communicate` and thus to the + subprocess's stdin. If used it must be a byte sequence, or a string if + ``universal_newlines=True``. When used, the internal :class:`Popen` object + is automatically created with ``stdin=PIPE``, and the *stdin* argument may + not be used as well. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The - :exc:`TimeoutExpired` exception will be re-raised after the child process - has terminated. + If *check* is True, and the process exits with a non-zero exit code, a + :exc:`CalledProcessError` exception will be raised. Attributes of that + exception hold the arguments, the exit code, and stdout and stderr if they + were captured. Examples:: - >>> subprocess.check_call(["ls", "-l"]) - 0 + >>> subprocess.run(["ls", "-l"]) # doesn't capture output + CompletedProcess(args=['ls', '-l'], returncode=0) - >>> subprocess.check_call("exit 1", shell=True) + >>> subprocess.run("exit 1", shell=True, check=True) Traceback (most recent call last): - ... + ... subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 - .. note:: + >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) + CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, + stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. + .. versionadded:: 3.5 - .. versionchanged:: 3.3 - *timeout* was added. - - -.. function:: check_output(args, *, input=None, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) - - Run command with arguments and return its output. - - If the return code was non-zero it raises a :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`~CalledProcessError.returncode` attribute and any output in the - :attr:`~CalledProcessError.output` attribute. - - The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the use of keyword-only notation - in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this functions passes all - supplied arguments other than *input* and *timeout* directly through to - that interface. In addition, *stdout* is not permitted as an argument, as - it is used internally to collect the output from the subprocess. - - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The - :exc:`TimeoutExpired` exception will be re-raised after the child process - has terminated. - - The *input* argument is passed to :meth:`Popen.communicate` and thus to the - subprocess's stdin. If used it must be a byte sequence, or a string if - ``universal_newlines=True``. When used, the internal :class:`Popen` object - is automatically created with ``stdin=PIPE``, and the *stdin* argument may - not be used as well. +.. class:: CompletedProcess - Examples:: + The return value from :func:`run`, representing a process that has finished. - >>> subprocess.check_output(["echo", "Hello World!"]) - b'Hello World!\n' + .. attribute:: args - >>> subprocess.check_output(["echo", "Hello World!"], universal_newlines=True) - 'Hello World!\n' + The arguments used to launch the process. This may be a list or a string. - >>> subprocess.check_output(["sed", "-e", "s/foo/bar/"], - ... input=b"when in the course of fooman events\n") - b'when in the course of barman events\n' + .. attribute:: returncode - >>> subprocess.check_output("exit 1", shell=True) - Traceback (most recent call last): - ... - subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + Exit status of the child process. Typically, an exit status of 0 indicates + that it ran successfully. - By default, this function will return the data as encoded bytes. The actual - encoding of the output data may depend on the command being invoked, so the - decoding to text will often need to be handled at the application level. + A negative value ``-N`` indicates that the child was terminated by signal + ``N`` (POSIX only). - This behaviour may be overridden by setting *universal_newlines* to - ``True`` as described below in :ref:`frequently-used-arguments`. + .. attribute:: stdout - To also capture standard error in the result, use - ``stderr=subprocess.STDOUT``:: + Captured stdout from the child process. A bytes sequence, or a string if + :func:`run` was called with ``universal_newlines=True``. None if stdout + was not captured. - >>> subprocess.check_output( - ... "ls non_existent_file; exit 0", - ... stderr=subprocess.STDOUT, - ... shell=True) - 'ls: non_existent_file: No such file or directory\n' + If you ran the process with ``stderr=subprocess.STDOUT``, stdout and + stderr will be combined in this attribute, and :attr:`stderr` will be + None. - .. note:: + .. attribute:: stderr - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. + Captured stderr from the child process. A bytes sequence, or a string if + :func:`run` was called with ``universal_newlines=True``. None if stderr + was not captured. - .. versionadded:: 3.1 + .. method:: check_returncode() - .. versionchanged:: 3.3 - *timeout* was added. + If :attr:`returncode` is non-zero, raise a :exc:`CalledProcessError`. - .. versionchanged:: 3.4 - *input* was added. + .. versionadded:: 3.5 .. data:: DEVNULL @@ -225,11 +164,22 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: output - Output of the child process if this exception is raised by + Output of the child process if it was captured by :func:`run` or :func:`check_output`. Otherwise, ``None``. + .. attribute:: stdout + + Alias for output, for symmetry with :attr:`stderr`. + + .. attribute:: stderr + + Stderr output of the child process if it was captured by :func:`run`. + Otherwise, ``None``. + .. versionadded:: 3.3 + .. versionchanged:: 3.5 + *stdout* and *stderr* attributes added .. exception:: CalledProcessError @@ -246,9 +196,20 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: output - Output of the child process if this exception is raised by + Output of the child process if it was captured by :func:`run` or :func:`check_output`. Otherwise, ``None``. + .. attribute:: stdout + + Alias for output, for symmetry with :attr:`stderr`. + + .. attribute:: stderr + + Stderr output of the child process if it was captured by :func:`run`. + Otherwise, ``None``. + + .. versionchanged:: 3.5 + *stdout* and *stderr* attributes added .. _frequently-used-arguments: @@ -635,6 +596,7 @@ Instances of the :class:`Popen` class have the following methods: must be bytes or, if *universal_newlines* was ``True``, a string. :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``. + The data will be bytes or, if *universal_newlines* was ``True``, strings. Note that if you want to send data to the process's stdin, you need to create the Popen object with ``stdin=PIPE``. Similarly, to get anything other than @@ -851,6 +813,112 @@ The :mod:`subprocess` module exposes the following constants. This flag is ignored if :data:`CREATE_NEW_CONSOLE` is specified. +.. _call-function-trio: + +Older high-level API +-------------------- + +Prior to Python 3.5, these three functions comprised the high level API to +subprocess. You can now use :func:`run` in many cases, but lots of existing code +calls these functions. + +.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run the command described by *args*. Wait for command to complete, then + return the :attr:`~Popen.returncode` attribute. + + This is equivalent to:: + + run(...).returncode + + (except that the *input* and *check* parameters are not supported) + + The arguments shown above are merely the most + common ones. The full function signature is largely the + same as that of the :class:`Popen` constructor - this function passes all + supplied arguments other than *timeout* directly through to that interface. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this + function. The child process will block if it generates enough + output to a pipe to fill up the OS pipe buffer as the pipes are + not being read from. + + .. versionchanged:: 3.3 + *timeout* was added. + +.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run command with arguments. Wait for command to complete. If the return + code was zero then return, otherwise raise :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`~CalledProcessError.returncode` attribute. + + This is equivalent to:: + + run(..., check=True) + + (except that the *input* parameter is not supported) + + The arguments shown above are merely the most + common ones. The full function signature is largely the + same as that of the :class:`Popen` constructor - this function passes all + supplied arguments other than *timeout* directly through to that interface. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this + function. The child process will block if it generates enough + output to a pipe to fill up the OS pipe buffer as the pipes are + not being read from. + + .. versionchanged:: 3.3 + *timeout* was added. + + +.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) + + Run command with arguments and return its output. + + If the return code was non-zero it raises a :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`~CalledProcessError.returncode` attribute and any output in the + :attr:`~CalledProcessError.output` attribute. + + This is equivalent to:: + + run(..., check=True, stdout=PIPE).stdout + + The arguments shown above are merely the most common ones. + The full function signature is largely the same as that of :func:`run` - + most arguments are passed directly through to that interface. + However, explicitly passing ``input=None`` to inherit the parent's + standard input file handle is not supported. + + By default, this function will return the data as encoded bytes. The actual + encoding of the output data may depend on the command being invoked, so the + decoding to text will often need to be handled at the application level. + + This behaviour may be overridden by setting *universal_newlines* to + ``True`` as described above in :ref:`frequently-used-arguments`. + + To also capture standard error in the result, use + ``stderr=subprocess.STDOUT``:: + + >>> subprocess.check_output( + ... "ls non_existent_file; exit 0", + ... stderr=subprocess.STDOUT, + ... shell=True) + 'ls: non_existent_file: No such file or directory\n' + + .. versionadded:: 3.1 + + .. versionchanged:: 3.3 + *timeout* was added. + + .. versionchanged:: 3.4 + Support for the *input* keyword argument was added. .. _subprocess-replacements: diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index 2503d335e8..ba2caff589 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -71,10 +71,6 @@ Examining Symbol Tables Return ``True`` if the block uses ``exec``. - .. method:: has_import_star() - - Return ``True`` if the block uses a starred from-import. - .. method:: get_identifiers() Return a list of names of symbols in this table. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 3024086a22..bb9bdc86a1 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -167,7 +167,7 @@ always available. .. data:: dont_write_bytecode - If this is true, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If this is true, Python won't try to write ``.pyc`` files on the import of source modules. This value is initially set to ``True`` or ``False`` depending on the :option:`-B` command line option and the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable, but you can set it @@ -576,6 +576,18 @@ always available. *service_pack_major*, *suite_mask*, and *product_type*. +.. function:: get_coroutine_wrapper() + + Returns ``None``, or a wrapper set by :func:`set_coroutine_wrapper`. + + .. versionadded:: 3.5 + See :pep:`492` for more details. + + .. note:: + This function has been added on a provisional basis (see :pep:`411` + for details.) Use it only for debugging purposes. + + .. data:: hash_info A :term:`struct sequence` giving parameters of the numeric hash @@ -718,6 +730,14 @@ always available. value of :func:`intern` around to benefit from it. +.. function:: is_finalizing() + + Return :const:`True` if the Python interpreter is + :term:`shutting down <interpreter shutdown>`, :const:`False` otherwise. + + .. versionadded:: 3.5 + + .. data:: last_type last_value last_traceback @@ -1053,6 +1073,46 @@ always available. thus not likely to be implemented elsewhere. +.. function:: set_coroutine_wrapper(wrapper) + + Allows intercepting creation of :term:`coroutine` objects (only ones that + are created by an :keyword:`async def` function; generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine` will not be + intercepted). + + The *wrapper* argument must be either: + + * a callable that accepts one argument (a coroutine object); + * ``None``, to reset the wrapper. + + If called twice, the new wrapper replaces the previous one. The function + is thread-specific. + + The *wrapper* callable cannot define new coroutines directly or indirectly:: + + def wrapper(coro): + async def wrap(coro): + return await coro + return wrap(coro) + sys.set_coroutine_wrapper(wrapper) + + async def foo(): + pass + + # The following line will fail with a RuntimeError, because + # ``wrapper`` creates a ``wrap(coro)`` coroutine: + foo() + + See also :func:`get_coroutine_wrapper`. + + .. versionadded:: 3.5 + See :pep:`492` for more details. + + .. note:: + This function has been added on a provisional basis (see :pep:`411` + for details.) Use it only for debugging purposes. + + .. data:: stdin stdout stderr @@ -1223,4 +1283,3 @@ always available. .. rubric:: Citations .. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ . - diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 05f29adb0b..adacb0ab3c 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -62,6 +62,23 @@ Some facts and figures: +------------------+---------------------------------------------+ | ``'r:xz'`` | Open for reading with lzma compression. | +------------------+---------------------------------------------+ + | ``'x'`` or | Create a tarfile exclusively without | + | ``'x:'`` | compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:gz'`` | Create a tarfile with gzip compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:bz2'`` | Create a tarfile with bzip2 compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:xz'`` | Create a tarfile with lzma compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ | ``'a' or 'a:'`` | Open for appending with no compression. The | | | file is created if it does not exist. | +------------------+---------------------------------------------+ @@ -82,9 +99,9 @@ Some facts and figures: If *fileobj* is specified, it is used as an alternative to a :term:`file object` opened in binary mode for *name*. It is supposed to be at position 0. - For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, :func:`tarfile.open` - accepts the keyword argument *compresslevel* to specify the compression level of - the file. + For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, ``'x:gz'``, + ``'x:bz2'``, :func:`tarfile.open` accepts the keyword argument + *compresslevel* to specify the compression level of the file. For special purposes, there is a second format for *mode*: ``'filemode|[compression]'``. :func:`tarfile.open` will return a :class:`TarFile` @@ -127,6 +144,8 @@ Some facts and figures: | | writing. | +-------------+--------------------------------------------+ + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. class:: TarFile @@ -252,8 +271,8 @@ be finalized; only the internally used file object will be closed. See the In this case, the file object's :attr:`name` attribute is used if it exists. *mode* is either ``'r'`` to read from an existing archive, ``'a'`` to append - data to an existing file or ``'w'`` to create a new file overwriting an existing - one. + data to an existing file, ``'w'`` to create a new file overwriting an existing + one or ``'x'`` to create a new file only if it's not exists. If *fileobj* is given, it is used for reading or writing data. If it can be determined, *mode* is overridden by *fileobj*'s mode. *fileobj* will be used @@ -292,12 +311,14 @@ be finalized; only the internally used file object will be closed. See the to be handled. The default settings will work for most users. See section :ref:`tar-unicode` for in-depth information. - .. versionchanged:: 3.2 - Use ``'surrogateescape'`` as the default for the *errors* argument. - The *pax_headers* argument is an optional dictionary of strings which will be added as a pax global header if *format* is :const:`PAX_FORMAT`. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. classmethod:: TarFile.open(...) @@ -328,11 +349,15 @@ be finalized; only the internally used file object will be closed. See the returned by :meth:`getmembers`. -.. method:: TarFile.list(verbose=True) +.. method:: TarFile.list(verbose=True, *, members=None) Print a table of contents to ``sys.stdout``. If *verbose* is :const:`False`, only the names of the members are printed. If it is :const:`True`, output - similar to that of :program:`ls -l` is produced. + similar to that of :program:`ls -l` is produced. If optional *members* is + given, it must be a subset of the list returned by :meth:`getmembers`. + + .. versionchanged:: 3.5 + Added the *members* parameter. .. method:: TarFile.next() @@ -342,7 +367,7 @@ be finalized; only the internally used file object will be closed. See the available. -.. method:: TarFile.extractall(path=".", members=None) +.. method:: TarFile.extractall(path=".", members=None, *, numeric_owner=False) Extract all members from the archive to the current working directory or directory *path*. If optional *members* is given, it must be a subset of the @@ -352,6 +377,10 @@ be finalized; only the internally used file object will be closed. See the reset each time a file is created in it. And, if a directory's permissions do not allow writing, extracting files to it will fail. + If *numeric_owner* is :const:`True`, the uid and gid numbers from the tarfile + are used to set the owner/group for the extracted files. Otherwise, the named + values from the tarfile are used. + .. warning:: Never extract archives from untrusted sources without prior inspection. @@ -359,8 +388,11 @@ be finalized; only the internally used file object will be closed. See the that have absolute filenames starting with ``"/"`` or filenames with two dots ``".."``. + .. versionchanged:: 3.5 + Added the *numeric_only* parameter. -.. method:: TarFile.extract(member, path="", set_attrs=True) + +.. method:: TarFile.extract(member, path="", set_attrs=True, *, numeric_owner=False) Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. *member* @@ -368,6 +400,10 @@ be finalized; only the internally used file object will be closed. See the directory using *path*. File attributes (owner, mtime, mode) are set unless *set_attrs* is false. + If *numeric_owner* is :const:`True`, the uid and gid numbers from the tarfile + are used to set the owner/group for the extracted files. Otherwise, the named + values from the tarfile are used. + .. note:: The :meth:`extract` method does not take care of several extraction issues. @@ -380,6 +416,9 @@ be finalized; only the internally used file object will be closed. See the .. versionchanged:: 3.2 Added the *set_attrs* parameter. + .. versionchanged:: 3.5 + Added the *numeric_only* parameter. + .. method:: TarFile.extractfile(member) Extract a member from the archive as a file object. *member* may be a filename @@ -802,4 +841,3 @@ In case of :const:`PAX_FORMAT` archives, *encoding* is generally not needed because all the metadata is stored using *UTF-8*. *encoding* is only used in the rare cases when binary pax headers are decoded or when strings with surrogate characters are stored. - diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 44d025defe..83f994170a 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -16,16 +16,18 @@ -------------- -This module generates temporary files and directories. It works on all -supported platforms. It provides three new functions, -:func:`NamedTemporaryFile`, :func:`mkstemp`, and :func:`mkdtemp`, which should -eliminate all remaining need to use the insecure :func:`mktemp` function. -Temporary file names created by this module no longer contain the process ID; -instead a string of six random characters is used. - -Also, all the user-callable functions now take additional arguments which -allow direct control over the location and name of temporary files. It is -no longer necessary to use the global *tempdir* variable. +This module creates temporary files and directories. It works on all +supported platforms. :class:`TemporaryFile`, :class:`NamedTemporaryFile`, +:class:`TemporaryDirectory`, and :class:`SpooledTemporaryFile` are high-level +interfaces which provide automatic cleanup and can be used as +context managers. :func:`mkstemp` and +:func:`mkdtemp` are lower-level functions which require manual cleanup. + +All the user-callable functions and constructors take additional arguments which +allow direct control over the location and name of temporary files and +directories. Files names used by this module include a string of +random characters which allows those files to be securely created in +shared temporary directories. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. @@ -34,25 +36,37 @@ The module defines the following user-callable items: .. function:: TemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None) Return a :term:`file-like object` that can be used as a temporary storage area. - The file is created using :func:`mkstemp`. It will be destroyed as soon + The file is created securely, using the same rules as :func:`mkstemp`. It will be destroyed as soon as it is closed (including an implicit close when the object is garbage - collected). Under Unix, the directory entry for the file is removed + collected). Under Unix, the directory entry for the file is either not created at all or is removed immediately after the file is created. Other platforms do not support this; your code should not rely on a temporary file created using this function having or not having a visible name in the file system. + The resulting object can be used as a context manager (see + :ref:`tempfile-examples`). On completion of the context or + destruction of the file object the temporary file will be removed + from the filesystem. + The *mode* parameter defaults to ``'w+b'`` so that the file created can be read and written without being closed. Binary mode is used so that it behaves consistently on all platforms without regard for the data that is stored. *buffering*, *encoding* and *newline* are interpreted as for :func:`open`. - The *dir*, *prefix* and *suffix* parameters are passed to :func:`mkstemp`. + The *dir*, *prefix* and *suffix* parameters have the same meaning + as with :func:`mkstemp`. The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose :attr:`!file` attribute is the - underlying true file object. This file-like object can be used in a - :keyword:`with` statement, just like a normal file. + underlying true file object. + + The :py:data:`os.O_TMPFILE` flag is used if it is available and works + (Linux-specific, requires Linux kernel 3.11 or later). + + .. versionchanged:: 3.5 + + The :py:data:`os.O_TMPFILE` flag is now used if available. .. function:: NamedTemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None, delete=True) @@ -94,10 +108,9 @@ The module defines the following user-callable items: .. function:: TemporaryDirectory(suffix='', prefix='tmp', dir=None) - This function creates a temporary directory using :func:`mkdtemp` - (the supplied arguments are passed directly to the underlying function). + This function securely creates a temporary directory using the same rules as :func:`mkdtemp`. The resulting object can be used as a context manager (see - :ref:`context-managers`). On completion of the context or destruction + :ref:`tempfile-examples`). On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the filesystem. @@ -112,7 +125,7 @@ The module defines the following user-callable items: .. versionadded:: 3.2 -.. function:: mkstemp(suffix='', prefix='tmp', dir=None, text=False) +.. function:: mkstemp(suffix=None, prefix=None, dir=None, text=False) Creates a temporary file in the most secure manner possible. There are no race conditions in the file's creation, assuming that the platform @@ -141,6 +154,16 @@ The module defines the following user-callable items: filename will have any nice properties, such as not requiring quoting when passed to external commands via ``os.popen()``. + *suffix*, *prefix*, and *dir* must all contain the same type, if specified. + If they are bytes, the returned name will be bytes instead of str. + If you want to force a bytes return value with otherwise default behavior, + pass ``suffix=b''``. + + A *prefix* value of ``None`` means use the return value of + :func:`gettempprefix` or :func:`gettempprefixb` as appropriate. + + A *suffix* value of ``None`` means use an appropriate empty value. + If *text* is specified, it indicates whether to open the file in binary mode (the default) or text mode. On some platforms, this makes no difference. @@ -149,8 +172,14 @@ The module defines the following user-callable items: file (as would be returned by :func:`os.open`) and the absolute pathname of that file, in that order. + .. versionchanged:: 3.5 + *suffix*, *prefix*, and *dir* may now be supplied in bytes in order to + obtain a bytes return value. Prior to this, only str was allowed. + *suffix* and *prefix* now accept and default to ``None`` to cause + an appropriate default value to be used. + -.. function:: mkdtemp(suffix='', prefix='tmp', dir=None) +.. function:: mkdtemp(suffix=None, prefix=None, dir=None) Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory's creation. The directory is @@ -164,50 +193,21 @@ The module defines the following user-callable items: :func:`mkdtemp` returns the absolute pathname of the new directory. + .. versionchanged:: 3.5 + *suffix*, *prefix*, and *dir* may now be supplied in bytes in order to + obtain a bytes return value. Prior to this, only str was allowed. + *suffix* and *prefix* now accept and default to ``None`` to cause + an appropriate default value to be used. -.. function:: mktemp(suffix='', prefix='tmp', dir=None) - - .. deprecated:: 2.3 - Use :func:`mkstemp` instead. - - Return an absolute pathname of a file that did not exist at the time the - call is made. The *prefix*, *suffix*, and *dir* arguments are the same - as for :func:`mkstemp`. - .. warning:: - - Use of this function may introduce a security hole in your program. By - the time you get around to doing anything with the file name it returns, - someone else may have beaten you to the punch. :func:`mktemp` usage can - be replaced easily with :func:`NamedTemporaryFile`, passing it the - ``delete=False`` parameter:: - - >>> f = NamedTemporaryFile(delete=False) - >>> f.name - '/tmp/tmptjujjt' - >>> f.write(b"Hello World!\n") - 13 - >>> f.close() - >>> os.unlink(f.name) - >>> os.path.exists(f.name) - False - -The module uses a global variable that tell it how to construct a -temporary name. They are initialized at the first call to any of the -functions above. The caller may change them, but this is discouraged; use -the appropriate function arguments, instead. - - -.. data:: tempdir +.. function:: gettempdir() - When set to a value other than ``None``, this variable defines the - default value for the *dir* argument to all the functions defined in this - module. + Return the name of the directory used for temporary files. This + defines the default value for the *dir* argument to all functions + in this module. - If ``tempdir`` is unset or ``None`` at any call to any of the above - functions, Python searches a standard list of directories and sets - *tempdir* to the first one which the calling user can create files in. - The list is: + Python searches a standard list of directories to find one which + the calling user can create files in. The list is: #. The directory named by the :envvar:`TMPDIR` environment variable. @@ -225,19 +225,43 @@ the appropriate function arguments, instead. #. As a last resort, the current working directory. + The result of this search is cached, see the description of + :data:`tempdir` below. -.. function:: gettempdir() +.. function:: gettempdirb() - Return the directory currently selected to create temporary files in. If - :data:`tempdir` is not ``None``, this simply returns its contents; otherwise, - the search described above is performed, and the result returned. + Same as :func:`gettempdir` but the return value is in bytes. + .. versionadded:: 3.5 .. function:: gettempprefix() Return the filename prefix used to create temporary files. This does not contain the directory component. +.. function:: gettempprefixb() + + Same as :func:`gettempprefixb` but the return value is in bytes. + + .. versionadded:: 3.5 + +The module uses a global variable to store the name of the directory +used for temporary files returned by :func:`gettempdir`. It can be +set directly to override the selection process, but this is discouraged. +All functions in this module take a *dir* argument which can be used +to specify the directory and this is the recommend approach. + +.. data:: tempdir + + When set to a value other than ``None``, this variable defines the + default value for the *dir* argument to all the functions defined in this + module. + + If ``tempdir`` is unset or ``None`` at any call to any of the above + functions except :func:`gettempprefix` it is initalized following the + algorithm described in :func:`gettempdir`. + +.. _tempfile-examples: Examples -------- @@ -271,3 +295,42 @@ Here are some examples of typical usage of the :mod:`tempfile` module:: >>> # directory and contents have been removed + +Deprecated functions and variables +---------------------------------- + +A historical way to create temporary files was to first generate a +file name with the :func:`mktemp` function and then create a file +using this name. Unfortunately this is not secure, because a different +process may create a file with this name in the time between the call +to :func:`mktemp` and the subsequent attempt to create the file by the +first process. The solution is to combine the two steps and create the +file immediately. This approach is used by :func:`mkstemp` and the +other functions described above. + +.. function:: mktemp(suffix='', prefix='tmp', dir=None) + + .. deprecated:: 2.3 + Use :func:`mkstemp` instead. + + Return an absolute pathname of a file that did not exist at the time the + call is made. The *prefix*, *suffix*, and *dir* arguments are the same + as for :func:`mkstemp`. + + .. warning:: + + Use of this function may introduce a security hole in your program. By + the time you get around to doing anything with the file name it returns, + someone else may have beaten you to the punch. :func:`mktemp` usage can + be replaced easily with :func:`NamedTemporaryFile`, passing it the + ``delete=False`` parameter:: + + >>> f = NamedTemporaryFile(delete=False) + >>> f.name + '/tmp/tmptjujjt' + >>> f.write(b"Hello World!\n") + 13 + >>> f.close() + >>> os.unlink(f.name) + >>> os.path.exists(f.name) + False diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 2fdaf8cb37..85cab3b227 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -568,6 +568,17 @@ The :mod:`test.support` module defines the following functions: def load_tests(*args): return load_package_tests(os.path.dirname(__file__), *args) +.. function:: detect_api_mismatch(ref_api, other_api, *, ignore=()): + + Returns the set of attributes, functions or methods of *ref_api* not + found on *other_api*, except for a defined list of items to be + ignored in this check specified in *ignore*. + + By default this skips private attributes beginning with '_' but + includes all magic methods, i.e. those starting and ending in '__'. + + .. versionadded:: 3.5 + The :mod:`test.support` module defines the following classes: diff --git a/Doc/library/time.rst b/Doc/library/time.rst index 8d8b7d4c55..3d335c8867 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -314,9 +314,9 @@ The module defines the following functions and data items: processes running for more than 49 days. On more recent versions of Windows and on other operating systems, :func:`monotonic` is system-wide. - Availability: Windows, Mac OS X, Linux, FreeBSD, OpenBSD, Solaris. - .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now always available. .. function:: perf_counter() @@ -350,6 +350,11 @@ The module defines the following functions and data items: requested by an arbitrary amount because of the scheduling of other activity in the system. + .. versionchanged:: 3.5 + The function now sleeps at least *secs* even if the sleep is interrupted + by a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale). + .. function:: strftime(format[, t]) diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 70df409fac..d1051f688c 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -59,18 +59,26 @@ Python Interface The module defines three convenience functions and a public class: -.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000) +.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.timeit` method with *number* executions. + The optional *globals* argument specifies a namespace in which to execute the + code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. -.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000) + +.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.repeat` method with the given *repeat* - count and *number* executions. + count and *number* executions. The optional *globals* argument specifies a + namespace in which to execute the code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. function:: default_timer() @@ -80,7 +88,7 @@ The module defines three convenience functions and a public class: :func:`time.perf_counter` is now the default timer. -.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>) +.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>, globals=None) Class for timing execution speed of small code snippets. @@ -88,7 +96,9 @@ The module defines three convenience functions and a public class: for setup, and a timer function. Both statements default to ``'pass'``; the timer function is platform-dependent (see the module doc string). *stmt* and *setup* may also contain multiple statements separated by ``;`` - or newlines, as long as they don't contain multi-line string literals. + or newlines, as long as they don't contain multi-line string literals. The + statement will by default be executed within timeit's namespace; this behavior + can be controlled by passing a namespace to *globals*. To measure the execution time of the first statement, use the :meth:`.timeit` method. The :meth:`.repeat` method is a convenience to call :meth:`.timeit` @@ -101,6 +111,8 @@ The module defines three convenience functions and a public class: will then be executed by :meth:`.timeit`. Note that the timing overhead is a little larger in this case because of the extra function calls. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. method:: Timer.timeit(number=1000000) @@ -169,7 +181,7 @@ Command-Line Interface When called as a program from the command line, the following form is used:: - python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...] + python -m timeit [-n N] [-r N] [-u U] [-s S] [-t] [-c] [-h] [statement ...] Where the following options are understood: @@ -198,6 +210,12 @@ Where the following options are understood: use :func:`time.time` (deprecated) +.. cmdoption:: -u, --unit=U + + specify a time unit for timer output; can select usec, msec, or sec + + .. versionadded:: 3.5 + .. cmdoption:: -c, --clock use :func:`time.clock` (deprecated) @@ -320,3 +338,17 @@ To give the :mod:`timeit` module access to functions you define, you can pass a if __name__ == '__main__': import timeit print(timeit.timeit("test()", setup="from __main__ import test")) + +Another option is to pass :func:`globals` to the *globals* parameter, which will cause the code +to be executed within your current global namespace. This can be more convenient +than individually specifying imports:: + + def f(x): + return x**2 + def g(x): + return x**4 + def h(x): + return x**8 + + import timeit + print(timeit.timeit('[func(42) for func in (f,g,h)]', globals=globals())) diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 4cd709814c..88fb38bc1d 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -93,6 +93,7 @@ The token constants are: DOUBLESLASH DOUBLESLASHEQUAL AT + ATEQUAL RARROW ELLIPSIS OP diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 15fbedcf33..8d216d07e3 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -22,15 +22,20 @@ The module defines the following functions: .. function:: print_tb(traceback, limit=None, file=None) - Print up to *limit* stack trace entries from *traceback*. If *limit* is omitted - or ``None``, all entries are printed. If *file* is omitted or ``None``, the - output goes to ``sys.stderr``; otherwise it should be an open file or file-like - object to receive the output. + Print up to *limit* stack trace entries from *traceback* (starting from + the caller's frame) if *limit* is positive. Otherwise, print the last + ``abs(limit)`` entries. If *limit* is omitted or ``None``, all entries + are printed. If *file* is omitted or ``None``, the output goes to + ``sys.stderr``; otherwise it should be an open file or file-like object + to receive the output. + + .. versionchanged:: 3.5 + Added negative *limit* support. .. function:: print_exception(type, value, traceback, limit=None, file=None, chain=True) - Print exception information and up to *limit* stack trace entries from + Print exception information and stack trace entries from *traceback* to *file*. This differs from :func:`print_tb` in the following ways: @@ -41,6 +46,7 @@ The module defines the following functions: prints the line where the syntax error occurred with a caret indicating the approximate position of the error. + The optional *limit* argument has the same meaning as for :func:`print_tb`. If *chain* is true (the default), then chained exceptions (the :attr:`__cause__` or :attr:`__context__` attributes of the exception) will be printed as well, like the interpreter itself does when printing an unhandled @@ -49,33 +55,41 @@ The module defines the following functions: .. function:: print_exc(limit=None, file=None, chain=True) - This is a shorthand for ``print_exception(*sys.exc_info())``. + This is a shorthand for ``print_exception(*sys.exc_info(), limit, file, + chain)``. .. function:: print_last(limit=None, file=None, chain=True) This is a shorthand for ``print_exception(sys.last_type, sys.last_value, - sys.last_traceback, limit, file)``. In general it will work only after - an exception has reached an interactive prompt (see :data:`sys.last_type`). + sys.last_traceback, limit, file, chain)``. In general it will work only + after an exception has reached an interactive prompt (see + :data:`sys.last_type`). .. function:: print_stack(f=None, limit=None, file=None) - This function prints a stack trace from its invocation point. The optional *f* - argument can be used to specify an alternate stack frame to start. The optional - *limit* and *file* arguments have the same meaning as for - :func:`print_exception`. + Print up to *limit* stack trace entries (starting from the invocation + point) if *limit* is positive. Otherwise, print the last ``abs(limit)`` + entries. If *limit* is omitted or ``None``, all entries are printed. + The optional *f* argument can be used to specify an alternate stack frame + to start. The optional *file* argument has the same meaning as for + :func:`print_tb`. + + .. versionchanged:: 3.5 + Added negative *limit* support. .. function:: extract_tb(traceback, limit=None) - Return a list of up to *limit* "pre-processed" stack trace entries extracted - from the traceback object *traceback*. It is useful for alternate formatting of - stack traces. If *limit* is omitted or ``None``, all entries are extracted. A - "pre-processed" stack trace entry is a 4-tuple (*filename*, *line number*, - *function name*, *text*) representing the information that is usually printed - for a stack trace. The *text* is a string with leading and trailing whitespace - stripped; if the source is not available it is ``None``. + Return a list of "pre-processed" stack trace entries extracted from the + traceback object *traceback*. It is useful for alternate formatting of + stack traces. The optional *limit* argument has the same meaning as for + :func:`print_tb`. A "pre-processed" stack trace entry is a 4-tuple + (*filename*, *line number*, *function name*, *text*) representing the + information that is usually printed for a stack trace. The *text* is a + string with leading and trailing whitespace stripped; if the source is + not available it is ``None``. .. function:: extract_stack(f=None, limit=None) @@ -136,6 +150,162 @@ The module defines the following functions: .. versionadded:: 3.4 +.. function:: walk_stack(f) + + Walk a stack following ``f.f_back`` from the given frame, yielding the frame + and line number for each frame. If *f* is ``None``, the current stack is + used. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +.. function:: walk_tb(tb) + + Walk a traceback following ``tb_next`` yielding the frame and line number + for each frame. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +The module also defines the following classes: + +:class:`TracebackException` Objects +----------------------------------- + +.. versionadded:: 3.5 + +:class:`TracebackException` objects are created from actual exceptions to +capture data for later printing in a lightweight fashion. + +.. class:: TracebackException(exc_type, exc_value, exc_traceback, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. attribute:: __cause__ + + A :class:`TracebackException` of the original ``__cause__``. + + .. attribute:: __context__ + + A :class:`TracebackException` of the original ``__context__``. + + .. attribute:: __suppress_context__ + + The ``__suppress_context__`` value from the original exception. + + .. attribute:: stack + + A :class:`StackSummary` representing the traceback. + + .. attribute:: exc_type + + The class of the original traceback. + + .. attribute:: filename + + For syntax errors - the file name where the error occurred. + + .. attribute:: lineno + + For syntax errors - the line number where the error occurred. + + .. attribute:: text + + For syntax errors - the text where the error occurred. + + .. attribute:: offset + + For syntax errors - the offset into the text where the error occurred. + + .. attribute:: msg + + For syntax errors - the compiler error message. + + .. classmethod:: from_exception(exc, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. method:: format(*, chain=True) + + Format the exception. + + If *chain* is not ``True``, ``__cause__`` and ``__context__`` will not + be formatted. + + The return value is a generator of strings, each ending in a newline and + some containing internal newlines. :func:`~traceback.print_exception` + is a wrapper around this method which just prints the lines to a file. + + The message indicating which exception occurred is always the last + string in the output. + + .. method:: format_exception_only() + + Format the exception part of the traceback. + + The return value is a generator of strings, each ending in a newline. + + Normally, the generator emits a single string; however, for + :exc:`SyntaxError` exceptions, it emits several lines that (when + printed) display detailed information about where the syntax + error occurred. + + The message indicating which exception occurred is always the last + string in the output. + + +:class:`StackSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`StackSummary` objects represent a call stack ready for formatting. + +.. class:: StackSummary + + .. classmethod:: extract(frame_gen, *, limit=None, lookup_lines=True, capture_locals=False) + + Construct a :class:`StackSummary` object from a frame generator (such as + is returned by :func:`~traceback.walk_stack` or + :func:`~traceback.walk_tb`). + + If *limit* is supplied, only this many frames are taken from *frame_gen*. + If *lookup_lines* is ``False``, the returned :class:`FrameSummary` + objects will not have read their lines in yet, making the cost of + creating the :class:`StackSummary` cheaper (which may be valuable if it + may not actually get formatted). If *capture_locals* is ``True`` the + local variables in each :class:`FrameSummary` are captured as object + representations. + + .. classmethod:: from_list(a_list) + + Construct a :class:`StackSummary` object from a supplied old-style list + of tuples. Each tuple should be a 4-tuple with filename, lineno, name, + line as the elements. + + +:class:`FrameSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`FrameSummary` objects represent a single frame in a traceback. + +.. class:: FrameSummary(filename, lineno, name, lookup_line=True, locals=None, line=None) + + Represent a single frame in the traceback or stack that is being formatted + or printed. It may optionally have a stringified version of the frames + locals included in it. If *lookup_line* is ``False``, the source code is not + looked up until the :class:`FrameSummary` has the :attr:`~FrameSummary.line` + attribute accessed (which also happens when casting it to a tuple). + :attr:`~FrameSummary.line` may be directly provided, and will prevent line + lookups happening at all. *locals* is an optional local variable + dictionary, and if supplied the variable representations are stored in the + summary for later display. .. _traceback-example: diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst index a04a432716..13c81a7fed 100644 --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -363,7 +363,7 @@ Filter Filter on traces of memory blocks. See the :func:`fnmatch.fnmatch` function for the syntax of - *filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are + *filename_pattern*. The ``'.pyc'`` file extension is replaced with ``'.py'``. Examples: @@ -374,6 +374,10 @@ Filter :mod:`tracemalloc` module * ``Filter(False, "<unknown>")`` excludes empty tracebacks + + .. versionchanged:: 3.5 + The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. + .. attribute:: inclusive If *inclusive* is ``True`` (include), only trace memory blocks allocated @@ -631,4 +635,3 @@ Traceback obj = Object() File "test.py", line 12 tb = tracemalloc.get_object_traceback(f()) - diff --git a/Doc/library/tulip_coro.dia b/Doc/library/tulip_coro.dia Binary files differindex eccf4fa7f5..70a33e3c00 100644 --- a/Doc/library/tulip_coro.dia +++ b/Doc/library/tulip_coro.dia diff --git a/Doc/library/tulip_coro.png b/Doc/library/tulip_coro.png Binary files differindex 65b6951550..36ced8ddbf 100644 --- a/Doc/library/tulip_coro.png +++ b/Doc/library/tulip_coro.png diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index efe5c54edf..30dd6eff2d 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1879,7 +1879,7 @@ Settings and special methods >>> cv = screen.getcanvas() >>> cv - <turtle.ScrolledCanvas object at ...> + <turtle.ScrolledCanvas object ...> .. function:: getshapes() @@ -2351,6 +2351,9 @@ The demo scripts are: | | pairwise in opposite | shapesize, tilt, | | | direction | get_shapepoly, update | +----------------+------------------------------+-----------------------+ +| sorting_animate| visual demonstration of | simple alignment, | +| | different sorting methods | randomization | ++----------------+------------------------------+-----------------------+ | tree | a (graphical) breadth | :func:`clone` | | | first tree (using generators)| | +----------------+------------------------------+-----------------------+ diff --git a/Doc/library/types.rst b/Doc/library/types.rst index abdb939d1a..eb27846aab 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -86,8 +86,16 @@ Standard names are defined for the following types: .. data:: GeneratorType - The type of :term:`generator`-iterator objects, produced by calling a - generator function. + The type of :term:`generator`-iterator objects, created by + generator functions. + + +.. data:: CoroutineType + + The type of :term:`coroutine` objects, created by + :keyword:`async def` functions. + + .. versionadded:: 3.5 .. data:: CodeType @@ -115,6 +123,10 @@ Standard names are defined for the following types: The type of :term:`modules <module>`. Constructor takes the name of the module to be created and optionally its :term:`docstring`. + .. note:: + Use :func:`importlib.util.module_from_spec` to create a new module if you + wish to set the various import-controlled attributes. + .. attribute:: __doc__ The :term:`docstring` of the module. Defaults to ``None``. @@ -267,3 +279,25 @@ Additional Utility Classes and Functions attributes on the class with the same name (see Enum for an example). .. versionadded:: 3.4 + + +Coroutine Utility Functions +--------------------------- + +.. function:: coroutine(gen_func) + + This function transforms a :term:`generator` function into a + :term:`coroutine function` which returns a generator-based coroutine. + The generator-based coroutine is still a :term:`generator iterator`, + but is also considered to be a :term:`coroutine` object and is + :term:`awaitable`. However, it may not necessarily implement + the :meth:`__await__` method. + + If *gen_func* is a generator function, it will be modified in-place. + + If *gen_func* is not a generator function, it will be wrapped. If it + returns an instance of :class:`collections.abc.Generator`, the instance + will be wrapped in an *awaitable* proxy object. All other types + of objects will be returned as is. + + .. versionadded:: 3.5 diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst new file mode 100644 index 0000000000..668523b52d --- /dev/null +++ b/Doc/library/typing.rst @@ -0,0 +1,524 @@ +:mod:`typing` --- Support for type hints +======================================== + +.. module:: typing + :synopsis: Support for type hints (see PEP 484). + +**Source code:** :source:`Lib/typing.py` + +-------------- + +This module supports type hints as specified by :pep:`484`. The most +fundamental support consists of the type :class:`Any`, :class:`Union`, +:class:`Tuple`, :class:`Callable`, :class:`TypeVar`, and +:class:`Generic`. For full specification please see :pep:`484`. For +a simplified introduction to type hints see :pep:`483`. + + +The function below takes and returns a string and is annotated as follows:: + + def greeting(name: str) -> str: + return 'Hello ' + name + +In the function ``greeting``, the argument ``name`` is expected to by of type +:class:`str` and the return type :class:`str`. Subtypes are accepted as +arguments. + +Type aliases +------------ + +A type alias is defined by assigning the type to the alias:: + + Vector = List[float] + +Callable +-------- + +Frameworks expecting callback functions of specific signatures might be +type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. + +For example:: + + from typing import Callable + + def feeder(get_next_item: Callable[[], str]) -> None: + # Body + + def async_query(on_success: Callable[[int], None], + on_error: Callable[[int, Exception], None]) -> None: + # Body + +It is possible to declare the return type of a callable without specifying +the call signature by substituting a literal ellipsis +for the list of arguments in the type hint: ``Callable[..., ReturnType]``. +``None`` as a type hint is a special case and is replaced by ``type(None)``. + +Generics +-------- + +Since type information about objects kept in containers cannot be statically +inferred in a generic way, abstract base classes have been extended to support +subscription to denote expected types for container elements. + +:: + + from typing import Mapping, Sequence + + def notify_by_email(employees: Sequence[Employee], + overrides: Mapping[str, str]) -> None: ... + +Generics can be parametrized by using a new factory available in typing +called :class:`TypeVar`. + +:: + + from typing import Sequence, TypeVar + + T = TypeVar('T') # Declare type variable + + def first(l: Sequence[T]) -> T: # Generic function + return l[0] + + +User-defined generic types +-------------------------- + +A user-defined class can be defined as a generic class. + +:: + + from typing import TypeVar, Generic + from logging import Logger + + T = TypeVar('T') + + class LoggedVar(Generic[T]): + def __init__(self, value: T, name: str, logger: Logger) -> None: + self.name = name + self.logger = logger + self.value = value + + def set(self, new: T) -> None: + self.log('Set ' + repr(self.value)) + self.value = new + + def get(self) -> T: + self.log('Get ' + repr(self.value)) + return self.value + + def log(self, message: str) -> None: + self.logger.info('{}: {}'.format(self.name, message)) + +``Generic[T]`` as a base class defines that the class ``LoggedVar`` takes a +single type parameter ``T`` . This also makes ``T`` valid as a type within the +class body. + +The :class:`Generic` base class uses a metaclass that defines +:meth:`__getitem__` so that ``LoggedVar[t]`` is valid as a type:: + + from typing import Iterable + + def zero_all_vars(vars: Iterable[LoggedVar[int]]) -> None: + for var in vars: + var.set(0) + +A generic type can have any number of type variables, and type variables may +be constrained:: + + from typing import TypeVar, Generic + ... + + T = TypeVar('T') + S = TypeVar('S', int, str) + + class StrangePair(Generic[T, S]): + ... + +Each type variable argument to :class:`Generic` must be distinct. +This is thus invalid:: + + from typing import TypeVar, Generic + ... + + T = TypeVar('T') + + class Pair(Generic[T, T]): # INVALID + ... + +You can use multiple inheritance with :class:`Generic`:: + + from typing import TypeVar, Generic, Sized + + T = TypeVar('T') + + class LinkedList(Sized, Generic[T]): + ... + +When inheriting from generic classes, some type variables could fixed:: + + from typing import TypeVar, Mapping + + T = TypeVar('T') + + class MyDict(Mapping[str, T]): + ... + +In this case ``MyDict`` has a single parameter, ``T``. + +Subclassing a generic class without specifying type parameters assumes +:class:`Any` for each position. In the following example, ``MyIterable`` is +not generic but implicitly inherits from ``Iterable[Any]``:: + + from typing import Iterable + + class MyIterable(Iterable): # Same as Iterable[Any] + +The metaclass used by :class:`Generic` is a subclass of :class:`abc.ABCMeta`. +A generic class can be an ABC by including abstract methods or properties, +and generic classes can also have ABCs as base classes without a metaclass +conflict. Generic metaclasses are not supported. + + +The :class:`Any` type +--------------------- + +A special kind of type is :class:`Any`. Every type is a subtype of +:class:`Any`. This is also true for the builtin type object. However, to the +static type checker these are completely different. + +When the type of a value is :class:`object`, the type checker will reject +almost all operations on it, and assigning it to a variable (or using it as a +return value) of a more specialized type is a type error. On the other hand, +when a value has type :class:`Any`, the type checker will allow all operations +on it, and a value of type :class:`Any` can be assigned to a variable (or used +as a return value) of a more constrained type. + + +Classes, functions, and decorators +---------------------------------- + +The module defines the following classes, functions and decorators: + +.. class:: Any + + Special type indicating an unconstrained type. + + * Any object is an instance of :class:`Any`. + * Any class is a subclass of :class:`Any`. + * As a special case, :class:`Any` and :class:`object` are subclasses of + each other. + +.. class:: TypeVar + + Type variable. + + Usage:: + + T = TypeVar('T') # Can be anything + A = TypeVar('A', str, bytes) # Must be str or bytes + + Type variables exist primarily for the benefit of static type + checkers. They serve as the parameters for generic types as well + as for generic function definitions. See class Generic for more + information on generic types. Generic functions work as follows:: + + def repeat(x: T, n: int) -> Sequence[T]: + """Return a list containing n references to x.""" + return [x]*n + + def longest(x: A, y: A) -> A: + """Return the longest of two strings.""" + return x if len(x) >= len(y) else y + + The latter example's signature is essentially the overloading + of ``(str, str) -> str`` and ``(bytes, bytes) -> bytes``. Also note + that if the arguments are instances of some subclass of :class:`str`, + the return type is still plain :class:`str`. + + At runtime, ``isinstance(x, T)`` will raise :exc:`TypeError`. In general, + :func:`isinstance` and :func:`issubclass` should not be used with types. + + Type variables may be marked covariant or contravariant by passing + ``covariant=True`` or ``contravariant=True``. See :pep:`484` for more + details. By default type variables are invariant. Alternatively, + a type variable may specify an upper bound using ``bound=<type>``. + This means that an actual type substituted (explicitly or implicitly) + for the type variable must be a subclass of the boundary type, + see :pep:`484`. + +.. class:: Union + + Union type; ``Union[X, Y]`` means either X or Y. + + To define a union, use e.g. ``Union[int, str]``. Details: + + * The arguments must be types and there must be at least one. + + * Unions of unions are flattened, e.g.:: + + Union[Union[int, str], float] == Union[int, str, float] + + * Unions of a single argument vanish, e.g.:: + + Union[int] == int # The constructor actually returns int + + * Redundant arguments are skipped, e.g.:: + + Union[int, str, int] == Union[int, str] + + * When comparing unions, the argument order is ignored, e.g.:: + + Union[int, str] == Union[str, int] + + * If :class:`Any` is present it is the sole survivor, e.g.:: + + Union[int, Any] == Any + + * You cannot subclass or instantiate a union. + + * You cannot write ``Union[X][Y]``. + + * You can use ``Optional[X]`` as a shorthand for ``Union[X, None]``. + +.. class:: Optional + + Optional type. + + ``Optional[X]`` is equivalent to ``Union[X, type(None)]``. + +.. class:: Tuple + + Tuple type; ``Tuple[X, Y]`` is the is the type of a tuple of two items + with the first item of type X and the second of type Y. + + Example: ``Tuple[T1, T2]`` is a tuple of two elements corresponding + to type variables T1 and T2. ``Tuple[int, float, str]`` is a tuple + of an int, a float and a string. + + To specify a variable-length tuple of homogeneous type, + use literal ellipsis, e.g. ``Tuple[int, ...]``. + +.. class:: Callable + + Callable type; ``Callable[[int], str]`` is a function of (int) -> str. + + The subscription syntax must always be used with exactly two + values: the argument list and the return type. The argument list + must be a list of types; the return type must be a single type. + + There is no syntax to indicate optional or keyword arguments, + such function types are rarely used as callback types. + ``Callable[..., ReturnType]`` could be used to type hint a callable + taking any number of arguments and returning ``ReturnType``. + A plain :class:`Callable` is equivalent to ``Callable[..., Any]``. + +.. class:: Generic + + Abstract base class for generic types. + + A generic type is typically declared by inheriting from an + instantiation of this class with one or more type variables. + For example, a generic mapping type might be defined as:: + + class Mapping(Generic[KT, VT]): + def __getitem__(self, key: KT) -> VT: + ... + # Etc. + + This class can then be used as follows:: + + X = TypeVar('X') + Y = TypeVar('Y') + + def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y: + try: + return mapping[key] + except KeyError: + return default + +.. class:: Iterable(Generic[T_co]) + + A generic version of the :class:`collections.abc.Iterable`. + +.. class:: Iterator(Iterable[T_co]) + + A generic version of the :class:`collections.abc.Iterator`. + +.. class:: SupportsInt + + An ABC with one abstract method ``__int__``. + +.. class:: SupportsFloat + + An ABC with one abstract method ``__float__``. + +.. class:: SupportsAbs + + An ABC with one abstract method ``__abs__`` that is covariant + in its return type. + +.. class:: SupportsRound + + An ABC with one abstract method ``__round__`` + that is covariant in its return type. + +.. class:: Reversible + + An ABC with one abstract method ``__reversed__`` returning + an ``Iterator[T_co]``. + +.. class:: Container(Generic[T_co]) + + A generic version of :class:`collections.abc.Container`. + +.. class:: AbstractSet(Sized, Iterable[T_co], Container[T_co]) + + A generic version of :class:`collections.abc.Set`. + +.. class:: MutableSet(AbstractSet[T]) + + A generic version of :class:`collections.abc.MutableSet`. + +.. class:: Mapping(Sized, Iterable[KT], Container[KT], Generic[VT_co]) + + A generic version of :class:`collections.abc.Mapping`. + +.. class:: MutableMapping(Mapping[KT, VT]) + + A generic version of :class:`collections.abc.MutableMapping`. + +.. class:: Sequence(Sized, Iterable[T_co], Container[T_co]) + + A generic version of :class:`collections.abc.Sequence`. + +.. class:: MutableSequence(Sequence[T]) + + A generic version of :class:`collections.abc.MutableSequence`. + +.. class:: ByteString(Sequence[int]) + + A generic version of :class:`collections.abc.ByteString`. + + This type represents the types :class:`bytes`, :class:`bytearray`, + and :class:`memoryview`. + + As a shorthand for this type, :class:`bytes` can be used to + annotate arguments of any of the types mentioned above. + +.. class:: List(list, MutableSequence[T]) + + Generic version of :class:`list`. + Useful for annotating return types. To annotate arguments it is preferred + to use abstract collection types such as :class:`Mapping`, :class:`Sequence`, + or :class:`AbstractSet`. + + This type may be used as follows:: + + T = TypeVar('T', int, float) + + def vec2(x: T, y: T) -> List[T]: + return [x, y] + + def slice__to_4(vector: Sequence[T]) -> List[T]: + return vector[0:4] + +.. class:: AbstractSet(set, MutableSet[T]) + + A generic version of :class:`collections.abc.Set`. + +.. class:: MappingView(Sized, Iterable[T_co]) + + A generic version of :class:`collections.abc.MappingView`. + +.. class:: KeysView(MappingView[KT_co], AbstractSet[KT_co]) + + A generic version of :class:`collections.abc.KeysView`. + +.. class:: ItemsView(MappingView, Generic[KT_co, VT_co]) + + A generic version of :class:`collections.abc.ItemsView`. + +.. class:: ValuesView(MappingView[VT_co]) + + A generic version of :class:`collections.abc.ValuesView`. + +.. class:: Dict(dict, MutableMapping[KT, VT]) + + A generic version of :class:`dict`. + The usage of this type is as follows:: + + def get_position_in_index(word_list: Dict[str, int], word: str) -> int: + return word_list[word] + +.. class:: Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) + +.. class:: io + + Wrapper namespace for I/O stream types. + + This defines the generic type ``IO[AnyStr]`` and aliases ``TextIO`` + and ``BinaryIO`` for respectively ``IO[str]`` and ``IO[bytes]``. + These representing the types of I/O streams such as returned by + :func:`open`. + +.. class:: re + + Wrapper namespace for regular expression matching types. + + This defines the type aliases ``Pattern`` and ``Match`` which + correspond to the return types from :func:`re.compile` and + :func:`re.match`. These types (and the corresponding functions) + are generic in ``AnyStr`` and can be made specific by writing + ``Pattern[str]``, ``Pattern[bytes]``, ``Match[str]``, or + ``Match[bytes]``. + +.. function:: NamedTuple(typename, fields) + + Typed version of namedtuple. + + Usage:: + + Employee = typing.NamedTuple('Employee', [('name', str), 'id', int)]) + + This is equivalent to:: + + Employee = collections.namedtuple('Employee', ['name', 'id']) + + The resulting class has one extra attribute: _field_types, + giving a dict mapping field names to types. (The field names + are in the _fields attribute, which is part of the namedtuple + API.) + +.. function:: cast(typ, val) + + Cast a value to a type. + + This returns the value unchanged. To the type checker this + signals that the return value has the designated type, but at + runtime we intentionally don't check anything (we want this + to be as fast as possible). + +.. function:: get_type_hints(obj) + + Return type hints for a function or method object. + + This is often the same as ``obj.__annotations__``, but it handles + forward references encoded as string literals, and if necessary + adds ``Optional[t]`` if a default value equal to None is set. + +.. decorator:: no_type_check(arg) + + Decorator to indicate that annotations are not type hints. + + The argument must be a class or function; if it is a class, it + applies recursively to all methods defined in that class (but not + to methods defined in its superclasses or subclasses). + + This mutates the function(s) in place. + +.. decorator:: no_type_check_decorator(decorator) + + Decorator to give another decorator the :func:`no_type_check` effect. + + This wraps the decorator with something that wraps the decorated + function in :func:`no_type_check`. diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index 3b3d3a0a8f..1430d9b403 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -15,8 +15,8 @@ This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in -this database is compiled from the `UCD version 6.3.0 -<http://www.unicode.org/Public/6.3.0/ucd>`_. +this database is compiled from the `UCD version 8.0.0 +<http://www.unicode.org/Public/8.0.0/ucd>`_. The module uses the same names and symbols as defined by Unicode Standard Annex #44, `"Unicode Character Database" @@ -166,6 +166,6 @@ Examples: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NameAliases.txt -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NamedSequences.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NamedSequences.txt diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index 48eefe900f..1b271c83bd 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -198,7 +198,7 @@ a :class:`MagicMock` for you. You can specify an alternative class of :class:`Mo the *new_callable* argument to :func:`patch`. -.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, **kwargs) +.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, unsafe=False, **kwargs) Create a new :class:`Mock` object. :class:`Mock` takes several optional arguments that specify the behaviour of the Mock object: @@ -235,6 +235,12 @@ the *new_callable* argument to :func:`patch`. this is a new Mock (created on first access). See the :attr:`return_value` attribute. + * *unsafe*: By default if any attribute starts with *assert* or + *assret* will raise an :exc:`AttributeError`. Passing ``unsafe=True`` + will allow access to these attributes. + + .. versionadded:: 3.5 + * *wraps*: Item for the mock object to wrap. If *wraps* is not None then calling the Mock will pass the call through to the wrapped object (returning the real result). Attribute access on the mock will return a @@ -315,6 +321,20 @@ the *new_callable* argument to :func:`patch`. >>> calls = [call(4), call(2), call(3)] >>> mock.assert_has_calls(calls, any_order=True) + .. method:: assert_not_called(*args, **kwargs) + + Assert the mock was never called. + + >>> m = Mock() + >>> m.hello.assert_not_called() + >>> obj = m.hello() + >>> m.hello.assert_not_called() + Traceback (most recent call last): + ... + AssertionError: Expected 'hello' to not have been called. Called 1 times. + + .. versionadded:: 3.5 + .. method:: reset_mock() @@ -1032,6 +1052,12 @@ patch default because it can be dangerous. With it switched on you can write passing tests against APIs that don't actually exist! + .. note:: + + .. versionchanged:: 3.5 + If you are patching builtins in a module then you don't + need to pass ``create=True``, it will be added by default. + Patch can be used as a :class:`TestCase` class decorator. It works by decorating each test method in the class. This reduces the boilerplate code when your test methods share a common patchings set. :func:`patch` finds @@ -1403,6 +1429,22 @@ It is also possible to stop all patches which have been started by using Stop all active patches. Only stops patches started with ``start``. +.. _patch-builtins: + +patch builtins +~~~~~~~~~~~~~~ +You can patch any builtins within a module. The following example patches +builtin :func:`ord`: + + >>> @patch('__main__.ord') + ... def test(mock_ord): + ... mock_ord.return_value = 101 + ... print(ord('c')) + ... + >>> test() + 101 + + TEST_PREFIX ~~~~~~~~~~~ @@ -1587,7 +1629,7 @@ The full list of supported magic methods is: * Context manager: ``__enter__`` and ``__exit__`` * Unary numeric methods: ``__neg__``, ``__pos__`` and ``__invert__`` * The numeric methods (including right hand and in-place variants): - ``__add__``, ``__sub__``, ``__mul__``, ``__div__``, ``__truediv__``, + ``__add__``, ``__sub__``, ``__mul__``, ``__matmul__``, ``__div__``, ``__truediv__``, ``__floordiv__``, ``__mod__``, ``__divmod__``, ``__lshift__``, ``__rshift__``, ``__and__``, ``__xor__``, ``__or__``, and ``__pow__`` * Numeric conversion methods: ``__complex__``, ``__int__``, ``__float__`` @@ -2006,7 +2048,7 @@ mock_open The mock of :meth:`~io.IOBase.read` changed to consume *read_data* rather than returning it on each call. - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5 *read_data* is now reset on each call to the *mock*. Using :func:`open` as a context manager is a great way to ensure your file handles @@ -2023,7 +2065,7 @@ Mocking context managers with a :class:`MagicMock` is common enough and fiddly enough that a helper function is useful. >>> m = mock_open() - >>> with patch('__main__.open', m, create=True): + >>> with patch('__main__.open', m): ... with open('foo', 'w') as h: ... h.write('some stuff') ... @@ -2038,7 +2080,7 @@ enough that a helper function is useful. And for reading files: - >>> with patch('__main__.open', mock_open(read_data='bibble'), create=True) as m: + >>> with patch('__main__.open', mock_open(read_data='bibble')) as m: ... with open('foo') as h: ... result = h.read() ... diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 1930f785a7..d3a3649f84 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -214,9 +214,16 @@ Command-line options Stop the test run on the first error or failure. +.. cmdoption:: --locals + + Show local variables in tracebacks. + .. versionadded:: 3.2 The command-line options ``-b``, ``-c`` and ``-f`` were added. +.. versionadded:: 3.5 + The command-line option ``--locals``. + The command line can also be used for test discovery, for running all of the tests in a project or just a subset. @@ -1541,6 +1548,20 @@ Loading and running tests :data:`unittest.defaultTestLoader`. Using a subclass or instance, however, allows customization of some configurable properties. + :class:`TestLoader` objects have the following attributes: + + + .. attribute:: errors + + A list of the non-fatal errors encountered while loading tests. Not reset + by the loader at any point. Fatal errors are signalled by the relevant + a method raising an exception to the caller. Non-fatal errors are also + indicated by a synthetic test that will raise the original error when + run. + + .. versionadded:: 3.5 + + :class:`TestLoader` objects have the following methods: @@ -1556,7 +1577,7 @@ Loading and running tests case is created for that method instead. - .. method:: loadTestsFromModule(module) + .. method:: loadTestsFromModule(module, pattern=None) Return a suite of all tests cases contained in the given module. This method searches *module* for classes derived from :class:`TestCase` and @@ -1573,11 +1594,18 @@ Loading and running tests If a module provides a ``load_tests`` function it will be called to load the tests. This allows modules to customize test loading. - This is the `load_tests protocol`_. + This is the `load_tests protocol`_. The *pattern* argument is passed as + the third argument to ``load_tests``. .. versionchanged:: 3.2 Support for ``load_tests`` added. + .. versionchanged:: 3.5 + The undocumented and unofficial *use_load_tests* default argument is + deprecated and ignored, although it is still accepted for backward + compatibility. The method also now accepts a keyword-only argument + *pattern* which is passed to ``load_tests`` as the third argument. + .. method:: loadTestsFromName(name, module=None) @@ -1603,6 +1631,12 @@ Loading and running tests The method optionally resolves *name* relative to the given *module*. + .. versionchanged:: 3.5 + If an :exc:`ImportError` or :exc:`AttributeError` occurs while traversing + *name* then a synthetic test that raises that error when run will be + returned. These errors are included in the errors accumulated by + self.errors. + .. method:: loadTestsFromNames(names, module=None) @@ -1629,18 +1663,22 @@ Loading and running tests the start directory is not the top level directory then the top level directory must be specified separately. - If importing a module fails, for example due to a syntax error, then this - will be recorded as a single error and discovery will continue. If the - import failure is due to :exc:`SkipTest` being raised, it will be recorded - as a skip instead of an error. + If importing a module fails, for example due to a syntax error, then + this will be recorded as a single error and discovery will continue. If + the import failure is due to :exc:`SkipTest` being raised, it will be + recorded as a skip instead of an error. - If a test package name (directory with :file:`__init__.py`) matches the - pattern then the package will be checked for a ``load_tests`` - function. If this exists then it will be called with *loader*, *tests*, - *pattern*. + If a package (a directory containing a file named :file:`__init__.py`) is + found, the package will be checked for a ``load_tests`` function. If this + exists then it will be called + ``package.load_tests(loader, tests, pattern)``. Test discovery takes care + to ensure that a package is only checked for tests once during an + invocation, even if the load_tests function itself calls + ``loader.discover``. - If ``load_tests`` exists then discovery does *not* recurse into the package, - ``load_tests`` is responsible for loading all tests in the package. + If ``load_tests`` exists then discovery does *not* recurse into the + package, ``load_tests`` is responsible for loading all tests in the + package. The pattern is deliberately not stored as a loader attribute so that packages can continue discovery themselves. *top_level_dir* is stored so @@ -1659,6 +1697,11 @@ Loading and running tests the same even if the underlying file system's ordering is not dependent on file name. + .. versionchanged:: 3.5 + Found packages are now checked for ``load_tests`` regardless of + whether their path matches *pattern*, because it is impossible for + a package name to match the default pattern. + The following attributes of a :class:`TestLoader` can be configured either by subclassing or assignment on an instance: @@ -1741,12 +1784,10 @@ Loading and running tests Set to ``True`` when the execution of tests should stop by :meth:`stop`. - .. attribute:: testsRun The total number of tests run so far. - .. attribute:: buffer If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between @@ -1756,7 +1797,6 @@ Loading and running tests .. versionadded:: 3.2 - .. attribute:: failfast If set to true :meth:`stop` will be called on the first failure or error, @@ -1764,6 +1804,11 @@ Loading and running tests .. versionadded:: 3.2 + .. attribute:: tb_locals + + If set to true then local variables will be shown in tracebacks. + + .. versionadded:: 3.5 .. method:: wasSuccessful() @@ -1774,7 +1819,6 @@ Loading and running tests Returns ``False`` if there were any :attr:`unexpectedSuccesses` from tests marked with the :func:`expectedFailure` decorator. - .. method:: stop() This method can be called to signal that the set of tests being run should @@ -1906,12 +1950,14 @@ Loading and running tests .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, failfast=False, \ - buffer=False, resultclass=None, warnings=None) + buffer=False, resultclass=None, warnings=None, *, tb_locals=False) A basic test runner implementation that outputs results to a stream. If *stream* is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class has a few configurable parameters, but is essentially very simple. Graphical - applications which run test suites should provide alternate implementations. + applications which run test suites should provide alternate implementations. Such + implementations should accept ``**kwargs`` as the interface to construct runners + changes when features are added to unittest. By default this runner shows :exc:`DeprecationWarning`, :exc:`PendingDeprecationWarning`, :exc:`ResourceWarning` and @@ -1930,6 +1976,9 @@ Loading and running tests The default stream is set to :data:`sys.stderr` at instantiation time rather than import time. + .. versionchanged:: 3.5 + Added the tb_locals parameter. + .. method:: _makeResult() This method returns the instance of ``TestResult`` used by :meth:`run`. @@ -2027,7 +2076,10 @@ test runs or test discovery by implementing a function called ``load_tests``. If a test module defines ``load_tests`` it will be called by :meth:`TestLoader.loadTestsFromModule` with the following arguments:: - load_tests(loader, standard_tests, None) + load_tests(loader, standard_tests, pattern) + +where *pattern* is passed straight through from ``loadTestsFromModule``. It +defaults to ``None``. It should return a :class:`TestSuite`. @@ -2049,21 +2101,12 @@ A typical ``load_tests`` function that loads tests from a specific set of suite.addTests(tests) return suite -If discovery is started, either from the command line or by calling -:meth:`TestLoader.discover`, with a pattern that matches a package -name then the package :file:`__init__.py` will be checked for ``load_tests``. - -.. note:: - - The default pattern is ``'test*.py'``. This matches all Python files - that start with ``'test'`` but *won't* match any test directories. - - A pattern like ``'test*'`` will match test packages as well as - modules. - -If the package :file:`__init__.py` defines ``load_tests`` then it will be -called and discovery not continued into the package. ``load_tests`` -is called with the following arguments:: +If discovery is started in a directory containing a package, either from the +command line or by calling :meth:`TestLoader.discover`, then the package +:file:`__init__.py` will be checked for ``load_tests``. If that function does +not exist, discovery will recurse into the package as though it were just +another directory. Otherwise, discovery of the package's tests will be left up +to ``load_tests`` which is called with the following arguments:: load_tests(loader, standard_tests, pattern) @@ -2082,6 +2125,11 @@ continue (and potentially modify) test discovery. A 'do nothing' standard_tests.addTests(package_tests) return standard_tests +.. versionchanged:: 3.5 + Discovery no longer checks package names for matching *pattern* due to the + impossibility of package names matching the default pattern. + + Class and Module Fixtures ------------------------- diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index fbbabcadf9..40098d0496 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -270,6 +270,11 @@ or on combining URL components into a URL string. :func:`urlunsplit`, removing possible *scheme* and *netloc* parts. + .. versionchanged:: 3.5 + + Behaviour updated to match the semantics defined in :rfc:`3986`. + + .. function:: urldefrag(url) If *url* contains a fragment identifier, return a modified version of *url* @@ -516,7 +521,8 @@ task isn't already covered by the URL parsing functions above. Example: ``unquote_to_bytes('a%26%EF')`` yields ``b'a&\xef'``. -.. function:: urlencode(query, doseq=False, safe='', encoding=None, errors=None) +.. function:: urlencode(query, doseq=False, safe='', encoding=None, \ + errors=None, quote_via=quote_plus) Convert a mapping object or a sequence of two-element tuples, which may contain :class:`str` or :class:`bytes` objects, to a "percent-encoded" @@ -525,8 +531,16 @@ task isn't already covered by the URL parsing functions above. properly encoded to bytes, otherwise it would result in a :exc:`TypeError`. The resulting string is a series of ``key=value`` pairs separated by ``'&'`` - characters, where both *key* and *value* are quoted using :func:`quote_plus` - above. When a sequence of two-element tuples is used as the *query* + characters, where both *key* and *value* are quoted using the *quote_via* + function. By default, :func:`quote_plus` is used to quote the values, which + means spaces are quoted as a ``'+'`` character and '/' characters are + encoded as ``%2F``, which follows the standard for GET requests + (``application/x-www-form-urlencoded``). An alternate function that can be + passed as *quote_via* is :func:`quote`, which will encode spaces as ``%20`` + and not encode '/' characters. For maximum control of what is quoted, use + ``quote`` and specify a value for *safe*. + + When a sequence of two-element tuples is used as the *query* argument, the first element of each tuple is a key and the second is a value. The value element in itself can be a sequence and in that case, if the optional parameter *doseq* is evaluates to *True*, individual @@ -535,7 +549,7 @@ task isn't already covered by the URL parsing functions above. string will match the order of parameter tuples in the sequence. The *safe*, *encoding*, and *errors* parameters are passed down to - :func:`quote_plus` (the *encoding* and *errors* parameters are only passed + *quote_via* (the *encoding* and *errors* parameters are only passed when a query element is a :class:`str`). To reverse this encoding process, :func:`parse_qs` and :func:`parse_qsl` are @@ -547,6 +561,9 @@ task isn't already covered by the URL parsing functions above. .. versionchanged:: 3.2 Query parameter supports bytes and string objects. + .. versionadded:: 3.5 + *quote_via* parameter. + .. seealso:: diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 75b95d9b4a..349ba70d36 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -115,6 +115,7 @@ The :mod:`urllib.request` module defines the following functions: .. versionchanged:: 3.4.3 *context* was added. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. @@ -287,13 +288,37 @@ The following classes are provided: fits. +.. class:: HTTPPasswordMgrWithPriorAuth() + + A variant of :class:`HTTPPasswordMgrWithDefaultRealm` that also has a + database of ``uri -> is_authenticated`` mappings. Can be used by a + BasicAuth handler to determine when to send authentication credentials + immediately instead of waiting for a ``401`` response first. + + .. versionadded:: 3.5 + + .. class:: AbstractBasicAuthHandler(password_mgr=None) This is a mixin class that helps with HTTP authentication, both to the remote host and to a proxy. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be - supported. + supported. If *passwd_mgr* also provides ``is_authenticated`` and + ``update_authenticated`` methods (see + :ref:`http-password-mgr-with-prior-auth`), then the handler will use the + ``is_authenticated`` result for a given URI to determine whether or not to + send authentication credentials with the request. If ``is_authenticated`` + returns ``True`` for the URI, credentials are sent. If ``is_authenticated`` + is ``False``, credentials are not sent, and then if a ``401`` response is + received the request is re-sent with the authentication credentials. If + authentication succeeds, ``update_authenticated`` is called to set + ``is_authenticated`` ``True`` for the URI, so that subsequent requests to + the URI or any of its super-URIs will automatically include the + authentication credentials. + + .. versionadded:: 3.5 + Added ``is_authenticated`` support. .. class:: HTTPBasicAuthHandler(password_mgr=None) @@ -845,6 +870,42 @@ These methods are available on :class:`HTTPPasswordMgr` and searched if the given *realm* has no matching user/password. +.. _http-password-mgr-with-prior-auth: + +HTTPPasswordMgrWithPriorAuth Objects +------------------------------------ + +This password manager extends :class:`HTTPPasswordMgrWithDefaultRealm` to support +tracking URIs for which authentication credentials should always be sent. + + +.. method:: HTTPPasswordMgrWithPriorAuth.add_password(realm, uri, user, \ + passwd, is_authenticated=False) + + *realm*, *uri*, *user*, *passwd* are as for + :meth:`HTTPPasswordMgr.add_password`. *is_authenticated* sets the initial + value of the ``is_authenticated`` flag for the given URI or list of URIs. + If *is_authenticated* is specified as ``True``, *realm* is ignored. + + +.. method:: HTTPPasswordMgr.find_user_password(realm, authuri) + + Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects + + +.. method:: HTTPPasswordMgrWithPriorAuth.update_authenticated(self, uri, \ + is_authenticated=False) + + Update the ``is_authenticated`` flag for the given *uri* or list + of URIs. + + +.. method:: HTTPPasswordMgrWithPriorAuth.is_authenticated(self, authuri) + + Returns the current state of the ``is_authenticated`` flag for + the given URI. + + .. _abstract-basic-auth-handler: AbstractBasicAuthHandler Objects diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index cc883b1b51..2e16077b8a 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -258,7 +258,7 @@ These method have the same issues as the and :meth:`keyrefs` method of are called in reverse order of creation. A finalizer will never invoke its callback during the later part of - the interpreter shutdown when module globals are liable to have + the :term:`interpreter shutdown` when module globals are liable to have been replaced by :const:`None`. .. method:: __call__() @@ -527,8 +527,8 @@ follows:: Starting with Python 3.4, :meth:`__del__` methods no longer prevent reference cycles from being garbage collected, and module globals are -no longer forced to :const:`None` during interpreter shutdown. So this -code should work without any issues on CPython. +no longer forced to :const:`None` during :term:`interpreter shutdown`. +So this code should work without any issues on CPython. However, handling of :meth:`__del__` methods is notoriously implementation specific, since it depends on internal details of the interpreter's garbage diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index a9e19dad98..de74c30250 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -184,10 +184,11 @@ This module provides a single class, :class:`Headers`, for convenient manipulation of WSGI response headers using a mapping-like interface. -.. class:: Headers(headers) +.. class:: Headers([headers]) Create a mapping-like object wrapping *headers*, which must be a list of header - name/value tuples as described in :pep:`3333`. + name/value tuples as described in :pep:`3333`. The default value of *headers* is + an empty list. :class:`Headers` objects support typical mapping operations including :meth:`__getitem__`, :meth:`get`, :meth:`__setitem__`, :meth:`setdefault`, @@ -251,6 +252,10 @@ manipulation of WSGI response headers using a mapping-like interface. Content-Disposition: attachment; filename="bud.gif" + .. versionchanged:: 3.5 + *headers* parameter is optional. + + :mod:`wsgiref.simple_server` -- a simple WSGI HTTP server --------------------------------------------------------- diff --git a/Doc/library/xml.sax.reader.rst b/Doc/library/xml.sax.reader.rst index 3ab60636c5..31ca260f7c 100644 --- a/Doc/library/xml.sax.reader.rst +++ b/Doc/library/xml.sax.reader.rst @@ -100,8 +100,10 @@ The :class:`XMLReader` interface supports the following methods: system identifier (a string identifying the input source -- typically a file name or an URL), a file-like object, or an :class:`InputSource` object. When :meth:`parse` returns, the input is completely processed, and the parser object - can be discarded or reset. As a limitation, the current implementation only - accepts byte streams; processing of character streams is for further study. + can be discarded or reset. + + .. versionchanged:: 3.5 + Added support of character streams. .. method:: XMLReader.getContentHandler() @@ -288,8 +290,7 @@ InputSource Objects .. method:: InputSource.setByteStream(bytefile) - Set the byte stream (a Python file-like object which does not perform - byte-to-character conversion) for this input source. + Set the byte stream (a :term:`binary file`) for this input source. The SAX parser will ignore this if there is also a character stream specified, but it will use a byte stream in preference to opening a URI connection itself. @@ -308,8 +309,7 @@ InputSource Objects .. method:: InputSource.setCharacterStream(charfile) - Set the character stream for this input source. (The stream must be a Python 1.6 - Unicode-wrapped file-like that performs conversion to strings.) + Set the character stream (a :term:`text file`) for this input source. If there is a character stream specified, the SAX parser will ignore any byte stream and will not attempt to open a URI connection to the system identifier. diff --git a/Doc/library/xml.sax.rst b/Doc/library/xml.sax.rst index e95d6b0f85..55f97999de 100644 --- a/Doc/library/xml.sax.rst +++ b/Doc/library/xml.sax.rst @@ -47,7 +47,11 @@ The convenience functions are: .. function:: parseString(string, handler, error_handler=handler.ErrorHandler()) Similar to :func:`parse`, but parses from a buffer *string* received as a - parameter. + parameter. *string* must be a :class:`str` instance or a + :term:`bytes-like object`. + + .. versionchanged:: 3.5 + Added support of :class:`str` instances. A typical SAX application uses three kinds of objects: readers, handlers and input sources. "Reader" in this context is another term for parser, i.e. some diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index cc5e83a706..e199931b55 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -27,7 +27,7 @@ between conformable Python objects and XML on the wire. constructed data. If you need to parse untrusted or unauthenticated data see :ref:`xml-vulnerabilities`. -.. versionchanged:: 3.4.3 +.. versionchanged:: 3.5 For https URIs, :mod:`xmlrpc.client` now performs all the necessary certificate and hostname checks by default @@ -129,7 +129,7 @@ between conformable Python objects and XML on the wire. :class:`Server` is retained as an alias for :class:`ServerProxy` for backwards compatibility. New code should use :class:`ServerProxy`. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Added the *context* argument. @@ -200,6 +200,11 @@ grouped under the reserved :attr:`system` attribute: no such string is available, an empty string is returned. The documentation string may contain HTML markup. +.. versionchanged:: 3.5 + + Instances of :class:`ServerProxy` support the :term:`context manager` protocol + for closing the underlying transport. + A working example follows. The server code:: @@ -217,9 +222,9 @@ The client code for the preceding server:: import xmlrpc.client - proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") - print("3 is even: %s" % str(proxy.is_even(3))) - print("100 is even: %s" % str(proxy.is_even(100))) + with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy: + print("3 is even: %s" % str(proxy.is_even(3))) + print("100 is even: %s" % str(proxy.is_even(100))) .. _datetime-objects: @@ -527,14 +532,14 @@ Example of Client Usage from xmlrpc.client import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server - server = ServerProxy("http://betty.userland.com") + with ServerProxy("http://betty.userland.com") as proxy: - print(server) + print(proxy) - try: - print(server.examples.getStateName(41)) - except Error as v: - print("ERROR", v) + try: + print(proxy.examples.getStateName(41)) + except Error as v: + print("ERROR", v) To access an XML-RPC server through a proxy, you need to define a custom transport. The following example shows how: diff --git a/Doc/library/zipapp.rst b/Doc/library/zipapp.rst new file mode 100644 index 0000000000..b4484c87d0 --- /dev/null +++ b/Doc/library/zipapp.rst @@ -0,0 +1,259 @@ +:mod:`zipapp` --- Manage executable python zip archives +======================================================= + +.. module:: zipapp + :synopsis: Manage executable python zip archives + + +.. index:: + single: Executable Zip Files + +.. versionadded:: 3.5 + +**Source code:** :source:`Lib/zipapp.py` + +-------------- + +This module provides tools to manage the creation of zip files containing +Python code, which can be :ref:`executed directly by the Python interpreter +<using-on-interface-options>`. The module provides both a +:ref:`zipapp-command-line-interface` and a :ref:`zipapp-python-api`. + + +Basic Example +------------- + +The following example shows how the :ref:`command-line-interface` +can be used to create an executable archive from a directory containing +Python code. When run, the archive will execute the ``main`` function from +the module ``myapp`` in the archive. + +.. code-block:: sh + + $ python -m zipapp myapp -m "myapp:main" + $ python myapp.pyz + <output from myapp> + + +.. _zipapp-command-line-interface: + +Command-Line Interface +---------------------- + +When called as a program from the command line, the following form is used: + +.. code-block:: sh + + $ python -m zipapp source [options] + +If *source* is a directory, this will create an archive from the contents of +*source*. If *source* is a file, it should be an archive, and it will be +copied to the target archive (or the contents of its shebang line will be +displayed if the --info option is specified). + +The following options are understood: + +.. program:: zipapp + +.. cmdoption:: -o <output>, --output=<output> + + Write the output to a file named *output*. If this option is not specified, + the output filename will be the same as the input *source*, with the + extension ``.pyz`` added. If an explicit filename is given, it is used as + is (so a ``.pyz`` extension should be included if required). + + An output filename must be specified if the *source* is an archive (and in + that case, *output* must not be the same as *source*). + +.. cmdoption:: -p <interpreter>, --python=<interpreter> + + Add a ``#!`` line to the archive specifying *interpreter* as the command + to run. Also, on POSIX, make the archive executable. The default is to + write no ``#!`` line, and not make the file executable. + +.. cmdoption:: -m <mainfn>, --main=<mainfn> + + Write a ``__main__.py`` file to the archive that executes *mainfn*. The + *mainfn* argument should have the form "pkg.mod:fn", where "pkg.mod" is a + package/module in the archive, and "fn" is a callable in the given module. + The ``__main__.py`` file will execute that callable. + + :option:`--main` cannot be specified when copying an archive. + +.. cmdoption:: --info + + Display the interpreter embedded in the archive, for diagnostic purposes. In + this case, any other options are ignored and SOURCE must be an archive, not a + directory. + +.. cmdoption:: -h, --help + + Print a short usage message and exit. + + +.. _zipapp-python-api: + +Python API +---------- + +The module defines two convenience functions: + + +.. function:: create_archive(source, target=None, interpreter=None, main=None) + + Create an application archive from *source*. The source can be any + of the following: + + * The name of a directory, or a :class:`pathlib.Path` object referring + to a directory, in which case a new application archive will be + created from the content of that directory. + * The name of an existing application archive file, or a :class:`pathlib.Path` + object referring to such a file, in which case the file is copied to + the target (modifying it to reflect the value given for the *interpreter* + argument). The file name should include the ``.pyz`` extension, if required. + * A file object open for reading in bytes mode. The content of the + file should be an application archive, and the file object is + assumed to be positioned at the start of the archive. + + The *target* argument determines where the resulting archive will be + written: + + * If it is the name of a file, or a :class:`pathlb.Path` object, + the archive will be written to that file. + * If it is an open file object, the archive will be written to that + file object, which must be open for writing in bytes mode. + * If the target is omitted (or None), the source must be a directory + and the target will be a file with the same name as the source, with + a ``.pyz`` extension added. + + The *interpreter* argument specifies the name of the Python + interpreter with which the archive will be executed. It is written as + a "shebang" line at the start of the archive. On POSIX, this will be + interpreted by the OS, and on Windows it will be handled by the Python + launcher. Omitting the *interpreter* results in no shebang line being + written. If an interpreter is specified, and the target is a + filename, the executable bit of the target file will be set. + + The *main* argument specifies the name of a callable which will be + used as the main program for the archive. It can only be specified if + the source is a directory, and the source does not already contain a + ``__main__.py`` file. The *main* argument should take the form + "pkg.module:callable" and the archive will be run by importing + "pkg.module" and executing the given callable with no arguments. It + is an error to omit *main* if the source is a directory and does not + contain a ``__main__.py`` file, as otherwise the resulting archive + would not be executable. + + If a file object is specified for *source* or *target*, it is the + caller's responsibility to close it after calling create_archive. + + When copying an existing archive, file objects supplied only need + ``read`` and ``readline``, or ``write`` methods. When creating an + archive from a directory, if the target is a file object it will be + passed to the ``zipfile.ZipFile`` class, and must supply the methods + needed by that class. + +.. function:: get_interpreter(archive) + + Return the interpreter specified in the ``#!`` line at the start of the + archive. If there is no ``#!`` line, return :const:`None`. + The *archive* argument can be a filename or a file-like object open + for reading in bytes mode. It is assumed to be at the start of the archive. + + +.. _zipapp-examples: + +Examples +-------- + +Pack up a directory into an archive, and run it. + +.. code-block:: sh + + $ python -m zipapp myapp + $ python myapp.pyz + <output from myapp> + +The same can be done using the :func:`create_archive` functon:: + + >>> import zipapp + >>> zipapp.create_archive('myapp.pyz', 'myapp') + +To make the application directly executable on POSIX, specify an interpreter +to use. + +.. code-block:: sh + + $ python -m zipapp myapp -p "/usr/bin/env python" + $ ./myapp.pyz + <output from myapp> + +To replace the shebang line on an existing archive, create a modified archive +using the :func:`create_archive` function:: + + >>> import zipapp + >>> zipapp.create_archive('old_archive.pyz', 'new_archive.pyz', '/usr/bin/python3') + +To update the file in place, do the replacement in memory using a :class:`BytesIO` +object, and then overwrite the source afterwards. Note that there is a risk +when overwriting a file in place that an error will result in the loss of +the original file. This code does not protect against such errors, but +production code should do so. Also, this method will only work if the archive +fits in memory:: + + >>> import zipapp + >>> import io + >>> temp = io.BytesIO() + >>> zipapp.create_archive('myapp.pyz', temp, '/usr/bin/python2') + >>> with open('myapp.pyz', 'wb') as f: + >>> f.write(temp.getvalue()) + +Note that if you specify an interpreter and then distribute your application +archive, you need to ensure that the interpreter used is portable. The Python +launcher for Windows supports most common forms of POSIX ``#!`` line, but there +are other issues to consider: + +* If you use "/usr/bin/env python" (or other forms of the "python" command, + such as "/usr/bin/python"), you need to consider that your users may have + either Python 2 or Python 3 as their default, and write your code to work + under both versions. +* If you use an explicit version, for example "/usr/bin/env python3" your + application will not work for users who do not have that version. (This + may be what you want if you have not made your code Python 2 compatible). +* There is no way to say "python X.Y or later", so be careful of using an + exact version like "/usr/bin/env python3.4" as you will need to change your + shebang line for users of Python 3.5, for example. + +The Python Zip Application Archive Format +----------------------------------------- + +Python has been able to execute zip files which contain a ``__main__.py`` file +since version 2.6. In order to be executed by Python, an application archive +simply has to be a standard zip file containing a ``__main__.py`` file which +will be run as the entry point for the application. As usual for any Python +script, the parent of the script (in this case the zip file) will be placed on +:data:`sys.path` and thus further modules can be imported from the zip file. + +The zip file format allows arbitrary data to be prepended to a zip file. The +zip application format uses this ability to prepend a standard POSIX "shebang" +line to the file (``#!/path/to/interpreter``). + +Formally, the Python zip application format is therefore: + +1. An optional shebang line, containing the characters ``b'#!'`` followed by an + interpreter name, and then a newline (``b'\n'``) character. The interpreter + name can be anything acceptable to the OS "shebang" processing, or the Python + launcher on Windows. The interpreter should be encoded in UTF-8 on Windows, + and in :func:`sys.getfilesystemencoding()` on POSIX. +2. Standard zipfile data, as generated by the :mod:`zipfile` module. The + zipfile content *must* include a file called ``__main__.py`` (which must be + in the "root" of the zipfile - i.e., it cannot be in a subdirectory). The + zipfile data can be compressed or uncompressed. + +If an application archive has a shebang line, it may have the executable bit set +on POSIX systems, to allow it to be executed directly. + +There is no requirement that the tools in this module are used to create +application archives - the module is a convenience, but archives in the above +format created by any means are acceptable to Python. + diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 10a094f8b1..d40315eaf8 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -134,12 +134,16 @@ ZipFile Objects Open a ZIP file, where *file* can be either a path to a file (a string) or a file-like object. The *mode* parameter should be ``'r'`` to read an existing - file, ``'w'`` to truncate and write a new file, or ``'a'`` to append to an - existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP + file, ``'w'`` to truncate and write a new file, ``'x'`` to exclusive create + and write a new file, or ``'a'`` to append to an existing file. + If *mode* is ``'x'`` and *file* refers to an existing file, + a :exc:`FileExistsError` will be raised. + If *mode* is ``'a'`` and *file* refers to an existing ZIP file, then additional files are added to it. If *file* does not refer to a ZIP file, then a new ZIP archive is appended to the file. This is meant for adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``a`` and the file does not exist at all, it is created. + If *mode* is ``r`` or ``a``, the file should be seekable. *compression* is the ZIP compression method to use when writing the archive, and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized @@ -151,7 +155,7 @@ ZipFile Objects extensions when the zipfile is larger than 2 GiB. If it is false :mod:`zipfile` will raise an exception when the ZIP file would require ZIP64 extensions. - If the file is created with mode ``'a'`` or ``'w'`` and then + If the file is created with mode ``'w'``, ``'x'`` or ``'a'`` and then :meth:`closed <close>` without adding any files to the archive, the appropriate ZIP structures for an empty archive will be written to the file. @@ -171,6 +175,10 @@ ZipFile Objects .. versionchanged:: 3.4 ZIP64 extensions are enabled by default. + .. versionchanged:: 3.5 + Added support for writing to unseekable streams. + Added support for the ``'x'`` mode. + .. method:: ZipFile.close() @@ -226,14 +234,8 @@ ZipFile Objects .. note:: - If the ZipFile was created by passing in a file-like object as the first - argument to the constructor, then the object returned by :meth:`.open` shares the - ZipFile's file pointer. Under these circumstances, the object returned by - :meth:`.open` should not be used after any additional operations are performed - on the ZipFile object. If the ZipFile was created by passing in a string (the - filename) as the first argument to the constructor, then :meth:`.open` will - create a new file object that will be held by the ZipExtFile, allowing it to - operate independently of the ZipFile. + Objects returned by :meth:`.open` can operate independently of the + ZipFile. .. note:: @@ -318,7 +320,8 @@ ZipFile Objects *arcname* (by default, this will be the same as *filename*, but without a drive letter and with leading path separators removed). If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for - the new entry. The archive must be open with mode ``'w'`` or ``'a'`` -- calling + the new entry. + The archive must be open with mode ``'w'``, ``'x'`` or ``'a'`` -- calling :meth:`write` on a ZipFile created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling :meth:`write` on a closed ZipFile will raise a :exc:`RuntimeError`. @@ -340,16 +343,16 @@ ZipFile Objects If ``arcname`` (or ``filename``, if ``arcname`` is not given) contains a null byte, the name of the file in the archive will be truncated at the null byte. - .. method:: ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type]) Write the string *bytes* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's an instance, at least the filename, date, and time must be given. If it's a - name, the date and time is set to the current date and time. The archive must be - opened with mode ``'w'`` or ``'a'`` -- calling :meth:`writestr` on a ZipFile - created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling - :meth:`writestr` on a closed ZipFile will raise a :exc:`RuntimeError`. + name, the date and time is set to the current date and time. + The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'`` -- calling + :meth:`writestr` on a ZipFile created with mode ``'r'`` will raise a + :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will + raise a :exc:`RuntimeError`. If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for the new entry, or in the *zinfo_or_arcname* @@ -377,7 +380,8 @@ The following data attributes are also available: .. attribute:: ZipFile.comment The comment text associated with the ZIP file. If assigning a comment to a - :class:`ZipFile` instance created with mode 'a' or 'w', this should be a + :class:`ZipFile` instance created with mode ``'w'``, ``'x'`` or ``'a'``, + this should be a string no longer than 65535 bytes. Comments longer than this will be truncated in the written archive when :meth:`close` is called. @@ -407,8 +411,7 @@ The :class:`PyZipFile` constructor takes the same parameters as the archive. If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``, - the corresponding file is a :file:`\*.pyo` file if available, else a - :file:`\*.pyc` file, compiling if necessary. + the corresponding file is a :file:`\*.pyc` file, compiling if necessary. If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or ``2``, only files with that optimization level (see :func:`compile`) are @@ -571,4 +574,3 @@ Instances have the following attributes: .. attribute:: ZipInfo.file_size Size of the uncompressed file. - diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 2cf508b85c..8a5d5d19df 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -20,10 +20,10 @@ subdirectory. For example, the path :file:`example.zip/lib/` would only import from the :file:`lib/` subdirectory within the archive. Any files may be present in the ZIP archive, but only files :file:`.py` and -:file:`.py[co]` are available for import. ZIP import of dynamic modules +:file:`.pyc` are available for import. ZIP import of dynamic modules (:file:`.pyd`, :file:`.so`) is disallowed. Note that if an archive only contains :file:`.py` files, Python will not attempt to modify the archive by adding the -corresponding :file:`.pyc` or :file:`.pyo` file, meaning that if a ZIP archive +corresponding :file:`.pyc` file, meaning that if a ZIP archive doesn't contain :file:`.pyc` files, importing may be rather slow. ZIP archives with an archive comment are currently not supported. @@ -161,4 +161,3 @@ Here is an example that imports a module from a ZIP archive - note that the >>> import jwzthreading >>> jwzthreading.__file__ 'example.zip/jwzthreading.py' - diff --git a/Doc/make.bat b/Doc/make.bat index 251f822abf..5ab80850a3 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -8,9 +8,23 @@ set this=%~n0 if "%SPHINXBUILD%" EQU "" set SPHINXBUILD=sphinx-build
if "%PYTHON%" EQU "" set PYTHON=py
-if DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles(x86)%
-if NOT DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles%
-if "%HTMLHELP%" EQU "" set HTMLHELP=%_PRGMFLS%\HTML Help Workshop\hhc.exe
+if "%1" NEQ "htmlhelp" goto :skiphhcsearch
+if exist "%HTMLHELP%" goto :skiphhcsearch
+
+rem Search for HHC in likely places
+set HTMLHELP=
+where hhc /q && set HTMLHELP=hhc && goto :skiphhcsearch
+where /R ..\externals hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles(x86)%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" (
+ echo.
+ echo.The HTML Help Workshop was not found. Set the HTMLHELP variable
+ echo.to the path to hhc.exe or download and install it from
+ echo.http://msdn.microsoft.com/en-us/library/ms669985
+ exit /B 1
+)
+:skiphhcsearch
if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/extensions/patchlevel.py`) do set DISTVERSION=%%v
@@ -36,7 +50,8 @@ if errorlevel 9009 ( echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
- goto end
+ popd
+ exit /B 1
)
rem Targets that do require sphinx-build and have their own label
@@ -76,15 +91,6 @@ if NOT "%PAPER%" == "" ( cmd /C %SPHINXBUILD% %SPHINXOPTS% -b%1 -dbuild\doctrees . %BUILDDIR%\%*
if "%1" EQU "htmlhelp" (
- if not exist "%HTMLHELP%" (
- echo.
- echo.The HTML Help Workshop was not found. Set the HTMLHELP variable
- echo.to the path to hhc.exe or download and install it from
- echo.http://msdn.microsoft.com/en-us/library/ms669985
- rem Set errorlevel to 1 and exit
- cmd /C exit /b 1
- goto end
- )
cmd /C "%HTMLHELP%" build\htmlhelp\python%DISTVERSION:.=%.hhp
rem hhc.exe seems to always exit with code 1, reset to 0 for less than 2
if not errorlevel 2 cmd /C exit /b 0
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 5dd17eb030..01cfd6dbd2 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -51,6 +51,9 @@ Summarizing: : | `with_stmt` : | `funcdef` : | `classdef` + : | `async_with_stmt` + : | `async_for_stmt` + : | `async_funcdef` suite: `stmt_list` NEWLINE | NEWLINE INDENT `statement`+ DEDENT statement: `stmt_list` NEWLINE | `compound_stmt` stmt_list: `simple_stmt` (";" `simple_stmt`)* [";"] @@ -660,6 +663,130 @@ can be used to create instance variables with different implementation details. :pep:`3129` - Class Decorators +Coroutines +========== + +.. versionadded:: 3.5 + +.. index:: statement: async def +.. _`async def`: + +Coroutine function definition +----------------------------- + +.. productionlist:: + async_funcdef: [`decorators`] "async" "def" `funcname` "(" [`parameter_list`] ")" ["->" `expression`] ":" `suite` + +.. index:: + keyword: async + keyword: await + +Execution of Python coroutines can be suspended and resumed at many points +(see :term:`coroutine`). In the body of a coroutine, any ``await`` and +``async`` identifiers become reserved keywords; :keyword:`await` expressions, +:keyword:`async for` and :keyword:`async with` can only be used in +coroutine bodies. + +Functions defined with ``async def`` syntax are always coroutine functions, +even if they do not contain ``await`` or ``async`` keywords. + +It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in +``async def`` coroutines. + +An example of a coroutine function:: + + async def func(param1, param2): + do_stuff() + await some_coroutine() + + +.. index:: statement: async for +.. _`async for`: + +The :keyword:`async for` statement +---------------------------------- + +.. productionlist:: + async_for_stmt: "async" `for_stmt` + +An :term:`asynchronous iterable` is able to call asynchronous code in its +*iter* implementation, and :term:`asynchronous iterator` can call asynchronous +code in its *next* method. + +The ``async for`` statement allows convenient iteration over asynchronous +iterators. + +The following code:: + + async for TARGET in ITER: + BLOCK + else: + BLOCK2 + +Is semantically equivalent to:: + + iter = (ITER) + iter = await type(iter).__aiter__(iter) + running = True + while running: + try: + TARGET = await type(iter).__anext__(iter) + except StopAsyncIteration: + running = False + else: + BLOCK + else: + BLOCK2 + +See also :meth:`__aiter__` and :meth:`__anext__` for details. + +It is a :exc:`SyntaxError` to use ``async for`` statement outside of an +:keyword:`async def` function. + + +.. index:: statement: async with +.. _`async with`: + +The :keyword:`async with` statement +----------------------------------- + +.. productionlist:: + async_with_stmt: "async" `with_stmt` + +An :term:`asynchronous context manager` is a :term:`context manager` that is +able to suspend execution in its *enter* and *exit* methods. + +The following code:: + + async with EXPR as VAR: + BLOCK + +Is semantically equivalent to:: + + mgr = (EXPR) + aexit = type(mgr).__aexit__ + aenter = type(mgr).__aenter__(mgr) + exc = True + + VAR = await aenter + try: + BLOCK + except: + if not await aexit(mgr, *sys.exc_info()): + raise + else: + await aexit(mgr, None, None, None) + +See also :meth:`__aenter__` and :meth:`__aexit__` for details. + +It is a :exc:`SyntaxError` to use ``async with`` statement outside of an +:keyword:`async def` function. + +.. seealso:: + + :pep:`492` - Coroutines with async and await syntax + + .. rubric:: Footnotes .. [#] The exception is propagated to the invocation stack unless diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 4acf13e97b..e1d71b8115 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -616,6 +616,16 @@ Callable types exception is raised and the iterator will have reached the end of the set of values to be returned. + Coroutine functions + .. index:: + single: coroutine; function + + A function or method which is defined using :keyword:`async def` is called + a :dfn:`coroutine function`. Such a function, when called, returns a + :term:`coroutine` object. It may contain :keyword:`await` expressions, + as well as :keyword:`async with` and :keyword:`async for` statements. See + also the :ref:`coroutine-objects` section. + Built-in functions .. index:: object: built-in function @@ -1990,6 +2000,7 @@ left undefined. .. method:: object.__add__(self, other) object.__sub__(self, other) object.__mul__(self, other) + object.__matmul__(self, other) object.__truediv__(self, other) object.__floordiv__(self, other) object.__mod__(self, other) @@ -2006,15 +2017,16 @@ left undefined. builtin: pow builtin: pow - These methods are called to implement the binary arithmetic operations (``+``, - ``-``, ``*``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, - ``>>``, ``&``, ``^``, ``|``). For instance, to evaluate the expression - ``x + y``, where *x* is an instance of a class that has an :meth:`__add__` - method, ``x.__add__(y)`` is called. The :meth:`__divmod__` method should be the - equivalent to using :meth:`__floordiv__` and :meth:`__mod__`; it should not be - related to :meth:`__truediv__`. Note that :meth:`__pow__` should be defined - to accept an optional third argument if the ternary version of the built-in - :func:`pow` function is to be supported. + These methods are called to implement the binary arithmetic operations + (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, + :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``). For instance, to + evaluate the expression ``x + y``, where *x* is an instance of a class that + has an :meth:`__add__` method, ``x.__add__(y)`` is called. The + :meth:`__divmod__` method should be the equivalent to using + :meth:`__floordiv__` and :meth:`__mod__`; it should not be related to + :meth:`__truediv__`. Note that :meth:`__pow__` should be defined to accept + an optional third argument if the ternary version of the built-in :func:`pow` + function is to be supported. If one of those methods does not support the operation with the supplied arguments, it should return ``NotImplemented``. @@ -2023,6 +2035,7 @@ left undefined. .. method:: object.__radd__(self, other) object.__rsub__(self, other) object.__rmul__(self, other) + object.__rmatmul__(self, other) object.__rtruediv__(self, other) object.__rfloordiv__(self, other) object.__rmod__(self, other) @@ -2038,14 +2051,14 @@ left undefined. builtin: divmod builtin: pow - These methods are called to implement the binary arithmetic operations (``+``, - ``-``, ``*``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, - ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected (swapped) operands. - These functions are only called if the left operand does not support the - corresponding operation and the operands are of different types. [#]_ For - instance, to evaluate the expression ``x - y``, where *y* is an instance of - a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` is called if - ``x.__sub__(y)`` returns *NotImplemented*. + These methods are called to implement the binary arithmetic operations + (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, + :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected + (swapped) operands. These functions are only called if the left operand does + not support the corresponding operation and the operands are of different + types. [#]_ For instance, to evaluate the expression ``x - y``, where *y* is + an instance of a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` + is called if ``x.__sub__(y)`` returns *NotImplemented*. .. index:: builtin: pow @@ -2063,6 +2076,7 @@ left undefined. .. method:: object.__iadd__(self, other) object.__isub__(self, other) object.__imul__(self, other) + object.__imatmul__(self, other) object.__itruediv__(self, other) object.__ifloordiv__(self, other) object.__imod__(self, other) @@ -2074,17 +2088,17 @@ left undefined. object.__ior__(self, other) These methods are called to implement the augmented arithmetic assignments - (``+=``, ``-=``, ``*=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, ``>>=``, - ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation - in-place (modifying *self*) and return the result (which could be, but does - not have to be, *self*). If a specific method is not defined, the augmented - assignment falls back to the normal methods. For instance, if *x* is an - instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent - to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` - are considered, as with the evaluation of ``x + y``. In certain situations, - augmented assignment can result in unexpected errors (see - :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in - fact part of the data model. + (``+=``, ``-=``, ``*=``, ``@=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, + ``>>=``, ``&=``, ``^=``, ``|=``). These methods should attempt to do the + operation in-place (modifying *self*) and return the result (which could be, + but does not have to be, *self*). If a specific method is not defined, the + augmented assignment falls back to the normal methods. For instance, if *x* + is an instance of a class with an :meth:`__iadd__` method, ``x += y`` is + equivalent to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and + ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. In + certain situations, augmented assignment can result in unexpected errors (see + :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in fact + part of the data model. .. method:: object.__neg__(self) @@ -2254,6 +2268,155 @@ special methods (the special method *must* be set on the class object itself in order to be consistently invoked by the interpreter). +.. index:: + single: coroutine + +Coroutines +========== + + +Awaitable Objects +----------------- + +An :term:`awaitable` object generally implements an :meth:`__await__` method. +:term:`Coroutine` objects returned from :keyword:`async def` functions +are awaitable. + +.. note:: + + The :term:`generator iterator` objects returned from generators + decorated with :func:`types.coroutine` or :func:`asyncio.coroutine` + are also awaitable, but they do not implement :meth:`__await__`. + +.. method:: object.__await__(self) + + Must return an :term:`iterator`. Should be used to implement + :term:`awaitable` objects. For instance, :class:`asyncio.Future` implements + this method to be compatible with the :keyword:`await` expression. + +.. versionadded:: 3.5 + +.. seealso:: :pep:`492` for additional information about awaitable objects. + + +.. _coroutine-objects: + +Coroutine Objects +----------------- + +:term:`Coroutine` objects are :term:`awaitable` objects. +A coroutine's execution can be controlled by calling :meth:`__await__` and +iterating over the result. When the coroutine has finished executing and +returns, the iterator raises :exc:`StopIteration`, and the exception's +:attr:`~StopIteration.value` attribute holds the return value. If the +coroutine raises an exception, it is propagated by the iterator. Coroutines +should not directly raise unhandled :exc:`StopIteration` exceptions. + +Coroutines also have the methods listed below, which are analogous to +those of generators (see :ref:`generator-methods`). However, unlike +generators, coroutines do not directly support iteration. + +.. method:: coroutine.send(value) + + Starts or resumes execution of the coroutine. If *value* is ``None``, + this is equivalent to advancing the iterator returned by + :meth:`__await__`. If *value* is not ``None``, this method delegates + to the :meth:`~generator.send` method of the iterator that caused + the coroutine to suspend. The result (return value, + :exc:`StopIteration`, or other exception) is the same as when + iterating over the :meth:`__await__` return value, described above. + +.. method:: coroutine.throw(type[, value[, traceback]]) + + Raises the specified exception in the coroutine. This method delegates + to the :meth:`~generator.throw` method of the iterator that caused + the coroutine to suspend, if it has such a method. Otherwise, + the exception is raised at the suspension point. The result + (return value, :exc:`StopIteration`, or other exception) is the same as + when iterating over the :meth:`__await__` return value, described + above. If the exception is not caught in the coroutine, it propagates + back to the caller. + +.. method:: coroutine.close() + + Causes the coroutine to clean itself up and exit. If the coroutine + is suspended, this method first delegates to the :meth:`~generator.close` + method of the iterator that caused the coroutine to suspend, if it + has such a method. Then it raises :exc:`GeneratorExit` at the + suspension point, causing the coroutine to immediately clean itself up. + Finally, the coroutine is marked as having finished executing, even if + it was never started. + + Coroutine objects are automatically closed using the above process when + they are about to be destroyed. + + +Asynchronous Iterators +---------------------- + +An *asynchronous iterable* is able to call asynchronous code in its +``__aiter__`` implementation, and an *asynchronous iterator* can call +asynchronous code in its ``__anext__`` method. + +Asynchronous iterators can be used in a :keyword:`async for` statement. + +.. method:: object.__aiter__(self) + + Must return an *awaitable* resulting in an *asynchronous iterator* object. + +.. method:: object.__anext__(self) + + Must return an *awaitable* resulting in a next value of the iterator. Should + raise a :exc:`StopAsyncIteration` error when the iteration is over. + +An example of an asynchronous iterable object:: + + class Reader: + async def readline(self): + ... + + async def __aiter__(self): + return self + + async def __anext__(self): + val = await self.readline() + if val == b'': + raise StopAsyncIteration + return val + +.. versionadded:: 3.5 + + +Asynchronous Context Managers +----------------------------- + +An *asynchronous context manager* is a *context manager* that is able to +suspend execution in its ``__aenter__`` and ``__aexit__`` methods. + +Asynchronous context managers can be used in a :keyword:`async with` statement. + +.. method:: object.__aenter__(self) + + This method is semantically similar to the :meth:`__enter__`, with only + difference that it must return an *awaitable*. + +.. method:: object.__aexit__(self, exc_type, exc_value, traceback) + + This method is semantically similar to the :meth:`__exit__`, with only + difference that it must return an *awaitable*. + +An example of an asynchronous context manager class:: + + class AsyncContextManager: + async def __aenter__(self): + await log('entering context') + + async def __aexit__(self, exc_type, exc, tb): + await log('exiting context') + +.. versionadded:: 3.5 + + .. rubric:: Footnotes .. [#] It *is* possible in some cases to change an object's type, under certain diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 29ef0b7766..5c33ce6f0d 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -390,6 +390,7 @@ on the right hand side of an assignment statement. to sub-generators easy. .. index:: object: generator +.. _generator-methods: Generator-iterator methods ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -443,12 +444,12 @@ is already executing raises a :exc:`ValueError` exception. .. method:: generator.close() Raises a :exc:`GeneratorExit` at the point where the generator function was - paused. If the generator function then raises :exc:`StopIteration` (by - exiting normally, or due to already being closed) or :exc:`GeneratorExit` (by - not catching the exception), close returns to its caller. If the generator - yields a value, a :exc:`RuntimeError` is raised. If the generator raises any - other exception, it is propagated to the caller. :meth:`close` does nothing - if the generator has already exited due to an exception or normal exit. + paused. If the generator function then exits gracefully, is already closed, + or raises :exc:`GeneratorExit` (by not catching the exception), close + returns to its caller. If the generator yields a value, a + :exc:`RuntimeError` is raised. If the generator raises any other exception, + it is propagated to the caller. :meth:`close` does nothing if the generator + has already exited due to an exception or normal exit. .. index:: single: yield; examples @@ -811,6 +812,20 @@ a class instance: if that method was called. +.. _await: + +Await expression +================ + +Suspend the execution of :term:`coroutine` on an :term:`awaitable` object. +Can only be used inside a :term:`coroutine function`. + +.. productionlist:: + await: ["await"] `primary` + +.. versionadded:: 3.5 + + .. _power: The power operator @@ -820,7 +835,7 @@ The power operator binds more tightly than unary operators on its left; it binds less tightly than unary operators on its right. The syntax is: .. productionlist:: - power: `primary` ["**" `u_expr`] + power: `await` ["**" `u_expr`] Thus, in an unparenthesized sequence of power and unary operators, the operators are evaluated from right to left (this does not constrain the evaluation order @@ -891,8 +906,9 @@ from the power operator, there are only two levels, one for multiplicative operators and one for additive operators: .. productionlist:: - m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "//" `u_expr` | `m_expr` "/" `u_expr` - : | `m_expr` "%" `u_expr` + m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "@" `m_expr` | + : `m_expr` "//" `u_expr`| `m_expr` "/" `u_expr` | + : `m_expr` "%" `u_expr` a_expr: `m_expr` | `a_expr` "+" `m_expr` | `a_expr` "-" `m_expr` .. index:: single: multiplication @@ -903,6 +919,13 @@ the other must be a sequence. In the former case, the numbers are converted to a common type and then multiplied together. In the latter case, sequence repetition is performed; a negative repetition factor yields an empty sequence. +.. index:: single: matrix multiplication + +The ``@`` (at) operator is intended to be used for matrix multiplication. No +builtin Python types implement this operator. + +.. versionadded:: 3.5 + .. index:: exception: ZeroDivisionError single: division @@ -1466,13 +1489,16 @@ precedence and have a left-to-right chaining feature as described in the +-----------------------------------------------+-------------------------------------+ | ``+``, ``-`` | Addition and subtraction | +-----------------------------------------------+-------------------------------------+ -| ``*``, ``/``, ``//``, ``%`` | Multiplication, division, remainder | -| | [#]_ | +| ``*``, ``@``, ``/``, ``//``, ``%`` | Multiplication, matrix | +| | multiplication division, | +| | remainder [#]_ | +-----------------------------------------------+-------------------------------------+ | ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT | +-----------------------------------------------+-------------------------------------+ | ``**`` | Exponentiation [#]_ | +-----------------------------------------------+-------------------------------------+ +| ``await`` ``x`` | Await expression | ++-----------------------------------------------+-------------------------------------+ | ``x[index]``, ``x[index:index]``, | Subscription, slicing, | | ``x(arguments...)``, ``x.attribute`` | call, attribute reference | +-----------------------------------------------+-------------------------------------+ diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 16d28d8c73..d549b2674e 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -339,6 +339,7 @@ of what happens during the loading portion of import:: module = None if spec.loader is not None and hasattr(spec.loader, 'create_module'): + # It is assumed 'exec_module' will also be defined on the loader. module = spec.loader.create_module(spec) if module is None: module = ModuleType(spec.name) @@ -427,7 +428,7 @@ Module loaders may opt in to creating the module object during loading by implementing a :meth:`~importlib.abc.Loader.create_module` method. It takes one argument, the module spec, and returns the new module object to use during loading. ``create_module()`` does not need to set any attributes -on the module object. If the loader does not define ``create_module()``, the +on the module object. If the method returns ``None``, the import machinery will create the new module itself. .. versionadded:: 3.4 @@ -459,7 +460,13 @@ import machinery will create the new module itself. * If loading fails, the loader must remove any modules it has inserted into :data:`sys.modules`, but it must remove **only** the failing - module, and only if the loader itself has loaded it explicitly. + module(s), and only if the loader itself has loaded the module(s) + explicitly. + +.. versionchanged:: 3.5 + A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but + ``create_module()`` is not. Starting in Python 3.6 it will be an error to not + define ``create_module()`` on a loader attached to a ModuleSpec. Submodules ---------- @@ -674,7 +681,7 @@ path entry finder that knows how to handle that particular kind of path. The default set of path entry finders implement all the semantics for finding modules on the file system, handling special file types such as Python source -code (``.py`` files), Python byte code (``.pyc`` and ``.pyo`` files) and +code (``.py`` files), Python byte code (``.pyc`` files) and shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport` module in the standard library, the default path entry finders also handle loading all of these file types (other than shared libraries) from zipfiles. @@ -788,6 +795,15 @@ hook` callables on :data:`sys.path_hooks`, then the following protocol is used to ask the finder for a module spec, which is then used when loading the module. +The current working directory -- denoted by an empty string -- is handled +slightly differently from other entries on :data:`sys.path`. First, if the +current working directory is found to not exist, no value is stored in +:data:`sys.path_importer_cache`. Second, the value for the current working +directory is looked up fresh for each module lookup. Third, the path used for +:data:`sys.path_importer_cache` and returned by +:meth:`importlib.machinery.PathFinder.find_spec` will be the actual current +working directory and not the empty string. + Path entry finder protocol -------------------------- diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index a1265cfad1..8ad975f71f 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -313,7 +313,7 @@ The Unicode category codes mentioned above stand for: * *Nd* - decimal numbers * *Pc* - connector punctuations * *Other_ID_Start* - explicit list of characters in `PropList.txt - <http://www.unicode.org/Public/6.3.0/ucd/PropList.txt>`_ to support backwards + <http://www.unicode.org/Public/8.0.0/ucd/PropList.txt>`_ to support backwards compatibility * *Other_ID_Continue* - likewise @@ -692,7 +692,7 @@ Operators The following tokens are operators:: - + - * ** / // % + + - * ** / // % @ << >> & | ^ ~ < > <= >= == != @@ -708,7 +708,7 @@ The following tokens serve as delimiters in the grammar:: ( ) [ ] { } , : . ; @ = -> - += -= *= /= //= %= + += -= *= /= //= %= @= &= |= ^= >>= <<= **= The period can also occur in floating-point and imaginary literals. A sequence @@ -729,4 +729,4 @@ occurrence outside string literals and comments is an unconditional error:: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NameAliases.txt diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 8946b4f354..5f605408b7 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -281,7 +281,7 @@ operation and an assignment statement: .. productionlist:: augmented_assignment_stmt: `augtarget` `augop` (`expression_list` | `yield_expression`) augtarget: `identifier` | `attributeref` | `subscription` | `slicing` - augop: "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**=" + augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" : | ">>=" | "<<=" | "&=" | "^=" | "|=" (See section :ref:`primaries` for the syntax definitions of the last three diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 64a56655f7..d44b052540 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -34,7 +34,7 @@ import suspicious ISSUE_URI = 'https://bugs.python.org/issue%s' -SOURCE_URI = 'https://hg.python.org/cpython/file/3.4/%s' +SOURCE_URI = 'https://hg.python.org/cpython/file/3.5/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists from docutils.parsers.rst.states import Body diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 032531d3cb..f7c3bde160 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -121,6 +121,8 @@ library/ipaddress,,:db8,>>> ipaddress.IPv6Address('2001:db8::1000') library/ipaddress,,::,>>> ipaddress.IPv6Address('2001:db8::1000') library/ipaddress,,:db8,IPv6Address('2001:db8::1000') library/ipaddress,,::,IPv6Address('2001:db8::1000') +library/ipaddress,,:db8,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer" +library/ipaddress,,::,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer" library/ipaddress,,::,"""::abc:7:def""" library/ipaddress,,:def,"""::abc:7:def""" library/ipaddress,,::,::FFFF/96 @@ -138,8 +140,6 @@ library/itertools,,:stop,elements from seq[start:stop:step] library/logging.handlers,,:port,host:port library/mmap,,:i2,obj[i1:i2] library/multiprocessing,,`,# Add more tasks using `put()` -library/multiprocessing,,`,">>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]`" -library/multiprocessing,,`,">>> l._callmethod('__getitem__', (slice(2, 7),)) # equiv to `l[2:7]`" library/multiprocessing,,:queue,">>> QueueManager.register('get_queue', callable=lambda:queue)" library/multiprocessing,,`,# register the Foo class; make `f()` and `g()` accessible via proxy library/multiprocessing,,`,# register the Foo class; make `g()` and `_h()` accessible via proxy @@ -176,8 +176,6 @@ library/ssl,,:MyState,State or Province Name (full name) [Some-State]:MyState library/ssl,,:ops,Email Address []:ops@myserver.mygroup.myorganization.com library/ssl,,:Some,"Locality Name (eg, city) []:Some City" library/ssl,,:US,Country Name (2 letter code) [AU]:US -library/stdtypes,,::,>>> a[::-1].tolist() -library/stdtypes,,::,>>> a[::2].tolist() library/stdtypes,,:end,s[start:end] library/stdtypes,,::,>>> hash(v[::-2]) == hash(b'abcefg'[::-2]) library/stdtypes,,:len,s[len(s):len(s)] @@ -207,16 +205,6 @@ library/venv,,:param,":param nodist: If True, setuptools and pip are not install library/venv,,:param,":param progress: If setuptools or pip are installed, the progress of the" library/venv,,:param,":param nopip: If True, pip is not installed into the created" library/venv,,:param,:param context: The information for the environment creation request -library/xml.etree.elementtree,,:sometag,prefix:sometag -library/xml.etree.elementtree,,:fictional,"<actors xmlns:fictional=""http://characters.example.com""" -library/xml.etree.elementtree,,:character,<fictional:character>Lancelot</fictional:character> -library/xml.etree.elementtree,,:character,<fictional:character>Archie Leach</fictional:character> -library/xml.etree.elementtree,,:character,<fictional:character>Sir Robin</fictional:character> -library/xml.etree.elementtree,,:character,<fictional:character>Gunther</fictional:character> -library/xml.etree.elementtree,,:character,<fictional:character>Commander Clement</fictional:character> -library/xml.etree.elementtree,,:actor,"for actor in root.findall('real_person:actor', ns):" -library/xml.etree.elementtree,,:name,"name = actor.find('real_person:name', ns)" -library/xml.etree.elementtree,,:character,"for char in actor.findall('role:character', ns):" library/xmlrpc.client,,:pass,http://user:pass@host:port/path library/xmlrpc.client,,:pass,user:pass library/xmlrpc.client,,:port,http://user:pass@host:port/path @@ -243,7 +231,6 @@ tutorial/stdlib2,,:start,extra = data[start:start+extra_size] tutorial/stdlib2,,:start,"fields = struct.unpack('<IIIHH', data[start:start+16])" tutorial/stdlib2,,:start,filename = data[start:start+filenamesize] tutorial/stdlib2,,:Warning,WARNING:root:Warning:config file server.conf not found -tutorial/venv,,:c7b9645a6f35,"Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25)" using/cmdline,,:category,action:message:category:module:line using/cmdline,,:errorhandler,:errorhandler using/cmdline,,:line,action:message:category:module:line @@ -264,6 +251,11 @@ whatsnew/2.4,,:System, whatsnew/2.5,,:memory,:memory: whatsnew/2.5,,:step,[start:stop:step] whatsnew/2.5,,:stop,[start:stop:step] +whatsnew/2.7,,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +whatsnew/2.7,,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') +whatsnew/2.7,,:Sunday,'2009:4:Sunday' +whatsnew/2.7,,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" +whatsnew/2.7,,::,"export PYTHONWARNINGS=all,error:::Cookie:0" whatsnew/3.2,,:affe,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," whatsnew/3.2,,:affe,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') whatsnew/3.2,,:beef,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," @@ -279,14 +271,27 @@ whatsnew/3.2,,:feed,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe: whatsnew/3.2,,:gz,">>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:" whatsnew/3.2,,:location,zope9-location = ${zope9:location} whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf -whatsnew/changelog,,:platform,:platform: whatsnew/changelog,,:gz,": TarFile opened with external fileobj and ""w:gz"" mode didn't" -whatsnew/changelog,,:PythonCmd,"With Tk < 8.5 _tkinter.c:PythonCmd() raised UnicodeDecodeError, caused" -whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" -whatsnew/changelog,,:password,user:password -whatsnew/2.7,780,:Sunday,'2009:4:Sunday' -whatsnew/2.7,907,::,"export PYTHONWARNINGS=all,error:::Cookie:0" -whatsnew/2.7,907,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" -whatsnew/2.7,1657,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') -whatsnew/2.7,1657,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +library/tarfile,149,:xz,'x:xz' +library/xml.etree.elementtree,290,:sometag,prefix:sometag +library/xml.etree.elementtree,301,:fictional,"<actors xmlns:fictional=""http://characters.example.com""" +library/xml.etree.elementtree,301,:character,<fictional:character>Lancelot</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Archie Leach</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Sir Robin</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Gunther</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Commander Clement</fictional:character> +library/xml.etree.elementtree,332,:actor,"for actor in root.findall('real_person:actor', ns):" +library/xml.etree.elementtree,332,:name,"name = actor.find('real_person:name', ns)" +library/xml.etree.elementtree,332,:character,"for char in actor.findall('role:character', ns):" +library/zipapp,31,:main,"$ python -m zipapp myapp -m ""myapp:main""" +library/zipapp,82,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg.mod"" is a" +library/zipapp,155,:callable,"""pkg.module:callable"" and the archive will be run by importing" +library/stdtypes,,::,>>> m[::2].tolist() +library/sys,1115,`,# ``wrapper`` creates a ``wrap(coro)`` coroutine: +tutorial/venv,77,:c7b9645a6f35,"Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25)" +whatsnew/3.5,,:root,'WARNING:root:warning\n' +whatsnew/3.5,,:warning,'WARNING:root:warning\n' +whatsnew/3.5,,::,>>> addr6 = ipaddress.IPv6Address('::1') +whatsnew/3.5,,:root,ERROR:root:exception +whatsnew/3.5,,:exception,ERROR:root:exception diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html index abdf070772..78e9c4f99e 100644 --- a/Doc/tools/templates/indexsidebar.html +++ b/Doc/tools/templates/indexsidebar.html @@ -3,8 +3,7 @@ <h3>Docs for other versions</h3> <ul> <li><a href="https://docs.python.org/2.7/">Python 2.7 (stable)</a></li> - <li><a href="https://docs.python.org/3.3/">Python 3.3 (stable)</a></li> - <li><a href="https://docs.python.org/3.5/">Python 3.5 (in development)</a></li> + <li><a href="https://docs.python.org/3.4/">Python 3.4 (stable)</a></li> <li><a href="https://www.python.org/doc/versions/">Old versions</a></li> </ul> diff --git a/Doc/tutorial/appendix.rst b/Doc/tutorial/appendix.rst index ce50c1e8c3..e04459b2d9 100644 --- a/Doc/tutorial/appendix.rst +++ b/Doc/tutorial/appendix.rst @@ -40,7 +40,7 @@ Executable Python Scripts On BSD'ish Unix systems, Python scripts can be made directly executable, like shell scripts, by putting the line :: - #!/usr/bin/env python3.4 + #!/usr/bin/env python3.5 (assuming that the interpreter is on the user's :envvar:`PATH`) at the beginning of the script and giving the file an executable mode. The ``#!`` must be the @@ -107,7 +107,7 @@ of your user site-packages directory. Start Python and run this code:: >>> import site >>> site.getusersitepackages() - '/home/user/.local/lib/python3.4/site-packages' + '/home/user/.local/lib/python3.5/site-packages' Now you can create a file named :file:`usercustomize.py` in that directory and put anything you want in it. It will affect every invocation of Python, unless diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index 1ea299fbed..0d51480177 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -73,10 +73,11 @@ objects: Return the number of times *x* appears in the list. -.. method:: list.sort() +.. method:: list.sort(key=None, reverse=False) :noindex: - Sort the items of the list in place. + Sort the items of the list in place (the arguments can be used for sort + customization, see :func:`sorted` for their explanation). .. method:: list.reverse() @@ -611,18 +612,18 @@ returns a new sorted list while leaving the source unaltered. :: orange pear -To change a sequence you are iterating over while inside the loop (for -example to duplicate certain items), it is recommended that you first make -a copy. Looping over a sequence does not implicitly make a copy. The slice -notation makes this especially convenient:: +It is sometimes tempting to change a list while you are looping over it; +however, it is often simpler and safer to create a new list instead. :: - >>> words = ['cat', 'window', 'defenestrate'] - >>> for w in words[:]: # Loop over a slice copy of the entire list. - ... if len(w) > 6: - ... words.insert(0, w) + >>> import math + >>> raw_data = [56.2, float('NaN'), 51.7, 55.3, 52.5, float('NaN'), 47.8] + >>> filtered_data = [] + >>> for value in raw_data: + ... if not math.isnan(value): + ... filtered_data.append(value) ... - >>> words - ['defenestrate', 'cat', 'window', 'defenestrate'] + >>> filtered_data + [56.2, 51.7, 55.3, 52.5, 47.8] .. _tut-conditions: diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index dd6b1a9f38..e966085f93 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -10,13 +10,13 @@ Using the Python Interpreter Invoking the Interpreter ======================== -The Python interpreter is usually installed as :file:`/usr/local/bin/python3.4` +The Python interpreter is usually installed as :file:`/usr/local/bin/python3.5` on those machines where it is available; putting :file:`/usr/local/bin` in your Unix shell's search path makes it possible to start it by typing the command: .. code-block:: text - python3.4 + python3.5 to the shell. [#]_ Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local @@ -24,11 +24,11 @@ Python guru or system administrator. (E.g., :file:`/usr/local/python` is a popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python34`, though you can change this when you're running the +:file:`C:\\Python35`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python34 + set path=%path%;C:\python35 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -96,8 +96,8 @@ with the *secondary prompt*, by default three dots (``...``). The interpreter prints a welcome message stating its version number and a copyright notice before printing the first prompt:: - $ python3.4 - Python 3.4 (default, Mar 16 2014, 09:25:04) + $ python3.5 + Python 3.5 (default, Sep 16 2015, 09:25:04) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index fd361aed69..9ae64b0cf7 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -216,15 +216,15 @@ Some tips for experts: statements, the ``-OO`` switch removes both assert statements and __doc__ strings. Since some programs may rely on having these available, you should only use this option if you know what you're doing. "Optimized" modules have - a .pyo rather than a .pyc suffix and are usually smaller. Future releases may + an ``opt-`` tag and are usually smaller. Future releases may change the effects of optimization. -* A program doesn't run any faster when it is read from a ``.pyc`` or ``.pyo`` +* A program doesn't run any faster when it is read from a ``.pyc`` file than when it is read from a ``.py`` file; the only thing that's faster - about ``.pyc`` or ``.pyo`` files is the speed with which they are loaded. + about ``.pyc`` files is the speed with which they are loaded. -* The module :mod:`compileall` can create .pyc files (or .pyo files when - :option:`-O` is used) for all modules in a directory. +* The module :mod:`compileall` can create .pyc files for all modules in a + directory. * There is more detail on this process, including a flow chart of the decisions, in PEP 3147. @@ -548,4 +548,3 @@ modules found in a package. .. [#] In fact function definitions are also 'statements' that are 'executed'; the execution of a module-level function definition enters the function name in the module's global symbol table. - diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index 72d51de1c0..0954ebae06 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -15,7 +15,7 @@ operating system:: >>> import os >>> os.getcwd() # Return the current working directory - 'C:\\Python34' + 'C:\\Python35' >>> os.chdir('/server/accesslogs') # Change current working directory >>> os.system('mkdir today') # Run the command mkdir in the system shell 0 diff --git a/Doc/tutorial/stdlib2.rst b/Doc/tutorial/stdlib2.rst index c0197ea5ef..f7d2a0ac2a 100644 --- a/Doc/tutorial/stdlib2.rst +++ b/Doc/tutorial/stdlib2.rst @@ -18,7 +18,7 @@ abbreviated displays of large or deeply nested containers:: >>> import reprlib >>> reprlib.repr(set('supercalifragilisticexpialidocious')) - "set(['a', 'c', 'd', 'e', 'f', 'g', ...])" + "{'a', 'c', 'd', 'e', 'f', 'g', ...}" The :mod:`pprint` module offers more sophisticated control over printing both built-in and user defined objects in a way that is readable by the interpreter. @@ -277,7 +277,7 @@ applications include caching objects that are expensive to create:: Traceback (most recent call last): File "<stdin>", line 1, in <module> d['primary'] # entry was automatically removed - File "C:/python34/lib/weakref.py", line 46, in __getitem__ + File "C:/python35/lib/weakref.py", line 46, in __getitem__ o = self.data[key]() KeyError: 'primary' diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index a46349bcde..c7210a0134 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -190,13 +190,16 @@ Miscellaneous options .. cmdoption:: -b - Issue a warning when comparing str and bytes. Issue an error when the + Issue a warning when comparing :class:`bytes` or :class:`bytearray` with + :class:`str` or :class:`bytes` with :class:`int`. Issue an error when the option is given twice (:option:`-bb`). + .. versionchanged: 3.5 + Affects comparisons of :class:`bytes` with :class:`int`. .. cmdoption:: -B - If given, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If given, Python won't try to write ``.pyc`` files on the import of source modules. See also :envvar:`PYTHONDONTWRITEBYTECODE`. diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index 45bdd5a46d..02bcbeeb7f 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -21,11 +21,11 @@ subdirectory (on Windows, this is ``Lib\site-packages``). On Windows, you may have to invoke the ``pyvenv`` script as follows, if you don't have the relevant PATH and PATHEXT settings:: - c:\Temp>c:\Python34\python c:\Python34\Tools\Scripts\pyvenv.py myenv + c:\Temp>c:\Python35\python c:\Python35\Tools\Scripts\pyvenv.py myenv or equivalently:: - c:\Temp>c:\Python34\python -m venv myenv + c:\Temp>c:\Python35\python -m venv myenv The command, if run with ``-h``, will show the available options:: diff --git a/Doc/using/win_installer.png b/Doc/using/win_installer.png Binary files differnew file mode 100644 index 0000000000..00c88a830f --- /dev/null +++ b/Doc/using/win_installer.png diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index c05f72a853..2116c106de 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -7,22 +7,213 @@ ************************* .. sectionauthor:: Robert Lehmann <lehmannro@gmail.com> +.. sectionauthor:: Steve Dower <steve.dower@microsoft.com> This document aims to give an overview of Windows-specific behaviour you should know about when using Python on Microsoft Windows. -.. XXX (ncoghlan) - - This looks rather stale to me... - - Installing Python ================= -Unlike most Unix systems and services, Windows does not require Python natively -and thus does not pre-install a version of Python. However, the CPython team +Unlike most Unix systems and services, Windows does not include a system +supported installation of Python. To make Python available, the CPython team has compiled Windows installers (MSI packages) with every `release -<https://www.python.org/download/releases/>`_ for many years. +<https://www.python.org/download/releases/>`_ for many years. These installers +are primarily intended to add a per-user installation of Python, with the +core interpreter and library being used by a single user. The installer is also +able to install for all users of a single machine, and a separate ZIP file is +available for application-local distributions. + +Installation Steps +------------------ + +Four Python 3.5 installers are available for download - two each for the 32-bit +and 64-bit versions of the interpreter. The *web installer* is a small initial +download, and it will automatically download the required components as +necessary. The *offline installer* includes the components necessary for a +default installation and only requires an internet connection for optional +features. See :ref:`install-layout-option` for other ways to avoid downloading +during installation. + +After starting the installer, one of two options may be selected: + +.. image:: win_installer.png + +If you select "Install Now": + +* You will *not* need to be an administrator (unless a system update for the + C Runtime Library is required or you install the :ref:`launcher` for all + users) +* Python will be installed into your user directory +* The :ref:`launcher` will be installed according to the option at the bottom + of the first pace +* The standard library, test suite, launcher and pip will be installed +* If selected, the install directory will be added to your :envvar:`PATH` +* Shortcuts will only be visible for the current user + +Selecting "Customize installation" will allow you to select the features to +install, the installation location and other options or post-install actions. +To install debugging symbols or binaries, you will need to use this option. + +To perform an all-users installation, you should select "Customize +installation". In this case: + +* You may be required to provide administrative credentials or approval +* Python will be installed into the Program Files directory +* The :ref:`launcher` will be installed into the Windows directory +* Optional features may be selected during installation +* The standard library can be pre-compiled to bytecode +* If selected, the install directory will be added to the system :envvar:`PATH` +* Shortcuts are available for all users + +.. _install-quiet-option: + +Installing Without UI +--------------------- + +All of the options available in the installer UI can also be specified from the +command line, allowing scripted installers to replicate an installation on many +machines without user interaction. These options may also be set without +suppressing the UI in order to change some of the defaults. + +To completely hide the installer UI and install Python silently, pass the +``/quiet`` option. To skip past the user interaction but still display +progress and errors, pass the ``/passive`` option. The ``/uninstall`` +option may be passed to immediately begin removing Python - no prompt will be +displayed. + +All other options are passed as ``name=value``, where the value is usually +``0`` to disable a feature, ``1`` to enable a feature, or a path. The full list +of available options is shown below. + ++---------------------------+--------------------------------------+--------------------------+ +| Name | Description | Default | ++===========================+======================================+==========================+ +| InstallAllUsers | Perform a system-wide installation. | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| TargetDir | The installation directory | Selected based on | +| | | InstallAllUsers | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultAllUsersTargetDir | The default installation directory | :file:`%ProgramFiles%\\\ | +| | for all-user installs | Python X.Y` or :file:`\ | +| | | %ProgramFiles(x86)%\\\ | +| | | Python X.Y` | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultJustForMeTargetDir | The default install directory for | :file:`%LocalAppData%\\\ | +| | just-for-me installs | Programs\\PythonXY` or | +| | | :file:`%LocalAppData%\\\ | +| | | Programs\\PythonXY-32` | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultCustomTargetDir | The default custom install directory | (empty) | +| | displayed in the UI | | ++---------------------------+--------------------------------------+--------------------------+ +| AssociateFiles | Create file associations if the | 1 | +| | launcher is also installed. | | ++---------------------------+--------------------------------------+--------------------------+ +| CompileAll | Compile all ``.py`` files to | 0 | +| | ``.pyc``. | | ++---------------------------+--------------------------------------+--------------------------+ +| PrependPath | Add install and Scripts directories | 0 | +| | tho :envvar:`PATH` and ``.PY`` to | | +| | :envvar:`PATHEXT` | | ++---------------------------+--------------------------------------+--------------------------+ +| Shortcuts | Create shortcuts for the interpreter,| 1 | +| | documentation and IDLE if installed. | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_doc | Install Python manual | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_debug | Install debug binaries | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_dev | Install developer headers and | 1 | +| | libraries | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_exe | Install :file:`python.exe` and | 1 | +| | related files | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_launcher | Install :ref:`launcher`. | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| InstallLauncherAllUsers | Installs :ref:`launcher` for all | 1 | +| | users. | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_lib | Install standard library and | 1 | +| | extension modules | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_pip | Install bundled pip and setuptools | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_symbols | Install debugging symbols (`*`.pdb) | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_tcltk | Install Tcl/Tk support and IDLE | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_test | Install standard library test suite | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_tools | Install utility scripts | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| LauncherOnly | Only installs the launcher. This | 0 | +| | will override most other options. | | ++---------------------------+--------------------------------------+--------------------------+ +| SimpleInstall | Disable most install UI | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| SimpleInstallDescription | A custom message to display when the | (empty) | +| | simplified install UI is used. | | ++---------------------------+--------------------------------------+--------------------------+ + +For example, to silently install a default, system-wide Python installation, +you could use the following command (from an elevated command prompt):: + + python-3.5.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0 + +To allow users to easily install a personal copy of Python without the test +suite, you could provide a shortcut with the following command. This will +display a simplified initial page and disallow customization:: + + python-3.5.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0 + SimpleInstall=1 SimpleInstallDescription="Just for me, no test suite." + +(Note that omitting the launcher also omits file associations, and is only +recommended for per-user installs when there is also a system-wide installation +that included the launcher.) + +The options listed above can also be provided in a file named ``unattend.xml`` +alongside the executable. This file specifies a list of options and values. +When a value is provided as an attribute, it will be converted to a number if +possible. Values provided as element text are always left as strings. This +example file sets the same options and the previous example:: + + <Options> + <Option Name="InstallAllUsers" Value="no" /> + <Option Name="Include_launcher" Value="0" /> + <Option Name="Include_test" Value="no" /> + <Option Name="SimpleInstall" Value="yes" /> + <Option Name="SimpleInstallDescription">Just for me, no test suite</Option> + </Options> + +.. _install-layout-option: + +Installing Without Downloading +------------------------------ + +As some features of Python are not included in the initial installer download, +selecting those features may require an internet connection. To avoid this +need, all possible components may be downloaded on-demand to create a complete +*layout* that will no longer require an internet connection regardless of the +selected features. Note that this download may be bigger than required, but +where a large number of installations are going to be performed it is very +useful to have a locally cached copy. + +Execute the following command from Command Prompt to download all possible +required files. Remember to substitute ``python-3.5.0.exe`` for the actual +name of your installer, and to create layouts in their own directories to +avoid collisions between files with the same name. + +:: + + python-3.5.0.exe /layout [optional target directory] + +You may also specify the ``/quiet`` option to hide the progress display. + + +Other Platforms +--------------- With ongoing development of Python, some platforms that used to be supported earlier are no longer supported (due to the lack of users or developers). @@ -66,19 +257,31 @@ key features: `ActivePython <http://www.activestate.com/activepython/>`_ Installer with multi-platform compatibility, documentation, PyWin32 -`Enthought Python Distribution <https://www.enthought.com/products/epd/>`_ - Popular modules (such as PyWin32) with their respective documentation, tool - suite for building extensible Python applications +`Anaconda <http://www.continuum.io/downloads/>`_ + Popular scientific modules (such as numpy, scipy and pandas) and the + ``conda`` package manager. + +`Canopy <https://www.enthought.com/products/canopy/>`_ + A "comprehensive Python analysis environment" with editors and other + development tools. -Notice that these packages are likely to install *older* versions of Python. +`WinPython <https://winpython.github.io/>`_ + Windows-specific distribution with prebuilt scientific packages and + tools for building packages. + +Note that these packages may not include the latest versions of Python or +other libraries, and are not maintained or supported by the core Python team. Configuring Python ================== -In order to run Python flawlessly, you might have to change certain environment -settings in Windows. +To run Python conveniently from a command prompt, you might consider changing +some default environment variables in Windows. While the installer provides an +option to configure the PATH and PATHEXT variables for you, this is only +reliable for a single, system-wide installation. If you regularly use multiple +versions of Python, consider using the :ref:`launcher`. .. _setting-envvars: @@ -86,163 +289,86 @@ settings in Windows. Excursus: Setting environment variables --------------------------------------- -Windows has a built-in dialog for changing environment variables (following -guide applies to XP classical view): Right-click the icon for your machine -(usually located on your Desktop and called "My Computer") and choose -:menuselection:`Properties` there. Then, open the :guilabel:`Advanced` tab -and click the :guilabel:`Environment Variables` button. +Windows allows environment variables to be configured permanently at both the +User level and the System level, or temporarily in a command prompt. + +To temporarily set environment variables, open Command Prompt and use the +:command:`set` command:: + + C:\>set PATH=C:\Program Files\Python 3.5;%PATH% + C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib + C:\>python -In short, your path is: +These changes will apply to any further commands executed in that console, and +will be inherited by any applications started from the console. - :menuselection:`My Computer - --> Properties - --> Advanced - --> Environment Variables` +Including the variable name within percent signs will expand to the existing +value, allowing you to add your new value at either the start or the end. +Modifying :envvar:`PATH` by adding the directory containing +:program:`python.exe` to the start is a common way to ensure the correct version +of Python is launched. +To permanently modify the default environment variables, click Start and search +for 'edit environment variables', or open System properties, :guilabel:`Advanced +system settings` and click the :guilabel:`Environment Variables` button. In this dialog, you can add or modify User and System variables. To change System variables, you need non-restricted access to your machine (i.e. Administrator rights). -Another way of adding variables to your environment is using the :command:`set` -command:: - - set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib - -To make this setting permanent, you could add the corresponding command line to -your :file:`autoexec.bat`. :program:`msconfig` is a graphical interface to this -file. +.. note:: -Viewing environment variables can also be done more straight-forward: The -command prompt will expand strings wrapped into percent signs automatically:: + Windows will concatenate User variables *after* System variables, which may + cause unexpected results when modifying :envvar:`PATH`. - echo %PATH% - -Consult :command:`set /?` for details on this behaviour. + The :envvar:`PYTHONPATH` variable is used by all versions of Python 2 and + Python 3, so you should not permanently configure this variable unless it + only includes code that is compatible with all of your installed Python + versions. .. seealso:: - http://support.microsoft.com/kb/100843 + http://support.microsoft.com/kb/100843 Environment variables in Windows NT - http://support.microsoft.com/kb/310519 + http://technet.microsoft.com/en-us/library/cc754250.aspx + The SET command, for temporarily modifying environment variables + + http://technet.microsoft.com/en-us/library/cc755104.aspx + The SETX command, for permanently modifying environment variables + + http://support.microsoft.com/kb/310519 How To Manage Environment Variables in Windows XP - http://www.chem.gla.ac.uk/~louis/software/faq/q1.html + http://www.chem.gla.ac.uk/~louis/software/faq/q1.html Setting Environment variables, Louis J. Farrugia - .. _windows-path-mod: Finding the Python executable ----------------------------- -.. versionchanged:: 3.3 +.. versionchanged:: 3.5 Besides using the automatically created start menu entry for the Python -interpreter, you might want to start Python in the command prompt. As of -Python 3.3, the installer has an option to set that up for you. - -At the "Customize Python 3.3" screen, an option called -"Add python.exe to search path" can be enabled to have the installer place -your installation into the :envvar:`%PATH%`. This allows you to type -:command:`python` to run the interpreter. Thus, you can also execute your +interpreter, you might want to start Python in the command prompt. The +installer for Python 3.5 and later has an option to set that up for you. + +On the first page of the installer, an option labelled "Add Python 3.5 to +PATH" can be selected to have the installer add the install location into the +:envvar:`PATH`. The location of the :file:`Scripts\\` folder is also added. +This allows you to type :command:`python` to run the interpreter, and +:command:`pip` or . Thus, you can also execute your scripts with command line options, see :ref:`using-on-cmdline` documentation. If you don't enable this option at install time, you can always re-run the -installer to choose it. - -The alternative is manually modifying the :envvar:`%PATH%` using the -directions in :ref:`setting-envvars`. You need to set your :envvar:`%PATH%` -environment variable to include the directory of your Python distribution, -delimited by a semicolon from other entries. An example variable could look -like this (assuming the first two entries are Windows' default):: - - C:\WINDOWS\system32;C:\WINDOWS;C:\Python33 - - -Finding modules ---------------- - -Python usually stores its library (and thereby your site-packages folder) in the -installation directory. So, if you had installed Python to -:file:`C:\\Python\\`, the default library would reside in -:file:`C:\\Python\\Lib\\` and third-party modules should be stored in -:file:`C:\\Python\\Lib\\site-packages\\`. - -This is how :data:`sys.path` is populated on Windows: - -* An empty entry is added at the start, which corresponds to the current - directory. - -* If the environment variable :envvar:`PYTHONPATH` exists, as described in - :ref:`using-on-envvars`, its entries are added next. Note that on Windows, - paths in this variable must be separated by semicolons, to distinguish them - from the colon used in drive identifiers (``C:\`` etc.). - -* Additional "application paths" can be added in the registry as subkeys of - :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the - ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have - semicolon-delimited path strings as their default value will cause each path - to be added to :data:`sys.path`. (Note that all known installers only use - HKLM, so HKCU is typically empty.) - -* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as - "Python Home". Otherwise, the path of the main Python executable is used to - locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a - Python home is found, the relevant sub-directories added to :data:`sys.path` - (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core - Python path is constructed from the PythonPath stored in the registry. - -* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in - the environment, and no registry entries can be found, a default path with - relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). - -The end result of all this is: - -* When running :file:`python.exe`, or any other .exe in the main Python - directory (either an installed version, or directly from the PCbuild - directory), the core path is deduced, and the core paths in the registry are - ignored. Other "application paths" in the registry are always read. - -* When Python is hosted in another .exe (different directory, embedded via COM, - etc), the "Python Home" will not be deduced, so the core path from the - registry is used. Other "application paths" in the registry are always read. - -* If Python can't find its home and there is no registry (eg, frozen .exe, some - very strange installation setup) you get a path with some default, but - relative, paths. - - -Executing scripts ------------------ - -As of Python 3.3, Python includes a launcher which facilitates running Python -scripts. See :ref:`launcher` for more information. - -Executing scripts without the Python launcher ---------------------------------------------- - -Without the Python launcher installed, Python scripts (files with the extension -``.py``) will be executed by :program:`python.exe` by default. This executable -opens a terminal, which stays open even if the program uses a GUI. If you do -not want this to happen, use the extension ``.pyw`` which will cause the script -to be executed by :program:`pythonw.exe` by default (both executables are -located in the top-level of your Python installation directory). This -suppresses the terminal window on startup. - -You can also make all ``.py`` scripts execute with :program:`pythonw.exe`, -setting this through the usual facilities, for example (might require -administrative rights): - -#. Launch a command prompt. -#. Associate the correct file group with ``.py`` scripts:: - - assoc .py=Python.File - -#. Redirect all Python files to the new executable:: - - ftype Python.File=C:\Path\to\pythonw.exe "%1" %* +installer, select Modify, and enable it. Alternatively, you can manually +modify the :envvar:`PATH` using the directions in :ref:`setting-envvars`. You +need to set your :envvar:`PATH` environment variable to include the directory +of your Python installation, delimited by a semicolon from other entries. An +example variable could look like this (assuming the first two entries already +existed):: + C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.5 .. _launcher: @@ -251,21 +377,26 @@ Python Launcher for Windows .. versionadded:: 3.3 -The Python launcher for Windows is a utility which aids in the location and -execution of different Python versions. It allows scripts (or the +The Python launcher for Windows is a utility which aids in locating and +executing of different Python versions. It allows scripts (or the command-line) to indicate a preference for a specific Python version, and will locate and execute that version. +Unlike the :envvar:`PATH` variable, the launcher will correctly select the most +appropriate version of Python. It will prefer per-user installations over +system-wide ones, and orders by language version rather than using the most +recently installed version. + Getting started --------------- From the command-line ^^^^^^^^^^^^^^^^^^^^^ -You should ensure the launcher is on your PATH - depending on how it was -installed it may already be there, but check just in case it is not. - -From a command-prompt, execute the following command: +System-wide installations of Python 3.3 and later will put the launcher on your +:envvar:`PATH`. The launcher is compatible with all available versions of +Python, so it does not matter which version is installed. To check that the +launcher is available, execute the following command in Command Prompt: :: @@ -291,6 +422,28 @@ If you have a Python 3.x installed, try the command: You should find the latest version of Python 3.x starts. +If you see the following error, you do not have the launcher installed: + +:: + + 'py' is not recognized as an internal or external command, + operable program or batch file. + +Per-user installations of Python do not add the launcher to :envvar:`PATH` +unless the option was selected on installation. + +Virtual environments +^^^^^^^^^^^^^^^^^^^^ + +.. versionadded:: 3.5 + +If the launcher is run with no explicit Python version specification, and a +virtual environment (created with the standard library :mod:`venv` module or +the external ``virtualenv`` tool) active, the launcher will run the virtual +environment's interpreter rather than the global one. To run the global +interpreter, either deactivate the virtual environment, or explicitly specify +the global Python version. + From a script ^^^^^^^^^^^^^ @@ -326,7 +479,7 @@ From file associations ^^^^^^^^^^^^^^^^^^^^^^ The launcher should have been associated with Python files (i.e. ``.py``, -``.pyw``, ``.pyc``, ``.pyo`` files) when it was installed. This means that +``.pyw``, ``.pyc`` files) when it was installed. This means that when you double-click on one of these files from Windows explorer the launcher will be used, and therefore you can use the same facilities described above to have the script specify the version which should be used. @@ -365,6 +518,16 @@ be used by the launcher without modification. If you are writing a new script on Windows which you hope will be useful on Unix, you should use one of the shebang lines starting with ``/usr``. +Any of the above virtual commands can be suffixed with an explicit version +(either just the major version, or the major and minor version) - for example +``/usr/bin/python2.7`` - which will cause that specific version to be located +and used. + +The ``/usr/bin/env`` form of shebang line has one further special property. +Before looking for installed Python interpreters, this form will search the +executable :envvar:`PATH` for a Python executable. This corresponds to the +behaviour of the Unix ``env`` program, which performs a :envvar:`PATH` search. + Arguments in shebang lines -------------------------- @@ -383,17 +546,16 @@ Customization Customization via INI files ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Two .ini files will be searched by the launcher - ``py.ini`` in the - current user's "application data" directory (i.e. the directory returned - by calling the Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) - and ``py.ini`` in the same directory as the launcher. The same .ini - files are used for both the 'console' version of the launcher (i.e. - py.exe) and for the 'windows' version (i.e. pyw.exe) +Two .ini files will be searched by the launcher - ``py.ini`` in the current +user's "application data" directory (i.e. the directory returned by calling the +Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) and ``py.ini`` in the +same directory as the launcher. The same .ini files are used for both the +'console' version of the launcher (i.e. py.exe) and for the 'windows' version +(i.e. pyw.exe) - Customization specified in the "application directory" will have - precedence over the one next to the executable, so a user, who may not - have write access to the .ini file next to the launcher, can override - commands in that global .ini file) +Customization specified in the "application directory" will have precedence over +the one next to the executable, so a user, who may not have write access to the +.ini file next to the launcher, can override commands in that global .ini file) Customizing default Python versions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -488,6 +650,99 @@ particular version was chosen and the exact command-line used to execute the target Python. + +.. finding_modules: + +Finding modules +=============== + +Python usually stores its library (and thereby your site-packages folder) in the +installation directory. So, if you had installed Python to +:file:`C:\\Python\\`, the default library would reside in +:file:`C:\\Python\\Lib\\` and third-party modules should be stored in +:file:`C:\\Python\\Lib\\site-packages\\`. + +This is how :data:`sys.path` is populated on Windows: + +* An empty entry is added at the start, which corresponds to the current + directory. + +* If the environment variable :envvar:`PYTHONPATH` exists, as described in + :ref:`using-on-envvars`, its entries are added next. Note that on Windows, + paths in this variable must be separated by semicolons, to distinguish them + from the colon used in drive identifiers (``C:\`` etc.). + +* Additional "application paths" can be added in the registry as subkeys of + :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the + ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have + semicolon-delimited path strings as their default value will cause each path + to be added to :data:`sys.path`. (Note that all known installers only use + HKLM, so HKCU is typically empty.) + +* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as + "Python Home". Otherwise, the path of the main Python executable is used to + locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a + Python home is found, the relevant sub-directories added to :data:`sys.path` + (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core + Python path is constructed from the PythonPath stored in the registry. + +* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in + the environment, and no registry entries can be found, a default path with + relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). + +If a ``pyvenv.cfg`` file is found alongside the main executable or in the +directory one level above the executable, the following variations apply: + +* If ``home`` is an absolute path and :envvar:`PYTHONHOME` is not set, this + path is used instead of the path to the main executable when deducing the + home location. + +* If ``applocal`` is set to true, the ``home`` property or the main executable + is always used as the home path, and all environment variables or registry + values affecting the path are ignored. The landmark file is not checked. + +The end result of all this is: + +* When running :file:`python.exe`, or any other .exe in the main Python + directory (either an installed version, or directly from the PCbuild + directory), the core path is deduced, and the core paths in the registry are + ignored. Other "application paths" in the registry are always read. + +* When Python is hosted in another .exe (different directory, embedded via COM, + etc), the "Python Home" will not be deduced, so the core path from the + registry is used. Other "application paths" in the registry are always read. + +* If Python can't find its home and there are no registry value (frozen .exe, + some very strange installation setup) you get a path with some default, but + relative, paths. + +For those who want to bundle Python into their application or distribution, the +following advice will prevent conflicts with other installations: + +* Include a ``pyvenv.cfg`` file alongside your executable containing + ``applocal = true``. This will ensure that your own directory will be used to + resolve paths even if you have included the standard library in a ZIP file. + It will also ignore user site-packages and other paths listed in the + registry. + +* If you are loading :file:`python3.dll` or :file:`python35.dll` in your own + executable, explicitly call :c:func:`Py_SetPath` or (at least) + :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`. + +* Clear and/or overwrite :envvar:`PYTHONPATH` and set :envvar:`PYTHONHOME` + before launching :file:`python.exe` from your application. + +* If you cannot use the previous suggestions (for example, you are a + distribution that allows people to run :file:`python.exe` directly), ensure + that the landmark file (:file:`Lib\\os.py`) exists in your install directory. + (Note that it will not be detected inside a ZIP file.) + +These will ensure that the files in a system-wide installation will not take +precedence over the copy of the standard library bundled with your application. +Otherwise, your users may experience problems using your application. Note that +the first suggestion is the best, as the other may still be susceptible to +non-standard paths in the registry and user site-packages. + Additional modules ================== @@ -498,7 +753,6 @@ and external, and snippets exist to use these features. The Windows-specific standard modules are documented in :ref:`mswin-specific-services`. - PyWin32 ------- @@ -557,20 +811,8 @@ latest release's source or just grab a fresh `checkout <https://docs.python.org/devguide/setup.html#getting-the-source-code>`_. The source tree contains a build solution and project files for Microsoft -Visual C++, which is the compiler used to build the official Python releases. -View the :file:`readme.txt` in their respective directories: - -+--------------------+--------------+-----------------------+ -| Directory | MSVC version | Visual Studio version | -+====================+==============+=======================+ -| :file:`PC/VS9.0/` | 9.0 | 2008 | -+--------------------+--------------+-----------------------+ -| :file:`PCbuild/` | 10.0 | 2010 | -+--------------------+--------------+-----------------------+ - -Note that any build directories within the :file:`PC` directory are not -necessarily fully supported. The :file:`PCbuild` directory contains the files -for the compiler used to build the official release. +Visual Studio 2015, which is the compiler used to build the official Python +releases. These files are in the :file:`PCbuild` directory. Check :file:`PCbuild/readme.txt` for general information on the build process. @@ -588,6 +830,83 @@ For extension modules, consult :ref:`building-on-windows`. by Trent Apted et al, 2007 +Embedded Distribution +===================== + +.. versionadded:: 3.5 + +The embedded distribution is a ZIP file containing a minimal Python environment. +It is intended for acting as part of another application, rather than being +directly accessed by end-users. + +When extracted, the embedded distribution is (almost) fully isolated from the +user's system, including environment variables, system registry settings, and +installed packages. The standard library is included as pre-compiled and +optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python35.dll``, +``python.exe`` and ``pythonw.exe`` are all provided. Tcl/tk (including all +dependants, such as Idle), pip and the Python documentation are not included. + +.. note:: + + The embedded distribution does not include the `Microsoft C Runtime + <http://www.microsoft.com/en-us/download/details.aspx?id=48145>`_ and it is + the responsibility of the application installer to provide this. The + runtime may have already been installed on a user's system previously or + automatically via Windows Update, and can be detected by finding + ``ucrtbase.dll`` in the system directory. + +Third-party packages should be installed by the application installer alongside +the embedded distribution. Using pip to manage dependencies as for a regular +Python installation is not supported with this distribution, though with some +care it may be possible to include and use pip for automatic updates. In +general, third-party packages should be treated as part of the application +("vendoring") so that the developer can ensure compatibility with newer +versions before providing updates to users. + +The two recommended use cases for this distribution are described below. + +Python Application +------------------ + +An application written in Python does not necessarily require users to be aware +of that fact. The embedded distribution may be used in this case to include a +private version of Python in an install package. Depending on how transparent it +should be (or conversely, how professional it should appear), there are two +options. + +Using a specialized executable as a launcher requires some coding, but provides +the most transparent experience for users. With a customized launcher, there are +no obvious indications that the program is running on Python: icons can be +customized, company and version information can be specified, and file +associations behave properly. In most cases, a custom launcher should simply be +able to call ``Py_Main`` with a hard-coded command line. + +The simpler approach is to provide a batch file or generated shortcut that +directly calls the ``python.exe`` or ``pythonw.exe`` with the required +command-line arguments. In this case, the application will appear to be Python +and not its actual name, and users may have trouble distinguishing it from other +running Python processes or file associations. + +With the latter approach, packages should be installed as directories alongside +the Python executable to ensure they are available on the path. With the +specialized launcher, packages can be located in other locations as there is an +opportunity to specify the search path before launching the application. + +Embedding Python +---------------- + +Applications written in native code often require some form of scripting +language, and the embedded Python distribution can be used for this purpose. In +general, the majority of the application is in native code, and some part will +either invoke ``python.exe`` or directly use ``python3.dll``. For either case, +extracting the embedded distribution to a subdirectory of the application +installation is sufficient to provide a loadable Python interpreter. + +As with the application use, packages can be installed to any location as there +is an opportunity to specify search paths before initializing the interpreter. +Otherwise, there is no fundamental differences between using the embedded +distribution and a regular installation. + Other resources =============== @@ -603,5 +922,3 @@ Other resources :pep:`397` - Python launcher for Windows The proposal for the launcher to be included in the Python distribution. - - diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst new file mode 100644 index 0000000000..cda8a6d04a --- /dev/null +++ b/Doc/whatsnew/3.5.rst @@ -0,0 +1,2482 @@ +**************************** + What's New In Python 3.5 +**************************** + +:Editors: Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io> + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.5, compared to 3.4. +Python 3.5 was released on September 13, 2015. See the +`changelog <https://docs.python.org/3.5/whatsnew/changelog.html>`_ for a full +list of changes. + +.. seealso:: + + :pep:`478` - Python 3.5 Release Schedule + + +Summary -- Release highlights +============================= + +New syntax features: + +* :ref:`PEP 492 <whatsnew-pep-492>`, coroutines with async and await syntax. +* :ref:`PEP 465 <whatsnew-pep-465>`, a new matrix multiplication operator: ``a @ b``. +* :ref:`PEP 448 <whatsnew-pep-448>`, additional unpacking generalizations. + + +New library modules: + +* :mod:`typing`: :ref:`PEP 484 -- Type Hints <whatsnew-pep-484>`. +* :mod:`zipapp`: :ref:`PEP 441 Improving Python ZIP Application Support + <whatsnew-zipapp>`. + + +New built-in features: + +* ``bytes % args``, ``bytearray % args``: :ref:`PEP 461 <whatsnew-pep-461>` -- + Adding ``%`` formatting to bytes and bytearray. + +* New :meth:`bytes.hex`, :meth:`bytearray.hex` and :meth:`memoryview.hex` + methods. (Contributed by Arnon Yaari in :issue:`9951`.) + +* :class:`memoryview` now supports tuple indexing (including multi-dimensional). + (Contributed by Antoine Pitrou in :issue:`23632`.) + +* Generators have a new ``gi_yieldfrom`` attribute, which returns the + object being iterated by ``yield from`` expressions. (Contributed + by Benno Leslie and Yury Selivanov in :issue:`24450`.) + +* A new :exc:`RecursionError` exception is now raised when maximum + recursion depth is reached. (Contributed by Georg Brandl + in :issue:`19235`.) + + +CPython implementation improvements: + +* When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), + :py:data:`sys.stdin` and :py:data:`sys.stdout` now use the + ``surrogateescape`` error handler, instead of the ``strict`` error handler. + (Contributed by Victor Stinner in :issue:`19977`.) + +* ``.pyo`` files are no longer used and have been replaced by a more flexible + scheme that includes the optimization level explicitly in ``.pyc`` name. + (See :ref:`PEP 488 overview <whatsnew-pep-488>`.) + +* Builtin and extension modules are now initialized in a multi-phase process, + which is similar to how Python modules are loaded. + (See :ref:`PEP 489 overview <whatsnew-pep-489>`.) + + +Significant improvements in the standard library: + +* :class:`collections.OrderedDict` is now + :ref:`implemented in C <whatsnew-ordereddict>`, which makes it + 4 to 100 times faster. + +* :mod:`ssl` module gained + :ref:`support for Memory BIO <whatsnew-sslmemorybio>`, which decouples SSL + protocol handling from network IO. + +* The new :func:`os.scandir` function provides a + :ref:`better and significantly faster way <whatsnew-pep-471>` + of directory traversal. + +* :func:`functools.lru_cache` has been mostly + :ref:`reimplemented in C <whatsnew-lrucache>`, yielding much better + performance. + +* The new :func:`subprocess.run` function provides a + :ref:`streamlined way to run subprocesses <whatsnew-subprocess>`. + +* The :mod:`traceback` module has been significantly + :ref:`enhanced <whatsnew-traceback>` for improved + performance and developer convenience. + + +Security improvements: + +* SSLv3 is now disabled throughout the standard library. + It can still be enabled by instantiating a :class:`ssl.SSLContext` + manually. (See :issue:`22638` for more details; this change was + backported to CPython 3.4 and 2.7.) + +* HTTP cookie parsing is now stricter, in order to protect + against potential injection attacks. (Contributed by Antoine Pitrou + in :issue:`22796`.) + + +Windows improvements: + +* A new installer for Windows has replaced the old MSI. + See :ref:`using-on-windows` for more information. + +* Windows builds now use Microsoft Visual C++ 14.0, and extension modules + should use the same. + + +Please read on for a comprehensive list of user-facing changes, including many +other smaller improvements, CPython optimizations, deprecations, and potential +porting issues. + + +New Features +============ + +.. _whatsnew-pep-492: + +PEP 492 - Coroutines with async and await syntax +------------------------------------------------ + +:pep:`492` greatly improves support for asynchronous programming in Python +by adding :term:`awaitable objects <awaitable>`, +:term:`coroutine functions <coroutine function>`, +:term:`asynchronous iteration <asynchronous iterable>`, +and :term:`asynchronous context managers <asynchronous context manager>`. + +Coroutine functions are declared using the new :keyword:`async def` syntax:: + + >>> async def coro(): + ... return 'spam' + +Inside a coroutine function, the new :keyword:`await` expression can be used +to suspend coroutine execution until the result is available. Any object +can be *awaited*, as long as it implements the :term:`awaitable` protocol by +defining the :meth:`__await__` method. + +PEP 492 also adds :keyword:`async for` statement for convenient iteration +over asynchronous iterables. + +An example of a rudimentary HTTP client written using the new syntax:: + + import asyncio + + async def http_get(domain): + reader, writer = await asyncio.open_connection(domain, 80) + + writer.write(b'\r\n'.join([ + b'GET / HTTP/1.1', + b'Host: %b' % domain.encode('latin-1'), + b'Connection: close', + b'', b'' + ])) + + async for line in reader: + print('>>>', line) + + writer.close() + + loop = asyncio.get_event_loop() + try: + loop.run_until_complete(http_get('example.com')) + finally: + loop.close() + + +Similarly to asynchronous iteration, there is a new syntax for asynchronous +context managers. The following script:: + + import asyncio + + async def coro(name, lock): + print('coro {}: waiting for lock'.format(name)) + async with lock: + print('coro {}: holding the lock'.format(name)) + await asyncio.sleep(1) + print('coro {}: releasing the lock'.format(name)) + + loop = asyncio.get_event_loop() + lock = asyncio.Lock() + coros = asyncio.gather(coro(1, lock), coro(2, lock)) + try: + loop.run_until_complete(coros) + finally: + loop.close() + +will output:: + + coro 2: waiting for lock + coro 2: holding the lock + coro 1: waiting for lock + coro 2: releasing the lock + coro 1: holding the lock + coro 1: releasing the lock + +Note that both :keyword:`async for` and :keyword:`async with` can only +be used inside a coroutine function declared with :keyword:`async def`. + +Coroutine functions are intended to be run inside a compatible event loop, +such as the :ref:`asyncio loop <asyncio-event-loop>`. + +.. seealso:: + + :pep:`492` -- Coroutines with async and await syntax + PEP written and implemented by Yury Selivanov. + + +.. _whatsnew-pep-465: + +PEP 465 - A dedicated infix operator for matrix multiplication +-------------------------------------------------------------- + +:pep:`465` adds the ``@`` infix operator for matrix multiplication. +Currently, no builtin Python types implement the new operator, however, it +can be implemented by defining :meth:`__matmul__`, :meth:`__rmatmul__`, +and :meth:`__imatmul__` for regular, reflected, and in-place matrix +multiplication. The semantics of these methods is similar to that of +methods defining other infix arithmetic operators. + +Matrix multiplication is a notably common operation in many fields of +mathematics, science, engineering, and the addition of ``@`` allows writing +cleaner code:: + + S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) + +instead of:: + + S = dot((dot(H, beta) - r).T, + dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r)) + +An upcoming release of NumPy 1.10 will add support for the new operator:: + + >>> import numpy + + >>> x = numpy.ones(3) + >>> x + array([ 1., 1., 1.]) + + >>> m = numpy.eye(3) + >>> m + array([[ 1., 0., 0.], + [ 0., 1., 0.], + [ 0., 0., 1.]]) + + >>> x @ m + array([ 1., 1., 1.]) + + +.. seealso:: + + :pep:`465` -- A dedicated infix operator for matrix multiplication + PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson. + + +.. _whatsnew-pep-448: + +PEP 448 - Additional Unpacking Generalizations +---------------------------------------------- + +:pep:`448` extends the allowed uses of the ``*`` iterable unpacking +operator and ``**`` dictionary unpacking operator. It is now possible +to use an arbitrary number of unpackings in function calls:: + + >>> print(*[1], *[2], 3, *[4, 5]) + 1 2 3 4 5 + + >>> def fn(a, b, c, d): + ... print(a, b, c, d) + ... + + >>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4}) + 1 2 3 4 + +Similarly, tuple, list, set, and dictionary displays allow multiple +unpackings:: + + >>> *range(4), 4 + (0, 1, 2, 3, 4) + + >>> [*range(4), 4] + [0, 1, 2, 3, 4] + + >>> {*range(4), 4, *(5, 6, 7)} + {0, 1, 2, 3, 4, 5, 6, 7} + + >>> {'x': 1, **{'y': 2}} + {'x': 1, 'y': 2} + +.. seealso:: + + :pep:`448` -- Additional Unpacking Generalizations + PEP written by Joshua Landau; implemented by Neil Girdhar, + Thomas Wouters, and Joshua Landau. + + +.. _whatsnew-pep-461: + +PEP 461 - % formatting support for bytes and bytearray +------------------------------------------------------ + +:pep:`461` adds support for ``%`` +:ref:`interpolation operator <bytes-formatting>` to :class:`bytes` +and :class:`bytearray`. + +While interpolation is usually thought of as a string operation, there are +cases where interpolation on ``bytes`` or ``bytearrays`` makes sense, and the +work needed to make up for this missing functionality detracts from the +overall readability of the code. This issue is particularly important when +dealing with wire format protocols, which are often a mixture of binary +and ASCII compatible text. + +Examples:: + + >>> b'Hello %b!' % b'World' + b'Hello World!' + + >>> b'x=%i y=%f' % (1, 2.5) + b'x=1 y=2.500000' + +Unicode is not allowed for ``%b``, but it is accepted by ``%a`` (equivalent of +``repr(obj).encode('ascii', 'backslashreplace')``):: + + >>> b'Hello %b!' % 'World' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' + + >>> b'price: %a' % '10€' + b"price: '10\\u20ac'" + +Note that ``%s`` and ``%r`` conversion types, although supported, should +only be used in codebases that need compatibility with Python 2. + +.. seealso:: + + :pep:`461` -- Adding % formatting to bytes and bytearray + PEP written by Ethan Furman; implemented by Neil Schemenauer and + Ethan Furman. + + +.. _whatsnew-pep-484: + +PEP 484 - Type Hints +-------------------- + +Function annotation syntax has been a Python feature since version 3.0 +(:pep:`3107`), however the semantics of annotations has been left undefined. + +Experience has shown that the majority of function annotation +uses were to provide type hints to function parameters and return values. It +became evident that it would be beneficial for Python users, if the +standard library included the base definitions and tools for type annotations. + +:pep:`484` introduces a :term:`provisional module <provisional api>` to +provide these standard definitions and tools, along with some conventions +for situations where annotations are not available. + +For example, here is a simple function whose argument and return type +are declared in the annotations:: + + def greeting(name: str) -> str: + return 'Hello ' + name + +While these annotations are available at runtime through the usual +:attr:`__annotations__` attribute, *no automatic type checking happens at +runtime*. Instead, it is assumed that a separate off-line type checker +(e.g. `mypy <http://mypy-lang.org>`_) will be used for on-demand +source code analysis. + +The type system supports unions, generic types, and a special type +named :class:`~typing.Any` which is consistent with (i.e. assignable to +and from) all types. + +.. seealso:: + + * :mod:`typing` module documentation + * :pep:`484` -- Type Hints + PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa; + implemented by Guido van Rossum. + * :pep:`483` -- The Theory of Type Hints + PEP written by Guido van Rossum + + +.. _whatsnew-pep-471: + +PEP 471 - os.scandir() function -- a better and faster directory iterator +------------------------------------------------------------------------- + +:pep:`471` adds a new directory iteration function, :func:`os.scandir`, +to the standard library. Additionally, :func:`os.walk` is now +implemented using ``scandir``, which makes it 3 to 5 times faster +on POSIX systems and 7 to 20 times faster on Windows systems. This is +largely achieved by greatly reducing the number of calls to :func:`os.stat` +required to walk a directory tree. + +Additionally, ``scandir`` returns an iterator, as opposed to returning +a list of file names, which improves memory efficiency when iterating +over very large directories. + +The following example shows a simple use of :func:`os.scandir` to display all +the files (excluding directories) in the given *path* that don't start with +``'.'``. The :meth:`entry.is_file() <os.DirEntry.is_file>` call will generally +not make an additional system call:: + + for entry in os.scandir(path): + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) + +.. seealso:: + + :pep:`471` -- os.scandir() function -- a better and faster directory iterator + PEP written and implemented by Ben Hoyt with the help of Victor Stinner. + + +.. _whatsnew-pep-475: + +PEP 475: Retry system calls failing with EINTR +---------------------------------------------- + +A :py:data:`errno.EINTR` error code is returned whenever a system call, that +is waiting for I/O, is interrupted by a signal. Previously, Python would +raise :exc:`InterruptedError` in such case. This meant that, when writing a +Python application, the developer had two choices: + +#. Ignore the ``InterruptedError``. +#. Handle the ``InterruptedError`` and attempt to restart the interrupted + system call at every call site. + +The first option makes an application fail intermittently. +The second option adds a large amount of boilerplate that makes the +code nearly unreadable. Compare:: + + print("Hello World") + +and:: + + while True: + try: + print("Hello World") + break + except InterruptedError: + continue + +:pep:`475` implements automatic retry of system calls on +``EINTR``. This removes the burden of dealing with ``EINTR`` +or :exc:`InterruptedError` in user code in most situations and makes +Python programs, including the standard library, more robust. Note that +the system call is only retried if the signal handler does not raise an +exception. + +Below is a list of functions which are now retried when interrupted +by a signal: + +* :func:`open` and :func:`io.open`; + +* functions of the :mod:`faulthandler` module; + +* :mod:`os` functions: :func:`~os.fchdir`, :func:`~os.fchmod`, + :func:`~os.fchown`, :func:`~os.fdatasync`, :func:`~os.fstat`, + :func:`~os.fstatvfs`, :func:`~os.fsync`, :func:`~os.ftruncate`, + :func:`~os.mkfifo`, :func:`~os.mknod`, :func:`~os.open`, + :func:`~os.posix_fadvise`, :func:`~os.posix_fallocate`, :func:`~os.pread`, + :func:`~os.pwrite`, :func:`~os.read`, :func:`~os.readv`, :func:`~os.sendfile`, + :func:`~os.wait3`, :func:`~os.wait4`, :func:`~os.wait`, + :func:`~os.waitid`, :func:`~os.waitpid`, :func:`~os.write`, + :func:`~os.writev`; + +* special cases: :func:`os.close` and :func:`os.dup2` now ignore + :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP + for the rationale); + +* :mod:`select` functions: :func:`devpoll.poll() <select.devpoll.poll>`, + :func:`epoll.poll() <select.epoll.poll>`, + :func:`kqueue.control() <select.kqueue.control>`, + :func:`poll.poll() <select.poll.poll>`, :func:`~select.select`; + +* methods of the :class:`~socket.socket` class: :meth:`~socket.socket.accept`, + :meth:`~socket.socket.connect` (except for non-blocking sockets), + :meth:`~socket.socket.recv`, :meth:`~socket.socket.recvfrom`, + :meth:`~socket.socket.recvmsg`, :meth:`~socket.socket.send`, + :meth:`~socket.socket.sendall`, :meth:`~socket.socket.sendmsg`, + :meth:`~socket.socket.sendto`; + +* :func:`signal.sigtimedwait` and :func:`signal.sigwaitinfo`; + +* :func:`time.sleep`. + +.. seealso:: + + :pep:`475` -- Retry system calls failing with EINTR + PEP and implementation written by Charles-François Natali and + Victor Stinner, with the help of Antoine Pitrou (the French connection). + + +.. _whatsnew-pep-479: + +PEP 479: Change StopIteration handling inside generators +-------------------------------------------------------- + +The interaction of generators and :exc:`StopIteration` in Python 3.4 and +earlier was sometimes surprising, and could conceal obscure bugs. Previously, +``StopIteration`` raised accidentally inside a generator function was +interpreted as the end of the iteration by the loop construct driving the +generator. + +:pep:`479` changes the behavior of generators: when a ``StopIteration`` +exception is raised inside a generator, it is replaced with a +:exc:`RuntimeError` before it exits the generator frame. The main goal of +this change is to ease debugging in the situation where an unguarded +:func:`next` call raises ``StopIteration`` and causes the iteration controlled +by the generator to terminate silently. This is particularly pernicious in +combination with the ``yield from`` construct. + +This is a backwards incompatible change, so to enable the new behavior, +a :term:`__future__` import is necessary:: + + >>> from __future__ import generator_stop + + >>> def gen(): + ... next(iter([])) + ... yield + ... + >>> next(gen()) + Traceback (most recent call last): + File "<stdin>", line 2, in gen + StopIteration + + The above exception was the direct cause of the following exception: + + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + RuntimeError: generator raised StopIteration + +Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be +raised whenever a ``StopIteration`` exception is raised inside a generator. + +.. seealso:: + + :pep:`479` -- Change StopIteration handling inside generators + PEP written by Chris Angelico and Guido van Rossum. Implemented by + Chris Angelico, Yury Selivanov and Nick Coghlan. + + +.. _whatsnew-pep-485: + +PEP 485: A function for testing approximate equality +---------------------------------------------------- + +:pep:`485` adds the :func:`math.isclose` and :func:`cmath.isclose` +functions which tell whether two values are approximately equal or +"close" to each other. Whether or not two values are considered +close is determined according to given absolute and relative tolerances. +Relative tolerance is the maximum allowed difference between ``isclose`` +arguments, relative to the larger absolute value:: + + >>> import math + >>> a = 5.0 + >>> b = 4.99998 + >>> math.isclose(a, b, rel_tol=1e-5) + True + >>> math.isclose(a, b, rel_tol=1e-6) + False + +It is also possible to compare two values using absolute tolerance, which +must be a non-negative value:: + + >>> import math + >>> a = 5.0 + >>> b = 4.99998 + >>> math.isclose(a, b, abs_tol=0.00003) + True + >>> math.isclose(a, b, abs_tol=0.00001) + False + +.. seealso:: + + :pep:`485` -- A function for testing approximate equality + PEP written by Christopher Barker; implemented by Chris Barker and + Tal Einat. + + +.. _whatsnew-pep-486: + +PEP 486: Make the Python Launcher aware of virtual environments +--------------------------------------------------------------- + +:pep:`486` makes the Windows launcher (see :pep:`397`) aware of an active +virtual environment. When the default interpreter would be used and the +``VIRTUAL_ENV`` environment variable is set, the interpreter in the virtual +environment will be used. + +.. seealso:: + + :pep:`486` -- Make the Python Launcher aware of virtual environments + PEP written and implemented by Paul Moore. + + +.. _whatsnew-pep-488: + +PEP 488: Elimination of PYO files +--------------------------------- + +:pep:`488` does away with the concept of ``.pyo`` files. This means that +``.pyc`` files represent both unoptimized and optimized bytecode. To prevent the +need to constantly regenerate bytecode files, ``.pyc`` files now have an +optional ``opt-`` tag in their name when the bytecode is optimized. This has the +side-effect of no more bytecode file name clashes when running under either +:option:`-O` or :option:`-OO`. Consequently, bytecode files generated from +:option:`-O`, and :option:`-OO` may now exist simultaneously. +:func:`importlib.util.cache_from_source` has an updated API to help with +this change. + +.. seealso:: + + :pep:`488` -- Elimination of PYO files + PEP written and implemented by Brett Cannon. + + +.. _whatsnew-pep-489: + +PEP 489: Multi-phase extension module initialization +---------------------------------------------------- + +:pep:`489` updates extension module initialization to take advantage of the +two step module loading mechanism introduced by :pep:`451` in Python 3.4. + +This change brings the import semantics of extension modules that opt-in to +using the new mechanism much closer to those of Python source and bytecode +modules, including the ability to use any valid identifier as a module name, +rather than being restricted to ASCII. + +.. seealso:: + + :pep:`489` -- Multi-phase extension module initialization + PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan; + implemented by Petr Viktorin. + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* Added the ``"namereplace"`` error handlers. The ``"backslashreplace"`` + error handlers now work with decoding and translating. + (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) + +* The :option:`-b` option now affects comparisons of :class:`bytes` with + :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`.) + +* New Kazakh ``kz1048`` and Tajik ``koi8_t`` :ref:`codecs <standard-encodings>`. + (Contributed by Serhiy Storchaka in :issue:`22682` and :issue:`22681`.) + +* Property docstrings are now writable. This is especially useful for + :func:`collections.namedtuple` docstrings. + (Contributed by Berker Peksag in :issue:`24064`.) + +* Circular imports involving relative imports are now supported. + (Contributed by Brett Cannon and Antoine Pitrou in :issue:`17636`.) + + +New Modules +=========== + +typing +------ + +The new :mod:`typing` :term:`provisional <provisional api>` module +provides standard definitions and tools for function type annotations. +See :ref:`Type Hints <whatsnew-pep-484>` for more information. + +.. _whatsnew-zipapp: + +zipapp +------ + +The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and +command line tool for creating executable Python Zip Applications, which +were introduced in Python 2.6 in :issue:`1739468`, but which were not well +publicized, either at the time or since. + +With the new module, bundling your application is as simple as putting all +the files, including a ``__main__.py`` file, into a directory ``myapp`` +and running:: + + $ python -m zipapp myapp + $ python myapp.pyz + +The module implementation has been contributed by Paul Moore in +:issue:`23491`. + +.. seealso:: + + :pep:`441` -- Improving Python ZIP Application Support + + +Improved Modules +================ + +argparse +-------- + +The :class:`~argparse.ArgumentParser` class now allows to disable +:ref:`abbreviated usage <prefix-matching>` of long options by setting +:ref:`allow_abbrev` to ``False``. (Contributed by Jonathan Paugh, +Steven Bethard, paul j3 and Daniel Eriksson in :issue:`14910`.) + + +asyncio +------- + +Since :mod:`asyncio` module is :term:`provisional <provisional api>`, +all changes introduced in Python 3.5 have also been backported to Python 3.4.x. + +Notable changes in the :mod:`asyncio` module since Python 3.4.0: + +* New debugging APIs: :meth:`loop.set_debug() <asyncio.BaseEventLoop.set_debug>` + and :meth:`loop.get_debug() <asyncio.BaseEventLoop.get_debug>` methods. + (Contributed by Victor Stinner.) + +* The proactor event loop now supports SSL. + (Contributed by Antoine Pitrou and Victor Stinner in :issue:`22560`.) + +* A new :meth:`loop.is_closed() <asyncio.BaseEventLoop.is_closed>` method to + check if the event loop is closed. + (Contributed by Victor Stinner in :issue:`21326`.) + +* A new :meth:`loop.create_task() <asyncio.BaseEventLoop.create_task>` + to conveniently create and schedule a new :class:`~asyncio.Task` + for a coroutine. The ``create_task`` method is also used by all + asyncio functions that wrap coroutines into tasks, such as + :func:`asyncio.wait`, :func:`asyncio.gather`, etc. + (Contributed by Victor Stinner.) + +* A new :meth:`transport.get_write_buffer_limits() <asyncio.WriteTransport.get_write_buffer_limits>` + method to inquire for *high-* and *low-* water limits of the flow + control. + (Contributed by Victor Stinner.) + +* The :func:`~asyncio.async` function is deprecated in favor of + :func:`~asyncio.ensure_future`. + (Contributed by Yury Selivanov.) + +* New :meth:`loop.set_task_factory() <asyncio.BaseEventLoop.set_task_factory>` + and :meth:`loop.set_task_factory() <asyncio.BaseEventLoop.get_task_factory>` + methods to customize the task factory that + :meth:`loop.create_task() <asyncio.BaseEventLoop.create_task>` method uses. + (Contributed by Yury Selivanov.) + +* New :meth:`Queue.join() <asyncio.Queue.join>` and + :meth:`Queue.task_done() <asyncio.Queue.task_done>` queue methods. + (Contributed by Victor Stinner.) + +* The ``JoinableQueue`` class was removed, in favor of the + :class:`asyncio.Queue` class. + (Contributed by Victor Stinner.) + +Updates in 3.5.1: + +* The :func:`~asyncio.ensure_future` function and all functions that + use it, such as :meth:`loop.run_until_complete() <asyncio.BaseEventLoop.run_until_complete>`, + now accept all kinds of :term:`awaitable objects <awaitable>`. + (Contributed by Yury Selivanov.) + + +bz2 +--- + +The :meth:`BZ2Decompressor.decompress <bz2.BZ2Decompressor.decompress>` +method now accepts an optional *max_length* argument to limit the maximum +size of decompressed data. (Contributed by Nikolaus Rath in :issue:`15955`.) + + +cgi +--- + +The :class:`~cgi.FieldStorage` class now supports the :term:`context manager` +protocol. (Contributed by Berker Peksag in :issue:`20289`.) + + +cmath +----- + +A new function :func:`~cmath.isclose` provides a way to test for approximate +equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + + +code +---- + +The :func:`InteractiveInterpreter.showtraceback() <code.InteractiveInterpreter.showtraceback>` +method now prints the full chained traceback, just like the interactive +interpreter. (Contributed by Claudiu Popa in :issue:`17442`.) + + +collections +----------- + +.. _whatsnew-ordereddict: + +The :class:`~collections.OrderedDict` class is now implemented in C, which +makes it 4 to 100 times faster. (Contributed by Eric Snow in :issue:`16991`.) + +:meth:`OrderedDict.items() <collections.OrderedDict.items>`, +:meth:`OrderedDict.keys() <collections.OrderedDict.keys>`, +:meth:`OrderedDict.values() <collections.OrderedDict.values>` views now support +:func:`reversed` iteration. +(Contributed by Serhiy Storchaka in :issue:`19505`.) + +The :class:`~collections.deque` class now defines +:meth:`~collections.deque.index`, :meth:`~collections.deque.insert`, and +:meth:`~collections.deque.copy`, as well as supports ``+`` and ``*`` operators. +This allows deques to be recognized as a :class:`~collections.abc.MutableSequence` +and improves their substitutability for lists. +(Contributed by Raymond Hettinger :issue:`23704`.) + +Docstrings produced by :func:`~collections.namedtuple` can now be updated:: + + Point = namedtuple('Point', ['x', 'y']) + Point.__doc__ = 'ordered pair' + Point.x.__doc__ = 'abscissa' + Point.y.__doc__ = 'ordinate' + +(Contributed by Berker Peksag in :issue:`24064`.) + +The :class:`~collections.UserString` class now implements +:meth:`__getnewargs__`, :meth:`__rmod__`, :meth:`~str.casefold`, +:meth:`~str.format_map`, :meth:`~str.isprintable`, and :meth:`~str.maketrans` +methods to match the corresponding methods of :class:`str`. +(Contributed by Joe Jevnik in :issue:`22189`.) + + +collections.abc +--------------- + +The :meth:`Sequence.index() <collections.abc.Sequence.index>` method now +accepts *start* and *stop* arguments to match the corresponding methods +of :class:`tuple`, :class:`list`, etc. +(Contributed by Devin Jeanpierre in :issue:`23086`.) + +A new :class:`~collections.abc.Generator` abstract base class. (Contributed +by Stefan Behnel in :issue:`24018`.) + +New :class:`~collections.abc.Awaitable` :class:`~collections.abc.Coroutine`, +:class:`~collections.abc.AsyncIterator`, and +:class:`~collections.abc.AsyncIterable` abstract base classes. +(Contributed by Yury Selivanov in :issue:`24184`.) + +For earlier Python versions, a backport of the new ABCs is available in an +external `PyPI package <https://pypi.python.org/pypi/backports_abc>`_. + + +compileall +---------- + +A new :mod:`compileall` option, :samp:`-j {N}`, allows to run *N* workers +sumultaneously to perform parallel bytecode compilation. +The :func:`~compileall.compile_dir` function has a corresponding ``workers`` +parameter. (Contributed by Claudiu Popa in :issue:`16104`.) + +Another new option, ``-r``, allows to control the maximum recursion +level for subdirectories. (Contributed by Claudiu Popa in :issue:`19628`.) + +The ``-q`` command line option can now be specified more than once, in +which case all output, including errors, will be suppressed. The corresponding +``quiet`` parameter in :func:`~compileall.compile_dir`, +:func:`~compileall.compile_file`, and :func:`~compileall.compile_path` can now +accept an integer value indicating the level of output suppression. +(Contributed by Thomas Kluyver in :issue:`21338`.) + + +concurrent.futures +------------------ + +The :meth:`Executor.map() <concurrent.futures.Executor.map>` method now accepts a +*chunksize* argument to allow batching of tasks to improve performance when +:meth:`~concurrent.futures.ProcessPoolExecutor` is used. +(Contributed by Dan O'Reilly in :issue:`11271`.) + +A number of workers in :class:`~concurrent.futures.ThreadPoolExecutor` constructor is +optional now. The default value equals to 5 times the number of CPUs. +(Contributed by Claudiu Popa in :issue:`21527`.) + + +configparser +------------ + +:mod:`configparser` now provides a way to customize the conversion +of values by specifying a dictionary of converters in +:class:`~configparser.ConfigParser` constructor, or by defining them +as methods in ``ConfigParser`` subclasses. Converters defined in +parser instance are inherited by its section proxies. + +Example:: + + >>> import configparser + >>> conv = {} + >>> conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()] + >>> cfg = configparser.ConfigParser(converters=conv) + >>> cfg.read_string(""" + ... [s] + ... list = a b c d e f g + ... """) + >>> cfg.get('s', 'list') + 'a b c d e f g' + >>> cfg.getlist('s', 'list') + ['a', 'b', 'c', 'd', 'e', 'f', 'g'] + >>> section = cfg['s'] + >>> section.getlist('list') + ['a', 'b', 'c', 'd', 'e', 'f', 'g'] + +(Contributed by Łukasz Langa in :issue:`18159`.) + + +contextlib +---------- + +The new :func:`~contextlib.redirect_stderr` :term:`context manager` (similar to +:func:`~contextlib.redirect_stdout`) makes it easier for utility scripts to +handle inflexible APIs that write their output to :data:`sys.stderr` and +don't provide any options to redirect it:: + + >>> import contextlib, io, logging + >>> f = io.StringIO() + >>> with contextlib.redirect_stderr(f): + ... logging.warning('warning') + ... + >>> f.getvalue() + 'WARNING:root:warning\n' + +(Contributed by Berker Peksag in :issue:`22389`.) + + +csv +--- + +The :meth:`~csv.csvwriter.writerow` method now supports arbitrary iterables, +not just sequences. (Contributed by Serhiy Storchaka in :issue:`23171`.) + + +curses +------ + +The new :func:`~curses.update_lines_cols` function updates :envvar:`LINES` +and :envvar:`COLS` environment variables. This is useful for detecting +manual screen resize. (Contributed by Arnon Yaari in :issue:`4254`.) + + +dbm +--- + +:func:`dumb.open <dbm.dumb.open>` always creates a new database when the flag +has the value ``"n"``. (Contributed by Claudiu Popa in :issue:`18039`.) + + +difflib +------- + +The charset of HTML documents generated by +:meth:`HtmlDiff.make_file() <difflib.HtmlDiff.make_file>` +can now be customized by using a new *charset* keyword-only argument. +The default charset of HTML document changed from ``"ISO-8859-1"`` +to ``"utf-8"``. +(Contributed by Berker Peksag in :issue:`2052`.) + +The :func:`~difflib.diff_bytes` function can now compare lists of byte +strings. This fixes a regression from Python 2. +(Contributed by Terry J. Reedy and Greg Ward in :issue:`17445`.) + + +distutils +--------- + +Both ``build`` and ``build_ext`` commands now accept a ``-j`` option to +enable parallel building of extension modules. +(Contributed by Antoine Pitrou in :issue:`5309`.) + +The :mod:`distutils` module now supports ``xz`` compression, and can be +enabled by passing ``xztar`` as an argument to ``bdist --format``. +(Contributed by Serhiy Storchaka in :issue:`16314`.) + + +doctest +------- + +The :func:`~doctest.DocTestSuite` function returns an empty +:class:`unittest.TestSuite` if *module* contains no docstrings instead of +raising :exc:`ValueError`. (Contributed by Glenn Jones in :issue:`15916`.) + + +email +----- + +A new policy option :attr:`Policy.mangle_from_ <email.policy.Policy.mangle_from_>` +controls whether or not lines that start with ``"From "`` in email bodies are +prefixed with a ``">"`` character by generators. The default is ``True`` for +:attr:`~email.policy.compat32` and ``False`` for all other policies. +(Contributed by Milan Oberkirch in :issue:`20098`.) + +A new +:meth:`Message.get_content_disposition() <email.message.Message.get_content_disposition>` +method provides easy access to a canonical value for the +:mailheader:`Content-Disposition` header. +(Contributed by Abhilash Raj in :issue:`21083`.) + +A new policy option :attr:`EmailPolicy.utf8 <email.policy.EmailPolicy.utf8>` +can be set to ``True`` to encode email headers using the UTF-8 charset instead +of using encoded words. This allows ``Messages`` to be formatted according to +:rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` +``SMTPUTF8`` extension. (Contributed by R. David Murray in +:issue:`24211`.) + +The :class:`mime.text.MIMEText <email.mime.text.MIMEText>` constructor now +accepts a :class:`charset.Charset <email.charset.Charset>` instance. +(Contributed by Claude Paroz and Berker Peksag in :issue:`16324`.) + + +enum +---- + +The :class:`~enum.Enum` callable has a new parameter *start* to +specify the initial number of enum values if only *names* are provided:: + + >>> Animal = enum.Enum('Animal', 'cat dog', start=10) + >>> Animal.cat + <Animal.cat: 10> + >>> Animal.dog + <Animal.dog: 11> + +(Contributed by Ethan Furman in :issue:`21706`.) + + +faulthandler +------------ + +:func:`~faulthandler.enable`, :func:`~faulthandler.register`, +:func:`~faulthandler.dump_traceback` and +:func:`~faulthandler.dump_traceback_later` functions now accept file +descriptors in addition to file-like objects. +(Contributed by Wei Wu in :issue:`23566`.) + + +functools +--------- + +.. _whatsnew-lrucache: + +Most of :func:`~functools.lru_cache` machinery is now implemented in C, making +it significantly faster. (Contributed by Matt Joiner, Alexey Kachayev, and +Serhiy Storchaka in :issue:`14373`.) + + +glob +---- + +:func:`~glob.iglob` and :func:`~glob.glob` functions now support recursive +search in subdirectories using the ``"**"`` pattern. +(Contributed by Serhiy Storchaka in :issue:`13968`.) + + +gzip +---- + +The *mode* argument of :class:`~gzip.GzipFile` constructor now +accepts ``"x"`` to request exclusive creation. +(Contributed by Tim Heaney in :issue:`19222`.) + + +heapq +----- + +Element comparison in :func:`~heapq.merge` can now be customized by +passing a :term:`key function` in a new optional *key* keyword argument, +and a new optional *reverse* keyword argument can be used to reverse element +comparison:: + + >>> import heapq + >>> a = ['9', '777', '55555'] + >>> b = ['88', '6666'] + >>> list(heapq.merge(a, b, key=len)) + ['9', '88', '777', '6666', '55555'] + >>> list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True)) + ['55555', '6666', '777', '88', '9'] + +(Contributed by Raymond Hettinger in :issue:`13742`.) + + +http +---- + +A new :class:`HTTPStatus <http.HTTPStatus>` enum that defines a set of +HTTP status codes, reason phrases and long descriptions written in English. +(Contributed by Demian Brecht in :issue:`21793`.) + + +http.client +----------- + +:meth:`HTTPConnection.getresponse() <http.client.HTTPConnection.getresponse>` +now raises a :exc:`~http.client.RemoteDisconnected` exception when a +remote server connection is closed unexpectedly. Additionally, if a +:exc:`ConnectionError` (of which ``RemoteDisconnected`` +is a subclass) is raised, the client socket is now closed automatically, +and will reconnect on the next request:: + + import http.client + conn = http.client.HTTPConnection('www.python.org') + for retries in range(3): + try: + conn.request('GET', '/') + resp = conn.getresponse() + except http.client.RemoteDisconnected: + pass + +(Contributed by Martin Panter in :issue:`3566`.) + + +idlelib and IDLE +---------------- + +Since idlelib implements the IDLE shell and editor and is not intended for +import by other programs, it gets improvements with every release. See +:file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.4.0, +as well as changes made in future 3.5.x releases. This file is also available +from the IDLE :menuselection:`Help --> About IDLE` dialog. + + +imaplib +------- + +The :class:`~imaplib.IMAP4` class now supports :term:`context manager` protocol. +When used in a :keyword:`with` statement, the IMAP4 ``LOGOUT`` +command will be called automatically at the end of the block. +(Contributed by Tarek Ziadé and Serhiy Storchaka in :issue:`4972`.) + +The :mod:`imaplib` module now supports :rfc:`5161` (ENABLE Extension) +and :rfc:`6855` (UTF-8 Support) via the :meth:`IMAP4.enable() <imaplib.IMAP4.enable>` +method. A new :attr:`IMAP4.utf8_enabled <imaplib.IMAP4.utf8_enabled>` +attribute, tracks whether or not :rfc:`6855` support is enabled. +(Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in +:issue:`21800`.) + +The :mod:`imaplib` module now automatically encodes non-ASCII string usernames +and passwords using UTF-8, as recommended by the RFCs. (Contributed by Milan +Oberkirch in :issue:`21800`.) + + +imghdr +------ + +The :func:`~imghdr.what` function now recognizes the +`OpenEXR <http://www.openexr.com>`_ format +(contributed by Martin Vignali and Claudiu Popa in :issue:`20295`), +and the `WebP <https://en.wikipedia.org/wiki/WebP>`_ format +(contributed by Fabrice Aneche and Claudiu Popa in :issue:`20197`.) + + +importlib +--------- + +The :class:`util.LazyLoader <importlib.util.LazyLoader>` class allows for +lazy loading of modules in applications where startup time is important. +(Contributed by Brett Cannon in :issue:`17621`.) + +The :func:`abc.InspectLoader.source_to_code() <importlib.abc.InspectLoader.source_to_code>` +method is now a static method. This makes it easier to initialize a module +object with code compiled from a string by running +``exec(code, module.__dict__)``. +(Contributed by Brett Cannon in :issue:`21156`.) + +The new :func:`util.module_from_spec() <importlib.util.module_from_spec>` +function is now the preferred way to create a new module. As opposed to +creating a :class:`types.ModuleType` instance directly, this new function +will set the various import-controlled attributes based on the passed-in +spec object. (Contributed by Brett Cannon in :issue:`20383`.) + + +inspect +------- + +Both :class:`~inspect.Signature` and :class:`~inspect.Parameter` classes are +now picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` +and :issue:`20334`.) + +A new +:meth:`BoundArguments.apply_defaults() <inspect.BoundArguments.apply_defaults>` +method provides a way to set default values for missing arguments:: + + >>> def foo(a, b='ham', *args): pass + >>> ba = inspect.signature(foo).bind('spam') + >>> ba.apply_defaults() + >>> ba.arguments + OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())]) + +(Contributed by Yury Selivanov in :issue:`24190`.) + +A new class method +:meth:`Signature.from_callable() <inspect.Signature.from_callable>` makes +subclassing of :class:`~inspect.Signature` easier. (Contributed +by Yury Selivanov and Eric Snow in :issue:`17373`.) + +The :func:`~inspect.signature` function now accepts a *follow_wrapped* +optional keyword argument, which, when set to ``False``, disables automatic +following of ``__wrapped__`` links. +(Contributed by Yury Selivanov in :issue:`20691`.) + +A set of new functions to inspect +:term:`coroutine functions <coroutine function>` and +:term:`coroutine objects <coroutine>` has been added: +:func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction`, +:func:`~inspect.isawaitable`, :func:`~inspect.getcoroutinelocals`, +and :func:`~inspect.getcoroutinestate`. +(Contributed by Yury Selivanov in :issue:`24017` and :issue:`24400`.) + +:func:`~inspect.stack`, :func:`~inspect.trace`, +:func:`~inspect.getouterframes`, and :func:`~inspect.getinnerframes` +functions now return a list of named tuples. +(Contributed by Daniel Shahaf in :issue:`16808`.) + + +io +-- + +A new :meth:`BufferedIOBase.readinto1() <io.BufferedIOBase.readinto1>` +method, that uses at most one call to the underlying raw stream's +:meth:`RawIOBase.read() <io.RawIOBase.read>` or +:meth:`RawIOBase.readinto() <io.RawIOBase.readinto>` methods. +(Contributed by Nikolaus Rath in :issue:`20578`.) + + +ipaddress +--------- + +Both :class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes +now accept an ``(address, netmask)`` tuple argument, so as to easily construct +network objects from existing addresses:: + + >>> import ipaddress + >>> ipaddress.IPv4Network(('127.0.0.0', 8)) + IPv4Network('127.0.0.0/8') + >>> ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0')) + IPv4Network('127.0.0.0/8') + +(Contributed by Peter Moody and Antoine Pitrou in :issue:`16531`.) + +A new :attr:`~ipaddress.IPv4Network.reverse_pointer` attribute for +:class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes +returns the name of the reverse DNS PTR record:: + + >>> import ipaddress + >>> addr = ipaddress.IPv4Address('127.0.0.1') + >>> addr.reverse_pointer + '1.0.0.127.in-addr.arpa' + >>> addr6 = ipaddress.IPv6Address('::1') + >>> addr6.reverse_pointer + '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa' + +(Contributed by Leon Weber in :issue:`20480`.) + + +json +---- + +The :mod:`json.tool` command line interface now preserves the order of keys in +JSON objects passed in input. The new ``--sort-keys`` option can be used +to sort the keys alphabetically. (Contributed by Berker Peksag +in :issue:`21650`.) + +JSON decoder now raises :exc:`~json.JSONDecodeError` instead of +:exc:`ValueError` to provide better context information about the error. +(Contributed by Serhiy Storchaka in :issue:`19361`.) + + +linecache +--------- + +A new :func:`~linecache.lazycache` function can be used to capture information +about a non-file based module to permit getting its lines later via +:func:`~linecache.getline`. This avoids doing I/O until a line is actually +needed, without having to carry the module globals around indefinitely. +(Contributed by Robert Collins in :issue:`17911`.) + + +locale +------ + +A new :func:`~locale.delocalize` function can be used to convert a string into +a normalized number string, taking the ``LC_NUMERIC`` settings into account:: + + >>> import locale + >>> locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8') + 'de_DE.UTF-8' + >>> locale.delocalize('1.234,56') + '1234.56' + >>> locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8') + 'en_US.UTF-8' + >>> locale.delocalize('1,234.56') + '1234.56' + +(Contributed by Cédric Krier in :issue:`13918`.) + + +logging +------- + +All logging methods (:class:`~logging.Logger` :meth:`~logging.Logger.log`, +:meth:`~logging.Logger.exception`, :meth:`~logging.Logger.critical`, +:meth:`~logging.Logger.debug`, etc.), now accept exception instances +as an *exc_info* argument, in addition to boolean values and exception +tuples:: + + >>> import logging + >>> try: + ... 1/0 + ... except ZeroDivisionError as ex: + ... logging.error('exception', exc_info=ex) + ERROR:root:exception + +(Contributed by Yury Selivanov in :issue:`20537`.) + +The :class:`handlers.HTTPHandler <logging.handlers.HTTPHandler>` class now +accepts an optional :class:`ssl.SSLContext` instance to configure SSL +settings used in an HTTP connection. +(Contributed by Alex Gaynor in :issue:`22788`.) + +The :class:`handlers.QueueListener <logging.handlers.QueueListener>` class now +takes a *respect_handler_level* keyword argument which, if set to ``True``, +will pass messages to handlers taking handler levels into account. +(Contributed by Vinay Sajip.) + + +lzma +---- + +The :meth:`LZMADecompressor.decompress() <lzma.LZMADecompressor.decompress>` +method now accepts an optional *max_length* argument to limit the maximum +size of decompressed data. +(Contributed by Martin Panter in :issue:`15955`.) + + +math +---- + +Two new constants have been added to the :mod:`math` module: :data:`~math.inf` +and :data:`~math.nan`. (Contributed by Mark Dickinson in :issue:`23185`.) + +A new function :func:`~math.isclose` provides a way to test for approximate +equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + +A new :func:`~math.gcd` function has been added. The :func:`fractions.gcd` +function is now deprecated. (Contributed by Mark Dickinson and Serhiy +Storchaka in :issue:`22486`.) + + +multiprocessing +--------------- + +:func:`sharedctypes.synchronized() <multiprocessing.sharedctypes.synchronized>` +objects now support the :term:`context manager` protocol. +(Contributed by Charles-François Natali in :issue:`21565`.) + + +operator +-------- + +:func:`~operator.attrgetter`, :func:`~operator.itemgetter`, +and :func:`~operator.methodcaller` objects now support pickling. +(Contributed by Josh Rosenberg and Serhiy Storchaka in :issue:`22955`.) + +New :func:`~operator.matmul` and :func:`~operator.imatmul` functions +to perform matrix multiplication. +(Contributed by Benjamin Peterson in :issue:`21176`.) + + +os +-- + +The new :func:`~os.scandir` function returning an iterator of +:class:`~os.DirEntry` objects has been added. If possible, :func:`~os.scandir` +extracts file attributes while scanning a directory, removing the need to +perform subsequent system calls to determine file type or attributes, which may +significantly improve performance. (Contributed by Ben Hoyt with the help +of Victor Stinner in :issue:`22524`.) + +On Windows, a new +:attr:`stat_result.st_file_attributes <os.stat_result.st_file_attributes>` +attribute is now available. It corresponds to the ``dwFileAttributes`` member +of the ``BY_HANDLE_FILE_INFORMATION`` structure returned by +``GetFileInformationByHandle()``. (Contributed by Ben Hoyt in :issue:`21719`.) + +The :func:`~os.urandom` function now uses ``getrandom()`` syscall on Linux 3.17 +or newer, and ``getentropy()`` on OpenBSD 5.6 and newer, removing the need to +use ``/dev/urandom`` and avoiding failures due to potential file descriptor +exhaustion. (Contributed by Victor Stinner in :issue:`22181`.) + +New :func:`~os.get_blocking` and :func:`~os.set_blocking` functions allow to +get and set a file descriptor blocking mode (:data:`~os.O_NONBLOCK`.) +(Contributed by Victor Stinner in :issue:`22054`.) + +The :func:`~os.truncate` and :func:`~os.ftruncate` functions are now supported +on Windows. (Contributed by Steve Dower in :issue:`23668`.) + +There is a new :func:`os.path.commonpath` function returning the longest +common sub-path of each passed pathname. Unlike the +:func:`os.path.commonprefix` function, it always returns a valid +path:: + + >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) + '/usr/l' + + >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) + '/usr' + +(Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) + + +pathlib +------- + +The new :meth:`Path.samefile() <pathlib.Path.samefile>` method can be used +to check whether the path points to the same file as other path, which can be +either an another :class:`~pathlib.Path` object, or a string:: + + >>> import pathlib + >>> p1 = pathlib.Path('/etc/hosts') + >>> p2 = pathlib.Path('/etc/../etc/hosts') + >>> p1.samefile(p2) + True + +(Contributed by Vajrasky Kok and Antoine Pitrou in :issue:`19775`.) + +The :meth:`Path.mkdir() <pathlib.Path.mkdir>` method how accepts a new optional +*exist_ok* argument to match ``mkdir -p`` and :func:`os.makrdirs` +functionality. (Contributed by Berker Peksag in :issue:`21539`.) + +There is a new :meth:`Path.expanduser() <pathlib.Path.expanduser>` method to +expand ``~`` and ``~user`` prefixes. (Contributed by Serhiy Storchaka and +Claudiu Popa in :issue:`19776`.) + +A new :meth:`Path.home() <pathlib.Path.home>` class method can be used to get +an instance of :class:`~pathlib.Path` object representing the user’s home +directory. +(Contributed by Victor Salgado and Mayank Tripathi in :issue:`19777`.) + +New :meth:`Path.write_text() <pathlib.Path.write_text>`, +:meth:`Path.read_text() <pathlib.Path.read_text>`, +:meth:`Path.write_bytes() <pathlib.Path.write_bytes>`, +:meth:`Path.read_bytes() <pathlib.Path.read_bytes>` methods to simplify +read/write operations on files. + +The following code snippet will create or rewrite existing file +``~/spam42``:: + + >>> import pathlib + >>> p = pathlib.Path('~/spam42') + >>> p.expanduser().write_text('ham') + 3 + +(Contributed by Christopher Welborn in :issue:`20218`.) + + +pickle +------ + +Nested objects, such as unbound methods or nested classes, can now be pickled +using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. +Protocol version 4 already supports these cases. (Contributed by Serhiy +Storchaka in :issue:`23611`.) + + +poplib +------ + +A new :meth:`POP3.utf8() <poplib.POP3.utf8>` command enables :rfc:`6856` +(Internationalized Email) support, if a POP server supports it. +(Contributed by Milan OberKirch in :issue:`21804`.) + + +re +-- + +References and conditional references to groups with fixed length are now +allowed in lookbehind assertions:: + + >>> import re + >>> pat = re.compile(r'(a|b).(?<=\1)c') + >>> pat.match('aac') + <_sre.SRE_Match object; span=(0, 3), match='aac'> + >>> pat.match('bbc') + <_sre.SRE_Match object; span=(0, 3), match='bbc'> + +(Contributed by Serhiy Storchaka in :issue:`9179`.) + +The number of capturing groups in regular expression is no longer limited by +100. (Contributed by Serhiy Storchaka in :issue:`22437`.) + +The :func:`~re.sub` and :func:`~re.subn` functions now replace unmatched +groups with empty strings instead of raising an exception. +(Contributed by Serhiy Storchaka in :issue:`1519638`.) + +The :class:`re.error` exceptions have new attributes: +:attr:`~re.error.msg`, :attr:`~re.error.pattern`, +:attr:`~re.error.pos`, :attr:`~re.error.lineno`, +and :attr:`~re.error.colno` that provide better context +information about the error:: + + >>> re.compile(""" + ... (?x) + ... .++ + ... """) + Traceback (most recent call last): + ... + sre_constants.error: multiple repeat at position 16 (line 3, column 7) + +(Contributed by Serhiy Storchaka in :issue:`22578`.) + + +readline +-------- + +A new :func:`~readline.append_history_file` function can be used to append +the specified number of trailing elements in history to the given file. +(Contributed by Bruno Cauet in :issue:`22940`.) + + +selectors +--------- + +The new :class:`~selectors.DevpollSelector` supports efficient +``/dev/poll`` polling on Solaris. +(Contributed by Giampaolo Rodola' in :issue:`18931`.) + + +shutil +------ + +The :func:`~shutil.move` function now accepts a *copy_function* argument, +allowing, for example, the :func:`~shutil.copy` function to be used instead of +the default :func:`~shutil.copy2` if there is a need to ignore file metadata +when moving. +(Contributed by Claudiu Popa in :issue:`19840`.) + +The :func:`~shutil.make_archive` function now supports the *xztar* format. +(Contributed by Serhiy Storchaka in :issue:`5411`.) + + +signal +------ + +On Windows, the :func:`~signal.set_wakeup_fd` function now also supports +socket handles. (Contributed by Victor Stinner in :issue:`22018`.) + +Various ``SIG*`` constants in the :mod:`signal` module have been converted into +:mod:`Enums <enum>`. This allows meaningful names to be printed +during debugging, instead of integer "magic numbers". +(Contributed by Giampaolo Rodola' in :issue:`21076`.) + + +smtpd +----- + +Both :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now +accept a *decode_data* keyword argument to determine if the ``DATA`` portion of +the SMTP transaction is decoded using the ``"utf-8"`` codec or is instead +provided to the +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +method as a byte string. The default is ``True`` for backward compatibility +reasons, but will change to ``False`` in Python 3.6. If *decode_data* is set +to ``False``, the ``process_message`` method must be prepared to accept keyword +arguments. +(Contributed by Maciej Szulik in :issue:`19662`.) + +The :class:`~smtpd.SMTPServer` class now advertises the ``8BITMIME`` extension +(:rfc:`6152`) if *decode_data* has been set ``True``. If the client +specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +via the *mail_options* keyword. +(Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.) + +The :class:`~smtpd.SMTPServer` class now also supports the ``SMTPUTF8`` +extension (:rfc:`6531`: Internationalized Email). If the client specified +``SMTPUTF8 BODY=8BITMIME`` on the ``MAIL`` command, they are passed to +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +via the *mail_options* keyword. It is the responsibility of the +``process_message`` method to correctly handle the ``SMTPUTF8`` data. +(Contributed by Milan Oberkirch in :issue:`21725`.) + +It is now possible to provide, directly or via name resolution, IPv6 +addresses in the :class:`~smtpd.SMTPServer` constructor, and have it +successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) + + +smtplib +------- + +A new :meth:`SMTP.auth() <smtplib.SMTP.auth>` method provides a convenient way to +implement custom authentication mechanisms. (Contributed by Milan +Oberkirch in :issue:`15014`.) + +The :meth:`SMTP.set_debuglevel() <smtplib.SMTP.set_debuglevel>` method now +accepts an additional debuglevel (2), which enables timestamps in debug +messages. (Contributed by Gavin Chappell and Maciej Szulik in :issue:`16914`.) + +Both :meth:`SMTP.sendmail() <smtplib.SMTP.sendmail>` and +:meth:`SMTP.send_message() <smtplib.SMTP.send_message>` methods now +support support :rfc:`6531` (SMTPUTF8). +(Contributed by Milan Oberkirch and R. David Murray in :issue:`22027`.) + + +sndhdr +------ + +:func:`~sndhdr.what` and :func:`~sndhdr.whathdr` functions now return +a :func:`~collections.namedtuple`. (Contributed by Claudiu Popa in +:issue:`18615`.) + + +socket +------ + +Functions with timeouts now use a monotonic clock, instead of a system clock. +(Contributed by Victor Stinner in :issue:`22043`.) + +A new :meth:`socket.sendfile() <socket.socket.sendfile>` method allows to +send a file over a socket by using the high-performance :func:`os.sendfile` +function on UNIX resulting in uploads being from 2 to 3 times faster than when +using plain :meth:`socket.send() <socket.socket.send>`. +(Contributed by Giampaolo Rodola' in :issue:`17552`.) + +The :meth:`socket.sendall() <socket.socket.sendall>` method no longer resets the +socket timeout every time bytes are received or sent. The socket timeout is +now the maximum total duration to send all data. +(Contributed by Victor Stinner in :issue:`23853`.) + +The *backlog* argument of the :meth:`socket.listen() <socket.socket.listen>` +method is now optional. By default it is set to +:data:`SOMAXCONN <socket.SOMAXCONN>` or to ``128`` whichever is less. +(Contributed by Charles-François Natali in :issue:`21455`.) + + +ssl +--- + +.. _whatsnew-sslmemorybio: + +Memory BIO Support +~~~~~~~~~~~~~~~~~~ + +(Contributed by Geert Jansen in :issue:`21965`.) + +The new :class:`~ssl.SSLObject` class has been added to provide SSL protocol +support for cases when the network I/O capabilities of :class:`~ssl.SSLSocket` +are not necessary or suboptimal. ``SSLObject`` represents +an SSL protocol instance, but does not implement any network I/O methods, and +instead provides a memory buffer interface. The new :class:`~ssl.MemoryBIO` +class can be used to pass data between Python and an SSL protocol instance. + +The memory BIO SSL support is primarily intended to be used in frameworks +implementing asynchronous I/O for which :class:`~ssl.SSLSocket`'s readiness +model ("select/poll") is inefficient. + +A new :meth:`SSLContext.wrap_bio() <ssl.SSLContext.wrap_bio>` method can be used +to create a new ``SSLObject`` instance. + + +Application-Layer Protocol Negotiation Support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +(Contributed by Benjamin Peterson in :issue:`20188`.) + +Where OpenSSL support is present, :mod:`ssl` module now implements +*Application-Layer Protocol Negotiation* TLS extension as described +in :rfc:`7301`. + +The new :meth:`SSLContext.set_alpn_protocols() <ssl.SSLContext.set_alpn_protocols>` +can be used to specify which protocols a socket should advertise during +the TLS handshake. + +The new +:meth:`SSLSocket.selected_alpn_protocol() <ssl.SSLSocket.selected_alpn_protocol>` +returns the protocol that was selected during the TLS handshake. +:data:`~ssl.HAS_ALPN` flag indicates whether APLN support is present. + + +Other Changes +~~~~~~~~~~~~~ + +There is a new :meth:`SSLSocket.version() <ssl.SSLSocket.version>` method to +query the actual protocol version in use. +(Contributed by Antoine Pitrou in :issue:`20421`.) + +The :class:`~ssl.SSLSocket` class now implements +a :meth:`SSLSocket.sendfile() <ssl.SSLSocket.sendfile>` method. +(Contributed by Giampaolo Rodola' in :issue:`17552`.) + +The :meth:`SSLSocket.send() <ssl.SSLSocket.send>` method now raises either +:exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` exception on a +non-blocking socket if the operation would block. Previously, it would return +``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) + +The :func:`~ssl.cert_time_to_seconds` function now interprets the input time +as UTC and not as local time, per :rfc:`5280`. Additionally, the return +value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) + +New :meth:`SSLObject.shared_ciphers() <ssl.SSLObject.shared_ciphers>` and +:meth:`SSLSocket.shared_ciphers() <ssl.SSLSocket.shared_ciphers>` methods return +the list of ciphers sent by the client during the handshake. +(Contributed by Benjamin Peterson in :issue:`23186`.) + +The :meth:`SSLSocket.do_handshake() <ssl.SSLSocket.do_handshake>`, +:meth:`SSLSocket.read() <ssl.SSLSocket.read>`, +:meth:`SSLSocket.shutdown() <ssl.SSLSocket.shutdown>`, and +:meth:`SSLSocket.write() <ssl.SSLSocket.write>` methods of :class:`~ssl.SSLSocket` +class no longer reset the socket timeout every time bytes are received or sent. +The socket timeout is now the maximum total duration of the method. +(Contributed by Victor Stinner in :issue:`23853`.) + +The :func:`~ssl.match_hostname` function now supports matching of IP addresses. +(Contributed by Antoine Pitrou in :issue:`23239`.) + + +sqlite3 +------- + +The :class:`~sqlite3.Row` class now fully supports sequence protocol, +in particular :func:`reversed` iteration and slice indexing. +(Contributed by Claudiu Popa in :issue:`10203`; by Lucas Sinclair, +Jessica McKellar, and Serhiy Storchaka in :issue:`13583`.) + + +.. _whatsnew-subprocess: + +subprocess +---------- + +The new :func:`~subprocess.run` function has been added. +It runs the specified command and and returns a +:class:`~subprocess.CompletedProcess` object, which describes a finished +process. The new API is more consistent and is the recommended approach +to invoking subprocesses in Python code that does not need to maintain +compatibility with earlier Python versions. +(Contributed by Thomas Kluyver in :issue:`23342`.) + +Examples:: + + >>> subprocess.run(["ls", "-l"]) # doesn't capture output + CompletedProcess(args=['ls', '-l'], returncode=0) + + >>> subprocess.run("exit 1", shell=True, check=True) + Traceback (most recent call last): + ... + subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + + >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) + CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, + stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') + + +sys +--- + +A new :func:`~sys.set_coroutine_wrapper` function allows setting a global +hook that will be called whenever a :term:`coroutine object <coroutine>` +is created by an :keyword:`async def` function. A corresponding +:func:`~sys.get_coroutine_wrapper` can be used to obtain a currently set +wrapper. Both functions are :term:`provisional <provisional api>`, +and are intended for debugging purposes only. (Contributed by Yury Selivanov +in :issue:`24017`.) + +A new :func:`~sys.is_finalizing` function can be used to check if the Python +interpreter is :term:`shutting down <interpreter shutdown>`. +(Contributed by Antoine Pitrou in :issue:`22696`.) + + +sysconfig +--------- + +The name of the user scripts directory on Windows now includes the first +two components of Python version. (Contributed by Paul Moore +in :issue:`23437`.) + + +tarfile +------- + +The *mode* argument of the :func:`~tarfile.open` function now accepts ``"x"`` +to request exclusive creation. (Contributed by Berker Peksag in :issue:`21717`.) + +:meth:`TarFile.extractall() <tarfile.TarFile.extractall>` and +:meth:`TarFile.extract() <tarfile.TarFile.extract>` methods now take a keyword +argument *numeric_only*. If set to ``True``, the extracted files and +directories will be owned by the numeric ``uid`` and ``gid`` from the tarfile. +If set to ``False`` (the default, and the behavior in versions prior to 3.5), +they will be owned by the named user and group in the tarfile. +(Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) + +The :meth:`TarFile.list() <tarfile.TarFile.list>` now accepts an optional +*members* keyword argument that can be set to a subset of the list returned +by :meth:`TarFile.getmembers() <tarfile.TarFile.getmembers>`. +(Contributed by Serhiy Storchaka in :issue:`21549`.) + + +threading +--------- + +Both :meth:`Lock.acquire() <threading.Lock.acquire>` and +:meth:`RLock.acquire() <threading.RLock.acquire>` methods +now use a monotonic clock for timeout management. +(Contributed by Victor Stinner in :issue:`22043`.) + + +time +---- + +The :func:`~time.monotonic` function is now always available. +(Contributed by Victor Stinner in :issue:`22043`.) + + +timeit +------ + +A new command line option ``-u`` or :samp:`--unit={U}` can be used to specify the time +unit for the timer output. Supported options are ``usec``, ``msec``, +or ``sec``. (Contributed by Julian Gindi in :issue:`18983`.) + +The :func:`~timeit.timeit` function has a new *globals* parameter for +specifying the namespace in which the code will be running. +(Contributed by Ben Roberts in :issue:`2527`.) + + +tkinter +------- + +The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment +on Windows has been replaced by a private function in the :mod:`_tkinter` +module which makes no permanent changes to environment variables. +(Contributed by Zachary Ware in :issue:`20035`.) + + +.. _whatsnew-traceback: + +traceback +--------- + +New :func:`~traceback.walk_stack` and :func:`~traceback.walk_tb` +functions to conveniently traverse frame and traceback objects. +(Contributed by Robert Collins in :issue:`17911`.) + +New lightweight classes: :class:`~traceback.TracebackException`, +:class:`~traceback.StackSummary`, and :class:`~traceback.FrameSummary`. +(Contributed by Robert Collins in :issue:`17911`.) + +Both :func:`~traceback.print_tb` and :func:`~traceback.print_stack` functions +now support negative values for the *limit* argument. +(Contributed by Dmitry Kazakov in :issue:`22619`.) + + +types +----- + +A new :func:`~types.coroutine` function to transform +:term:`generator <generator iterator>` and +:class:`generator-like <collections.abc.Generator>` objects into +:term:`awaitables <awaitable>`. +(Contributed by Yury Selivanov in :issue:`24017`.) + +A new :class:`~types.CoroutineType` is the type of :term:`coroutine` objects +created by :keyword:`async def` functions. +(Contributed by Yury Selivanov in :issue:`24400`.) + + +unicodedata +----------- + +The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 +<http://unicode.org/versions/Unicode8.0.0/>`_. + + +unittest +-------- + +The :meth:`TestLoader.loadTestsFromModule() <unittest.TestLoader.loadTestsFromModule>` +method now accepts a keyword-only argument *pattern* which is passed to +``load_tests`` as the third argument. Found packages are now checked for +``load_tests`` regardless of whether their path matches *pattern*, because it +is impossible for a package name to match the default pattern. +(Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.) + +Unittest discovery errors now are exposed in +:data:`TestLoader.errors <unittest.TestLoader.errors>` attribute of the +:class:`~unittest.TestLoader` instance. +(Contributed by Robert Collins in :issue:`19746`.) + +A new command line option ``--locals`` to show local variables in +tracebacks. (Contributed by Robert Collins in :issue:`22936`.) + + +unittest.mock +------------- + +The :class:`~unittest.mock.Mock` class has the following improvements: + +* Class constructor has a new *unsafe* parameter, which causes mock + objects to raise :exc:`AttributeError` on attribute names starting + with ``"assert"``. + (Contributed by Kushal Das in :issue:`21238`.) + +* A new :meth:`Mock.assert_not_called() <unittest.mock.Mock.assert_not_called>` + method to check if the mock object was called. + (Contributed by Kushal Das in :issue:`21262`.) + +The :class:`~unittest.mock.MagicMock` class now supports :meth:`__truediv__`, +:meth:`__divmod__` and :meth:`__matmul__` operators. +(Contributed by Johannes Baiter in :issue:`20968`, and Håkan Lövdahl +in :issue:`23581` and :issue:`23568`.) + +It is no longer necessary to explicitly pass ``create=True`` to the +:func:`~unittest.mock.patch` function when patching builtin names. +(Contributed by Kushal Das in :issue:`17660`.) + + +urllib +------ + +A new +:class:`request.HTTPPasswordMgrWithPriorAuth <urllib.request.HTTPPasswordMgrWithPriorAuth>` +class allows HTTP Basic Authentication credentials to be managed so as to +eliminate unnecessary ``401`` response handling, or to unconditionally send +credentials on the first request in order to communicate with servers that +return a ``404`` response instead of a ``401`` if the ``Authorization`` header +is not sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in +:issue:`7159`.) + +A new *quote_via* argument for the +:func:`parse.urlencode() <urllib.parse.urlencode>` +function provides a way to control the encoding of query parts if needed. +(Contributed by Samwyse and Arnon Yaari in :issue:`13866`.) + +The :func:`request.urlopen() <urllib.request.urlopen>` function accepts an +:class:`ssl.SSLContext` object as a *context* argument, which will be used for +the HTTPS connection. (Contributed by Alex Gaynor in :issue:`22366`.) + +The :func:`parse.urljoin() <urllib.parse.urljoin>` was updated to use the +:rfc:`3986` semantics for the resolution of relative URLs, rather than +:rfc:`1808` and :rfc:`2396`. +(Contributed by Demian Brecht and Senthil Kumaran in :issue:`22118`.) + + +wsgiref +------- + +The *headers* argument of the :class:`headers.Headers <wsgiref.headers.Headers>` +class constructor is now optional. +(Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) + + +xmlrpc +------ + +The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` class now supports +:term:`context manager` protocol. +(Contributed by Claudiu Popa in :issue:`20627`.) + +:class:`client.ServerProxy <xmlrpc.client.ServerProxy>` constructor now accepts +an optional :class:`ssl.SSLContext` instance. +(Contributed by Alex Gaynor in :issue:`22960`.) + + +xml.sax +------- + +SAX parsers now support a character stream of the +:class:`xmlreader.InputSource <xml.sax.xmlreader.InputSource>` object. +(Contributed by Serhiy Storchaka in :issue:`2175`.) + +:func:`~xml.sax.parseString` now accepts a :class:`str` instance. +(Contributed by Serhiy Storchaka in :issue:`10590`.) + + +zipfile +------- + +ZIP output can now be written to unseekable streams. +(Contributed by Serhiy Storchaka in :issue:`23252`.) + +The *mode* argument of :meth:`ZipFile.open() <zipfile.ZipFile.open>` method now +accepts ``"x"`` to request exclusive creation. +(Contributed by Serhiy Storchaka in :issue:`21717`.) + + +Other module-level changes +========================== + +Many functions in :mod:`mmap`, :mod:`ossaudiodev`, :mod:`socket`, +:mod:`ssl`, and :mod:`codecs` modules now accept writable +:term:`bytes-like objects <bytes-like object>`. +(Contributed by Serhiy Storchaka in :issue:`23001`.) + + +Optimizations +============= + +The :func:`os.walk` function has been sped up by 3 to 5 times on POSIX systems, +and by 7 to 20 times on Windows. This was done using the new :func:`os.scandir` +function, which exposes file information from the underlying ``readdir`` or +``FindFirstFile``/``FindNextFile`` system calls. (Contributed by +Ben Hoyt with help from Victor Stinner in :issue:`23605`.) + +Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less +memory for large objects. ``calloc()`` is used instead of ``malloc()`` to +allocate memory for these objects. +(Contributed by Victor Stinner in :issue:`21233`.) + +Some operations on :mod:`ipaddress` :class:`~ipaddress.IPv4Network` and +:class:`~ipaddress.IPv6Network` have been massively sped up, such as +:meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, +:func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. +The speed up can range from 3 to 15 times. +(Contributed by Antoine Pitrou, Michel Albert, and Markus in +:issue:`21486`, :issue:`21487`, :issue:`20826`, :issue:`23266`.) + +Pickling of :mod:`ipaddress` objects was optimized to produce significantly +smaller output. (Contributed by Serhiy Storchaka in :issue:`23133`.) + +Many operations on :class:`io.BytesIO` are now 50% to 100% faster. +(Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in +:issue:`22003`.) + +The :func:`marshal.dumps` function is now faster: 65-85% with versions 3 +and 4, 20-25% with versions 0 to 2 on typical data, and up to 5 times in +best cases. +(Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) + +The UTF-32 encoder is now 3 to 7 times faster. +(Contributed by Serhiy Storchaka in :issue:`15027`.) + +Regular expressions are now parsed up to 10% faster. +(Contributed by Serhiy Storchaka in :issue:`19380`.) + +The :func:`json.dumps` function was optimized to run with +``ensure_ascii=False`` as fast as with ``ensure_ascii=True``. +(Contributed by Naoki Inada in :issue:`23206`.) + +The :c:func:`PyObject_IsInstance` and :c:func:`PyObject_IsSubclass` +functions have been sped up in the common case that the second argument +has :class:`type` as its metaclass. +(Contributed Georg Brandl by in :issue:`22540`.) + +Method caching was slightly improved, yielding up to 5% performance +improvement in some benchmarks. +(Contributed by Antoine Pitrou in :issue:`22847`.) + +Objects from :mod:`random` module now use two times less memory on 64-bit +builds. (Contributed by Serhiy Storchaka in :issue:`23488`.) + +The :func:`property` getter calls are up to 25% faster. +(Contributed by Joe Jevnik in :issue:`23910`.) + +Instantiation of :class:`fractions.Fraction` is now up to 30% faster. +(Contributed by Stefan Behnel in :issue:`22464`.) + +String methods :meth:`~str.find`, :meth:`~str.rfind`, :meth:`~str.split`, +:meth:`~str.partition` and :keyword:`in` string operator are now significantly +faster for searching 1-character substrings. +(Contributed by Serhiy Storchaka in :issue:`23573`.) + + +Build and C API Changes +======================= + +New ``calloc`` functions were added: + +* :c:func:`PyMem_RawCalloc`, +* :c:func:`PyMem_Calloc`, +* :c:func:`PyObject_Calloc`, +* :c:func:`_PyObject_GC_Calloc`. + +(Contributed by Victor Stinner in :issue:`21233`.) + +New encoding/decoding helper functions: + +* :c:func:`Py_DecodeLocale` (replaced ``_Py_char2wchar()``), +* :c:func:`Py_EncodeLocale` (replaced ``_Py_wchar2char()``). + +(Contributed by Victor Stinner in :issue:`18395`.) + +A new :c:func:`PyCodec_NameReplaceErrors` function to replace the unicode +encode error with ``\N{...}`` escapes. +(Contributed by Serhiy Storchaka in :issue:`19676`.) + +A new :c:func:`PyErr_FormatV` function similar to :c:func:`PyErr_Format`, +but accepts a ``va_list`` argument. +(Contributed by Antoine Pitrou in :issue:`18711`.) + +A new :c:data:`PyExc_RecursionError` exception. +(Contributed by Georg Brandl in :issue:`19235`.) + +New :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2`, +and :c:func:`PyModule_ExecDef` functions introduced by :pep:`489` -- +multi-phase extension module initialization. +(Contributed by Petr Viktorin in :issue:`24268`.) + +New :c:func:`PyNumber_MatrixMultiply` and +:c:func:`PyNumber_InPlaceMatrixMultiply` functions to perform matrix +multiplication. +(Contributed by Benjamin Peterson in :issue:`21176`. See also :pep:`465` +for details.) + +The :c:member:`PyTypeObject.tp_finalize` slot is now part of stable ABI. + +Windows builds now require Microsoft Visual C++ 14.0, which +is available as part of `Visual Studio 2015 <http://www.visualstudio.com>`_. + +Extension modules now include platform information tag in their filename on +some platforms (the tag is optional, and CPython will import extensions without +it; although if the tag is present and mismatched, the extension won't be +loaded): + +* On Linux, extension module filenames end with + ``.cpython-<major><minor>m-<architecture>-<os>.pyd``: + + * ``<major>`` is the major number of the Python version; + for Python 3.5 this is ``3``. + + * ``<minor>`` is the minor number of the Python version; + for Python 3.5 this is ``5``. + + * ``<architecture>`` is the hardware architecture the extension module + was built to run on. It's most commonly either ``i386`` for 32-bit Intel + platforms or ``x86_64`` for 64-bit Intel (and AMD) platforms. + + * ``<os>`` is always ``linux-gnu``, except for extensions built to + talk to the 32-bit ABI on 64-bit platforms, in which case it is + ``linux-gnu32`` (and ``<architecture>`` will be ``x86_64``). + +* On Windows, extension module filenames end with + ``<debug>.cp<major><minor>-<platform>.pyd``: + + * ``<major>`` is the major number of the Python version; + for Python 3.5 this is ``3``. + + * ``<minor>`` is the minor number of the Python version; + for Python 3.5 this is ``5``. + + * ``<platform>`` is the platform the extension module was built for, + either ``win32`` for Win32, ``win_amd64`` for Win64, ``win_ia64`` for + Windows Itanium 64, and ``win_arm`` for Windows on ARM. + + * If built in debug mode, ``<debug>`` will be ``_d``, + otherwise it will be blank. + +* On OS X platforms, extension module filenames now end with ``-darwin.so``. + +* On all other platforms, extension module filenames are the same as they were + with Python 3.4. + + +Deprecated +========== + +New Keywords +------------ + +``async`` and ``await`` are not recommended to be used as variable, class, +function or module names. Introduced by :pep:`492` in Python 3.5, they will +become proper keywords in Python 3.7. + + +Deprecated Python Behavior +-------------------------- + +Raising :exc:`StopIteration` exception inside a generator will now generate a silent +:exc:`PendingDeprecationWarning`, which will become a non-silent deprecation +warning in Python 3.6 and will trigger a :exc:`RuntimeError` in Python 3.7. +See :ref:`PEP 479: Change StopIteration handling inside generators <whatsnew-pep-479>` +for details. + + +Unsupported Operating Systems +----------------------------- + +Windows XP is no longer supported by Microsoft, thus, per :PEP:`11`, CPython +3.5 is no longer officially supported on this OS. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +The :mod:`formatter` module has now graduated to full deprecation and is still +slated for removal in Python 3.6. + +The :func:`asyncio.async` function is deprecated in favor of +:func:`~asyncio.ensure_future`. + +The :mod:`smtpd` module has in the past always decoded the DATA portion of +email messages using the ``utf-8`` codec. This can now be controlled by the +new *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is +``True``, but this default is deprecated. Specify the *decode_data* keyword +with an appropriate value to avoid the deprecation warning. + +Directly assigning values to the :attr:`~http.cookies.Morsel.key`, +:attr:`~http.cookies.Morsel.value` and +:attr:`~http.cookies.Morsel.coded_value` of :class:`http.cookies.Morsel` +objects is deprecated. Use the :meth:`~http.cookies.Morsel.set` method +instead. In addition, the undocumented *LegalChars* parameter of +:meth:`~http.cookies.Morsel.set` is deprecated, and is now ignored. + +Passing a format string as keyword argument *format_string* to the +:meth:`~string.Formatter.format` method of the :class:`string.Formatter` +class has been deprecated. +(Contributed by Serhiy Storchaka in :issue:`23671`.) + +The :func:`platform.dist` and :func:`platform.linux_distribution` functions +are now deprecated and will be removed in Python 3.7. Linux distributions use +too many different ways of describing themselves, so the functionality is +left to a package. +(Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.) + +The previously undocumented ``from_function`` and ``from_builtin`` methods of +:class:`inspect.Signature` are deprecated. Use the new +:meth:`Signature.from_callable() <inspect.Signature.from_callable>` +method instead. (Contributed by Yury Selivanov in :issue:`24248`.) + +The :func:`inspect.getargspec` function is deprecated and scheduled to be +removed in Python 3.6. (See :issue:`20438` for details.) + +The :mod:`inspect` :func:`~inspect.getfullargspec`, +:func:`~inspect.getargvalues`, :func:`~inspect.getcallargs`, +:func:`~inspect.getargvalues`, :func:`~inspect.formatargspec`, and +:func:`~inspect.formatargvalues` functions are deprecated in favor of +:func:`inspect.signature` API. +(Contributed by Yury Selivanov in :issue:`20438`.) + +Use of :const:`re.LOCALE` flag with str patterns or :const:`re.ASCII` is now +deprecated. (Contributed by Serhiy Storchaka in :issue:`22407`.) + +Use of unrecognized special sequences consisting of ``'\'`` and an ASCII letter +in regular expression patterns and replacement patterns now raises a +deprecation warning and will be forbidden in Python 3.6. +(Contributed by Serhiy Storchaka in :issue:`23622`.) + +The undocumented and unofficial *use_load_tests* default argument of the +:meth:`unittest.TestLoader.loadTestsFromModule` method now is +deprecated and ignored. +(Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.) + + +Removed +======= + +API and Feature Removals +------------------------ + +The following obsolete and previously deprecated APIs and features have been +removed: + +* The ``__version__`` attribute has been dropped from the email package. The + email code hasn't been shipped separately from the stdlib for a long time, + and the ``__version__`` string was not updated in the last few releases. + +* The internal ``Netrc`` class in the :mod:`ftplib` module was deprecated in + 3.4, and has now been removed. + (Contributed by Matt Chaput in :issue:`6623`.) + +* The concept of ``.pyo`` files has been removed. + +* The JoinableQueue class in the provisional :mod:`asyncio` module was + deprecated in 3.4.4 and is now removed. + (Contributed by A. Jesse Jiryu Davis in :issue:`23464`.) + + +Porting to Python 3.5 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + + +Changes in Python behavior +-------------------------- + +* Due to an oversight, earlier Python versions erroneously accepted the + following syntax:: + + f(1 for x in [1], *args) + f(1 for x in [1], **kwargs) + + Python 3.5 now correctly raises a :exc:`SyntaxError`, as generator + expressions must be put in parentheses if not a sole argument to a function. + + +Changes in the Python API +------------------------- + +* :pep:`475`: System calls are now retried when interrupted by a signal instead + of raising :exc:`InterruptedError` if the Python signal handler does not + raise an exception. + +* Before Python 3.5, a :class:`datetime.time` object was considered to be false + if it represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. + +* The :meth:`ssl.SSLSocket.send()` method now raises either + :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` + on a non-blocking socket if the operation would block. Previously, + it would return ``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) + +* The ``__name__`` attribute of generator is now set from the function name, + instead of being set from the code name. Use ``gen.gi_code.co_name`` to + retrieve the code name. Generators also have a new ``__qualname__`` + attribute, the qualified name, which is now used for the representation + of a generator (``repr(gen)``). + (Contributed by Victor Stinner in :issue:`21205`.) + +* The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`, + :meth:`HTMLParser.error`, and the :exc:`HTMLParserError` exception have been + removed. (Contributed by Ezio Melotti in :issue:`15114`.) + The *convert_charrefs* argument of :class:`~html.parser.HTMLParser` is + now ``True`` by default. (Contributed by Berker Peksag in :issue:`21047`.) + +* Although it is not formally part of the API, it is worth noting for porting + purposes (ie: fixing tests) that error messages that were previously of the + form "'sometype' does not support the buffer protocol" are now of the form "a + :term:`bytes-like object` is required, not 'sometype'". + (Contributed by Ezio Melotti in :issue:`16518`.) + +* If the current directory is set to a directory that no longer exists then + :exc:`FileNotFoundError` will no longer be raised and instead + :meth:`~importlib.machinery.FileFinder.find_spec` will return ``None`` + **without** caching ``None`` in :data:`sys.path_importer_cache` which is + different than the typical case (:issue:`22834`). + +* HTTP status code and messages from :mod:`http.client` and :mod:`http.server` + were refactored into a common :class:`~http.HTTPStatus` enum. The values in + :mod:`http.client` and :mod:`http.server` remain available for backwards + compatibility. (Contributed by Demian Brecht in :issue:`21793`.) + +* When an import loader defines :meth:`importlib.machinery.Loader.exec_module` + it is now expected to also define + :meth:`~importlib.machinery.Loader.create_module` (raises a + :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader + inherits from :class:`importlib.abc.Loader` then there is nothing to do, else + simply define :meth:`~importlib.machinery.Loader.create_module` to return + ``None``. (Contributed by Brett Cannon in :issue:`23014`.) + +* The :func:`re.split` function always ignored empty pattern matches, so the + ``"x*"`` pattern worked the same as ``"x+"``, and the ``"\b"`` pattern never + worked. Now :func:`re.split` raises a warning if the pattern could match + an empty string. For compatibility use patterns that never match an empty + string (e.g. ``"x+"`` instead of ``"x*"``). Patterns that could only match + an empty string (such as ``"\b"``) now raise an error. + (Contributed by Serhiy Storchaka in :issue:`22818`.) + +* The :class:`http.cookies.Morsel` dict-like interface has been made self + consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key` + and :attr:`~http.cookies.Morsel.value` into account, + :meth:`~http.cookies.Morsel.copy` now results in a + :class:`~http.cookies.Morsel` instance rather than a :class:`dict`, and + :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the + keys in the update dictionary are invalid. In addition, the undocumented + *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and + is now ignored. (Contributed by Demian Brecht in :issue:`2211`.) + +* :pep:`488` has removed ``.pyo`` files from Python and introduced the optional + ``opt-`` tag in ``.pyc`` file names. The + :func:`importlib.util.cache_from_source` has gained an *optimization* + parameter to help control the ``opt-`` tag. Because of this, the + *debug_override* parameter of the function is now deprecated. `.pyo` files + are also no longer supported as a file argument to the Python interpreter and + thus serve no purpose when distributed on their own (i.e. sourcless code + distribution). Due to the fact that the magic number for bytecode has changed + in Python 3.5, all old `.pyo` files from previous versions of Python are + invalid regardless of this PEP. + +* The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_FD_FRAMES` + constant on linux 3.6 and greater. + +* The :func:`ssl.cert_time_to_seconds` function now interprets the input time + as UTC and not as local time, per :rfc:`5280`. Additionally, the return + value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) + +* The ``pygettext.py`` Tool now uses the standard +NNNN format for timezones in + the POT-Creation-Date header. + +* The :mod:`smtplib` module now uses :data:`sys.stderr` instead of previous + module level :data:`stderr` variable for debug output. If your (test) + program depends on patching the module level variable to capture the debug + output, you will need to update it to capture sys.stderr instead. + +* The :meth:`str.startswith` and :meth:`str.endswith` methods no longer return + ``True`` when finding the empty string and the indexes are completely out of + range. (Contributed by Serhiy Storchaka in :issue:`24284`.) + +* The :func:`inspect.getdoc` function now returns documentation strings + inherited from base classes. Documentation strings no longer need to be + duplicated if the inherited documentation is appropriate. To suppress an + inherited string, an empty string must be specified (or the documentation + may be filled in). This change affects the output of the :mod:`pydoc` + module and the :func:`help` function. + (Contributed by Serhiy Storchaka in :issue:`15582`.) + +* Nested :func:`functools.partial` calls are now flattened. If you were + relying on the previous behavior, you can now either add an attribute to a + :func:`functools.partial` object or you can create a subclass of + :func:`functools.partial`. + (Contributed by Alexander Belopolsky in :issue:`7830`.) + +Changes in the C API +-------------------- + +* The undocumented :c:member:`~PyMemoryViewObject.format` member of the + (non-public) :c:type:`PyMemoryViewObject` structure has been removed. + All extensions relying on the relevant parts in ``memoryobject.h`` + must be rebuilt. + +* The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + +* Removed non-documented macro :c:macro:`PyObject_REPR` which leaked references. + Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions + to format the :func:`repr` of the object. + (Contributed by Serhiy Storchaka in :issue:`22453`.) + +* Because the lack of the :attr:`__module__` attribute breaks pickling and + introspection, a deprecation warning now is raised for builtin type without + the :attr:`__module__` attribute. Would be an AttributeError in future. + (Contributed by Serhiy Storchaka in :issue:`20204`.) + +* As part of :pep:`492` implementation, ``tp_reserved`` slot of + :c:type:`PyTypeObject` was replaced with a + :c:member:`tp_as_async` slot. Refer to :ref:`coro-objects` for + new types, structures and functions. + diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index 29902e407c..edb55022bd 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 3.5.rst 3.4.rst 3.3.rst 3.2.rst |
